Influxdb not working after update

Troubleshooting
1

Introduction

After the last influxdb update, influxdb is not starting up. I noticed that the location of some influxdb folders changes. I hope that the attached logs and folder comparison helps to find the issue

Updated packages

The following packages will be upgraded:
influxdb jellyfin-server jellyfin-web libssl-dev libssl1.1 openssl telegraf

Updated influxdb failed

Setting up influxdb (1.11.7-1) ...

Installing new version of config file /etc/influxdb/influxdb.conf ...
Job for influxdb.service failed because the control process exited with error code.
See "systemctl status influxdb.service" and "journalctl -xe" for details.

systemctl error output

root@jhnPi:~# systemctl status influxdb.service
influxdb.service - InfluxDB is an open-source, distributed, time series database
     Loaded: loaded (/lib/systemd/system/influxdb.service; enabled; vendor preset: enabled)
    Drop-In: /etc/systemd/system/influxdb.service.d
             └─dietpi-services_edit.conf
     Active: failed (Result: exit-code) since Mon 2024-11-04 08:08:11 GMT; 1min 24s ago
       Docs: https://docs.influxdata.com/influxdb/
    Process: 6044 ExecStart=/usr/share/influxdb/scripts/influxd-systemd-start.sh (code=exited, status=1/FAILURE)
        CPU: 641ms

Nov 04 08:08:11 jhnPi systemd[1]: influxdb.service: Failed with result 'exit-code'.
Nov 04 08:08:11 jhnPi systemd[1]: Failed to start InfluxDB is an open-source, distributed, time series database.
Nov 04 08:08:11 jhnPi systemd[1]: influxdb.service: Scheduled restart job, restart counter is at 8.
Nov 04 08:08:11 jhnPi systemd[1]: Stopped InfluxDB is an open-source, distributed, time series database.
Nov 04 08:08:11 jhnPi systemd[1]: influxdb.service: Start request repeated too quickly.
Nov 04 08:08:11 jhnPi systemd[1]: influxdb.service: Failed with result 'exit-code'.
Nov 04 08:08:11 jhnPi systemd[1]: Failed to start InfluxDB is an open-source, distributed, time series database.

journalctl error output

Nov 04 08:08:04 jhnPi systemd[1]: influxdb.service: Scheduled restart job, restart counter is at 1.
Nov 04 08:08:04 jhnPi systemd[1]: Stopped InfluxDB is an open-source, distributed, time series database.
Nov 04 08:08:04 jhnPi systemd[1]: Starting InfluxDB is an open-source, distributed, time series database...
Nov 04 08:08:04 jhnPi influxd-systemd-start.sh[5801]: /usr/share/influxdb/scripts/influxd-systemd-start.sh: line 54: /var/lib/influxdb/influxd.pid: No such file or directory
Nov 04 08:08:05 jhnPi influxd-systemd-start.sh[5838]: ts=2024-11-04T08:08:05.115770Z lvl=info msg="InfluxDB starting" log_id=0seZJYzl000 version=v1.11.7 branch=unknown commit=1096cc4798eea5dfa47a6d91844c04a8f8b8a0fc
Nov 04 08:08:05 jhnPi influxd-systemd-start.sh[5838]: ts=2024-11-04T08:08:05.115870Z lvl=info msg="Go runtime" log_id=0seZJYzl000 version=go1.22.7 maxprocs=4
Nov 04 08:08:05 jhnPi influxd-systemd-start.sh[5838]: ts=2024-11-04T08:08:05.115924Z lvl=info msg="configured logger" log_id=0seZJYzl001 format=auto level=info
Nov 04 08:08:05 jhnPi influxd-systemd-start.sh[5838]: run: create server: mkdir all: mkdir /var/lib/influxdb: permission denied
Nov 04 08:08:05 jhnPi systemd[1]: influxdb.service: Control process exited, code=exited, status=1/FAILURE
Nov 04 08:08:05 jhnPi systemd[1]: influxdb.service: Failed with result 'exit-code'.

Compare DietPi-Services output

Pre Update

  ┌──────────────────────────────────────────┤ DietPi-Services | File viewer ├───────────────────────────────────────────┐
  │                                                                                                                      │
  │ ● influxdb.service - InfluxDB is an open-source, distributed, time series database                                   │
  │      Loaded: loaded (/lib/systemd/system/influxdb.service; enabled; vendor preset: enabled)                          │
  │     Drop-In: /etc/systemd/system/influxdb.service.d                                                                  │
  │              └─dietpi-services_edit.conf                                                                             │
  │      Active: active (running) since Mon 2024-11-04 08:28:13 GMT; 1min 5s ago                                         │
  │        Docs: https://docs.influxdata.com/influxdb/                                                                   │
  │     Process: 587 ExecStart=/usr/lib/influxdb/scripts/influxd-systemd-start.sh (code=exited, status=0/SUCCESS)        │
  │    Main PID: 590 (influxd)                                                                                           │
  │       Tasks: 14 (limit: 9293)                                                                                        │
  │      Memory: 5.0G                                                                                                    │
  │         CPU: 58.085s                                                                                                 │
  │      CGroup: /system.slice/influxdb.service                                                                          │
  │              └─590 /usr/bin/influxd -config /etc/influxdb/influxdb.conf                                              │
  │                                                                                                                      │
  │ Nov 04 08:28:53 jhnPi influxd-systemd-start.sh[590]: [httpd] 127.0.0.1 - root [04/Nov/2024:08:28:53 +0000] "POST     │
  │ /write?db=jhnPi_base&p=%5BREDACTED%5D&precision=n&rp=&u=root HTTP/1.1 " 204 0 "-" "-"                                │
  │ d3695d6c-9a86-11ef-808e-e45f011dcf29 7902                                                                            │

Post Update

      ┌──────────────────────────────────────────┤ DietPi-Services | File viewer ├───────────────────────────────────────────┐
      │                                                                                                                      │
      │ ● influxdb.service - InfluxDB is an open-source, distributed, time series database                                   │
      │      Loaded: loaded (/lib/systemd/system/influxdb.service; enabled; vendor preset: enabled)                          │
      │     Drop-In: /etc/systemd/system/influxdb.service.d                                                                  │
      │              └─dietpi-services_edit.conf                                                                             │
      │      Active: failed (Result: exit-code) since Mon 2024-11-04 08:08:11 GMT; 16min ago                                 │
      │        Docs: https://docs.influxdata.com/influxdb/                                                                   │
      │     Process: 6044 ExecStart=/usr/share/influxdb/scripts/influxd-systemd-start.sh (code=exited, status=1/FAILURE)     │
      │         CPU: 641ms                                                                                                   │
      │                                                                                                                      │
      │ Nov 04 08:08:11 jhnPi systemd[1]: influxdb.service: Failed with result 'exit-code'.                                  │
      │ Nov 04 08:08:11 jhnPi systemd[1]: Failed to start InfluxDB is an open-source, distributed, time series database.     │
      │ Nov 04 08:08:11 jhnPi systemd[1]: influxdb.service: Scheduled restart job, restart counter is at 8.                  │
      │ Nov 04 08:08:11 jhnPi systemd[1]: Stopped InfluxDB is an open-source, distributed, time series database.             │
      │ Nov 04 08:08:11 jhnPi systemd[1]: influxdb.service: Start request repeated too quickly.                              │
      │ Nov 04 08:08:11 jhnPi systemd[1]: influxdb.service: Failed with result 'exit-code'.                                  │
      │ Nov 04 08:08:11 jhnPi systemd[1]: Failed to start InfluxDB is an open-source, distributed, time series database.

Compare /usr/share/

Pre Update → influxdb not included

root@jhnPi:/usr/share# ls
X11              binfmts          dns         glib-2.0         libdrm        netfilter-persistent  php7.4-opcache   systemd
aclocal          bug              doc         gnupg            lighttpd      pam                   php7.4-readline  tabset
aclocal-1.16     ca-certificates  doc-base    grafana          lintian       pam-configs           php7.4-sqlite3   terminfo
adduser          cmake            dpkg        i18n             locale        perl                  php7.4-xml       ucf
alsa             common-licenses  emacs       info             lua           perl5                 php7.4-zip       xml
applications     console-setup    file        initramfs-tools  man           php                   pixmaps          zoneinfo
apport           consolefonts     fish        iptables         menu          php7.4-common         pkgconfig        zsh
autoconf         consoletrans     fontconfig  javascript       metainfo      php7.4-curl           polkit-1
automake-1.16    dbus-1           fonts       jellyfin         misc          php7.4-gd             proftpd
awk              debconf          gcc         jellyfin-ffmpeg  munin         php7.4-intl           python3
base-files       debianutils      gdb         keyrings         mysql         php7.4-json           readline
base-passwd      dhcpcd           git-core    libc-bin         mysql-common  php7.4-mbstring       rsync
bash-completion  dict             gitweb      libdbi-perl      nano          php7.4-mysql          sensible-utils

Post Update → influxdb included

root@jhnPi:/usr/share# ls
X11              binfmts          dns         glib-2.0         libdbi-perl   nano                  php7.4-mysql     sensible-utils
aclocal          bug              doc         gnupg            libdrm        netfilter-persistent  php7.4-opcache   systemd
aclocal-1.16     ca-certificates  doc-base    grafana          lighttpd      pam                   php7.4-readline  tabset
adduser          cmake            dpkg        i18n             lintian       pam-configs           php7.4-sqlite3   terminfo
alsa             common-licenses  emacs       *influxdb*       locale        perl                  php7.4-xml       ucf
applications     console-setup    file        info             lua           perl5                 php7.4-zip       xml
apport           consolefonts     fish        initramfs-tools  man           php                   pixmaps          zoneinfo
autoconf         consoletrans     fontconfig  iptables         menu          php7.4-common         pkgconfig        zsh
automake-1.16    dbus-1           fonts       javascript       metainfo      php7.4-curl           polkit-1
awk              debconf          gcc         jellyfin         misc          php7.4-gd             proftpd
base-files       debianutils      gdb         jellyfin-ffmpeg  munin         php7.4-intl           python3
base-passwd      dhcpcd           git-core    keyrings         mysql         php7.4-json           readline
bash-completion  dict             gitweb      libc-bin         mysql-common  php7.4-mbstring       rsync

Compare /var/lib/

Pre Update → influxdb included

root@jhnPi:/var/lib# ls
apt         dbus    dietpi  grafana    misc       pam       private      redis    telegraf
bluetooth   dhcp    dpkg    *influxdb* mosquitto  php       python       sudo     ucf
containerd  dhcpcd  git     jellyfin   mysql      polkit-1  raspberrypi  systemd  unbound

Post Update → influxdb not included

root@jhnPi:/var/lib# ls
apt         dbus    dietpi  grafana    misc       pam       private      redis    telegraf
bluetooth   dhcp    dpkg    jellyfin   mosquitto  php       python       sudo     ucf
containerd  dhcpcd  git     logrotate  mysql      polkit-1  raspberrypi  systemd  unbound
2

looks like the InfluxDB package removed the symbolic link / folder on /var/lib/. Maybe an issue with the official package itself. :thinking: Unfraternally this one is out control as we do don’t maintain the package itself, but we can recreate it and link it to the location we use on DietPi.

Can you share following

ls -la /var/lib/ /mnt/dietpi_userdata/
3

Yes, I can try this. But it’s a longer process. I have to backup, update, test, and restore to the backuped version. Should I try/ test anything else?

4

This js just to verify data available.

At the the end solution will be creation of a simple symbolic link

5

Pre-Update

root@jhnPi:~# ls -la /var/lib/ /mnt/dietpi_userdata/
/mnt/dietpi_userdata/:
total 60
drwxrwxr-x 15 dietpi        dietpi        4096 Nov 18  2023 .
drwxr-xr-x  6 root          root          4096 Sep 29 18:47 ..
drwxrwxr-x  2 dietpi        dietpi        4096 Nov 18  2023 Music
drwxrwxr-x  2 dietpi        dietpi        4096 Sep 10  2022 Pictures
drwxrwxr-x  2 dietpi        dietpi        4096 Sep 10  2022 Video
drwx--x--- 13 root          root          4096 Nov  4 08:27 docker-data
drwxrwxr-x  7 dietpi        dietpi        4096 Oct  1 11:21 downloads
drwxr-xr-x  9 grafana       grafana       4096 Nov  4 19:18 grafana
drwxr-xr-x  7 homeassistant homeassistant 4096 Nov  4 08:28 homeassistant
drwxr-xr-x  5 influxdb      influxdb      4096 Nov  4 08:27 influxdb
drwxr-x--- 11 jellyfin      jellyfin      4096 Sep 26 14:22 jellyfin
drwxr-xr-x  5 mysql         mysql         4096 Nov  4 08:27 mysql
drwxr-x---  8 www-data      www-data      4096 Oct 27  2023 nextcloud_data
drwxr-xr-x  6 nodered       nodered       4096 Nov  4 18:28 node-red
drwxr-xr-x  2 root          root          4096 Nov 19  2023 tmp

/var/lib/:
total 104
drwxr-xr-x 26 root      root     4096 Oct 18 00:16 .
drwxr-xr-x 12 root      root     4096 Oct 18 00:16 ..
drwxr-xr-x  3 root      root     4096 Oct 25 16:17 apt
drwx------  3 root      root     4096 Oct 10  2023 bluetooth
drwx--x--x 11 root      root     4096 Jun 28 09:41 containerd
drwxr-xr-x  2 root      root     4096 Oct 14  2023 dbus
drwxr-xr-x  2 root      root     4096 Apr 29  2023 dhcp
drwxr-xr-x  2 root      root     4096 Oct 27 18:06 dhcpcd
drwxr-xr-x  6 root      root     4096 Oct 18 00:16 dietpi
drwxr-xr-x  7 root      root     4096 Oct 25 16:17 dpkg
drwxr-xr-x  2 root      root     4096 Mar 10  2021 git
lrwxrwxrwx  1 grafana   grafana    28 Sep 10  2022 grafana -> /mnt/dietpi_userdata/grafana
lrwxrwxrwx  1 root      root       29 Sep 10  2022 influxdb -> /mnt/dietpi_userdata/influxdb
drwxr-x---  2 jellyfin  adm      4096 Nov 30  2023 jellyfin
drwxr-xr-x  2 root      root     4096 Jun 30  2022 misc
drwxr-xr-x  2 mosquitto root     4096 Nov  4 19:27 mosquitto
lrwxrwxrwx  1 root      root       26 Sep 10  2022 mysql -> /mnt/dietpi_userdata/mysql
drwxr-xr-x  2 root      root     4096 Jul 31  2022 pam
drwxr-xr-x  4 root      root     4096 Sep 10  2022 php
drwxr-xr-x  3 root      root     4096 Jul 31  2022 polkit-1
drwx------  2 root      root     4096 Jul 31  2022 private
drwxr-xr-x  2 root      root     4096 Sep 10  2022 python
drwxr-xr-x  3 root      root     4096 Sep 10  2022 raspberrypi
drwxr-x---  2 redis     redis    4096 Nov  4 08:27 redis
drwxr-xr-x  3 root      root     4096 Jul 31  2022 sudo
drwxr-xr-x 10 root      root     4096 May 13 18:08 systemd
drwxrwx---  2 root      telegraf 4096 Jun 15  2023 telegraf
drwxr-xr-x  3 root      root     4096 Sep 10  2022 ucf
drwxr-xr-x  2 unbound   unbound  4096 Nov  4 08:27 unbound

Post-Update

root@jhnPi:~# ls -la /var/lib/ /mnt/dietpi_userdata/
/mnt/dietpi_userdata/:
total 60
drwxrwxr-x 15 dietpi        dietpi        4096 Nov 18  2023 .
drwxr-xr-x  6 root          root          4096 Sep 29 18:47 ..
drwxrwxr-x  2 dietpi        dietpi        4096 Nov 18  2023 Music
drwxrwxr-x  2 dietpi        dietpi        4096 Sep 10  2022 Pictures
drwxrwxr-x  2 dietpi        dietpi        4096 Sep 10  2022 Video
drwx--x--- 13 root          root          4096 Nov  4 19:34 docker-data
drwxrwxr-x  7 dietpi        dietpi        4096 Oct  1 11:21 downloads
drwxr-xr-x  9 grafana       grafana       4096 Nov  4 19:34 grafana
drwxr-xr-x  7 homeassistant homeassistant 4096 Nov  4 19:35 homeassistant
drwxr-xr-x  5 influxdb      influxdb      4096 Nov  4 19:31 influxdb
drwxr-x--- 11 jellyfin      jellyfin      4096 Sep 26 14:22 jellyfin
drwxr-xr-x  5 mysql         mysql         4096 Nov  4 19:34 mysql
drwxr-x---  8 www-data      www-data      4096 Oct 27  2023 nextcloud_data
drwxr-xr-x  6 nodered       nodered       4096 Nov  4 19:35 node-red
drwxr-xr-x  2 root          root          4096 Nov 19  2023 tmp

/var/lib/:
total 108
drwxr-xr-x 27 root      root     4096 Nov  4 19:32 .
drwxr-xr-x 12 root      root     4096 Oct 18 00:16 ..
drwxr-xr-x  3 root      root     4096 Nov  4 19:32 apt
drwx------  3 root      root     4096 Oct 10  2023 bluetooth
drwx--x--x 11 root      root     4096 Jun 28 09:41 containerd
drwxr-xr-x  2 root      root     4096 Oct 14  2023 dbus
drwxr-xr-x  2 root      root     4096 Apr 29  2023 dhcp
drwxr-xr-x  2 root      root     4096 Oct 27 18:06 dhcpcd
drwxr-xr-x  6 root      root     4096 Oct 18 00:16 dietpi
drwxr-xr-x  7 root      root     4096 Nov  4 19:33 dpkg
drwxr-xr-x  2 root      root     4096 Mar 10  2021 git
lrwxrwxrwx  1 grafana   grafana    28 Sep 10  2022 grafana -> /mnt/dietpi_userdata/grafana
drwxr-x---  2 jellyfin  adm      4096 Nov 30  2023 jellyfin
drwxr-xr-x  2 root      root     4096 Nov  4 19:34 logrotate
drwxr-xr-x  2 root      root     4096 Jun 30  2022 misc
drwxr-xr-x  2 mosquitto root     4096 Nov  4 19:34 mosquitto
lrwxrwxrwx  1 root      root       26 Sep 10  2022 mysql -> /mnt/dietpi_userdata/mysql
drwxr-xr-x  2 root      root     4096 Jul 31  2022 pam
drwxr-xr-x  4 root      root     4096 Sep 10  2022 php
drwxr-xr-x  3 root      root     4096 Jul 31  2022 polkit-1
drwx------  2 root      root     4096 Jul 31  2022 private
drwxr-xr-x  2 root      root     4096 Sep 10  2022 python
drwxr-xr-x  3 root      root     4096 Sep 10  2022 raspberrypi
drwxr-x---  2 redis     redis    4096 Nov  4 19:34 redis
drwxr-xr-x  3 root      root     4096 Jul 31  2022 sudo
drwxr-xr-x 10 root      root     4096 May 13 18:08 systemd
drwxrwx---  2 root      telegraf 4096 Jun 15  2023 telegraf
drwxr-xr-x  3 root      root     4096 Sep 10  2022 ucf
drwxr-xr-x  2 unbound   unbound  4096 Nov  4 19:34 unbound
6

It would have been enough to share information from after applying the package update

Solution will be

ln -s /mnt/dietpi_userdata/influxdb /var/lib/influxdb
systemctl restart influxd
7

This worked. Thank you!

8

i had the exact same problem after upgrading from 9.7 to 9.8 - given that i have not customized anything in influxdb i guess this could be a more general problem. the symlink however indeed fixed it.

9

it’s a problem of the official InfluxDB package itself and has nothing to do with DietPi update.
Due to whatever reason, the package is removing /var/lib/influxdb

10

The next DietPi update will re-create the symlink after package upgrades. Does not help anyone who upgrades APT earlier, but at least a DietPi update always repairs it.

Background is that /var/lib/influxdb was explicitly part of the package file list, which is pretty uncommon for Debian packages and such data directories. Usually they are created only via maintainer scripts (those which are executed during the package configuration step), hence InfluxDB’s recent package version removes it from the file list as well. dpkg removes files and directories which were part of the old package but not part of the new package anymore, reasonably, but it skips directories which have other/unknown content. Hence, normally /var/lib/influxdb would not be removed on an upgrade, but in our case it is a symlink, where dpkg obviously behaves differently.

1 Like