netdata fails to start after reboot

cmonty14 · March 10, 2022, 8:35pm

Hello,

when I boot RPi4 running DietPi a can see errors related to netdata service.
Means the service fails to start because directory /var/log/netdata does not exist.

In fact this directory /var/log/netdata is missing after reboot if created during runtime.

I started investigation and I think this issue is related to the “Log System Mode”; I have enabled DietPi-RAMlog #2 because I think this suits best when booting DietPi from SSD.

I can see in file /boot/dietpi/.installed that the related log mode is enabled:

$ cat /boot/dietpi/.installed | grep INDEX_LOGGING
INDEX_LOGGING=-2

The hourly cronjob /etc/cron.hourly/dietpi will run script

/boot/dietpi/func/dietpi-logclear 0

and this script will "[…] Update current log files data to “$FP_BACKUP/*. Then clear contents.”

This script will sync the following files to local storage:

ls -l /root/logfile_storage/
insgesamt 600
drwxr-xr-x 2 root root   4096 10. Mär 20:17 apt
-rw-r--r-- 1 root root    662 10. Mär 20:17 dpkg.log
-rw-r--r-- 1 root root 592592 10. Mär 20:17 lastlog
drwxr-xr-x 2 root root   4096 10. Mär 07:51 unattended-upgrades
-rw-r--r-- 1 root root   5600 10. Mär 21:17 wtmp

It’s clear that netdata-logdirectory is not included.

My understanding is after reboot some script will restore any data from /root/logfile_storage/ to /var/log/netdata/.

If this is true there’s a logical failure regarding netdata, then.
Because netdata’s logfile will be removed after reboot and not be restored before service netdata is starting.
And this results in failure when netdata service is supposed to start after reboot.

Independant of this issue I would like to ask this question:
Why are you controlling netdata service with dietpi-service?
What’s the benefit of this?

THX

Joulinar · March 10, 2022, 9:13pm

Usually content of /var/log should be stored automatically to /var/tmp/dietpi/logs/dietpi-ramlog_store on shutdown. And content should be restored on reboot automatically.

The script used is /boot/dietpi/func/dietpi-ramlog, triggered by service dietpi-ramlog.service. How do you reboot? Using reboot command?. You can try to manually trigger the save to /var/tmp/dietpi/logs/dietpi-ramlog_store manually.

/boot/dietpi/func/dietpi-ramlog 1

It should looks like this

root@DietPi3:/var/log# /boot/dietpi/func/dietpi-ramlog 1
2022-03-10 22:08:25 [ INFO ] DietPi-RAMlog | Storing /var/log to /var/tmp/dietpi/logs/dietpi-ramlog_store...
2022-03-10 22:08:25 [  OK  ] DietPi-RAMlog | Stored /var/log to /var/tmp/dietpi/logs/dietpi-ramlog_store.
root@DietPi3:/var/log#

We it make sense to have some service controlled by DietPi? Because on thinks like backup we are able to stop services. Same on updates and software installations. A couple of important services are excluded like DNS server and SSH server.

cmonty14 · March 11, 2022, 8:05am

After reboot I can see

dietpi-ramlog.service is running
only logfile dietpi-ramlog.log in /var/tmp/dietpi/logs/

root@eddie:~# systemctl status dietpi-ramlog.service
● dietpi-ramlog.service - DietPi-RAMlog
     Loaded: loaded (/etc/systemd/system/dietpi-ramlog.service; enabled; vendor preset: enabled)
     Active: active (exited) since Thu 2022-03-10 21:38:08 CET; 11h ago
    Process: 389 ExecStartPre=/bin/mkdir -p /var/tmp/dietpi/logs (code=exited, status=0/SUCCESS)
    Process: 396 ExecStart=/bin/dash -c /boot/dietpi/func/dietpi-ramlog 0 >> /var/tmp/dietpi/logs/dietpi-ramlog.log 2>&1 (code=exited, status=0/SUCCESS)
   Main PID: 396 (code=exited, status=0/SUCCESS)
        CPU: 23ms

Mär 10 21:38:08 eddie systemd[1]: Starting DietPi-RAMlog...
Mär 10 21:38:08 eddie systemd[1]: Finished DietPi-RAMlog.
root@eddie:~# ls -l /var/tmp/dietpi/logs/
insgesamt 4
-rw-r--r-- 1 root root 227 10. Mär 21:38 dietpi-ramlog.log
root@eddie:~# ls -l /root/logfile_storage/
insgesamt 600
drwxr-xr-x 2 root root   4096 10. Mär 20:17 apt
-rw-r--r-- 1 root root    662 10. Mär 20:17 dpkg.log
-rw-r--r-- 1 root root 592592 10. Mär 20:17 lastlog
drwxr-xr-x 2 root root   4096 10. Mär 07:51 unattended-upgrades
-rw-r--r-- 1 root root   5600 10. Mär 21:17 wtmp

This is the content of file /var/tmp/dietpi/logs/dietpi-ramlog.log:

2022-03-10 21:38:08 [ INFO ] DietPi-RAMlog | Restoring /var/log from /var/tmp/dietpi/logs/dietpi-ramlog_store...
2022-03-10 21:38:08 [ INFO ] DietPi-RAMlog | /var/tmp/dietpi/logs/dietpi-ramlog_store does not exist. Aborting...

/boot/dietpi/func/dietpi-ramlog 1

It should looks like this

root@DietPi3:/var/log# /boot/dietpi/func/dietpi-ramlog 1
2022-03-10 22:08:25 [ INFO ] DietPi-RAMlog | Storing /var/log to /var/tmp/dietpi/logs/dietpi-ramlog_store...
2022-03-10 22:08:25 [  OK  ] DietPi-RAMlog | Stored /var/log to /var/tmp/dietpi/logs/dietpi-ramlog_store.
root@DietPi3:/var/log#

This is confirmed; after running /boot/dietpi/func/dietpi-ramlog 1 directory /var/tmp/dietpi/logs/dietpi-ramlog_store/ will be created and /var/log/ will be copied.

However, this job runs daily vs. job /boot/dietpi/func/dietpi-logclear 0 that runs hourly.
My understanding is that job /boot/dietpi/func/dietpi-logclear 0 will delete the content in /var/log/ before job /boot/dietpi/func/dietpi-ramlog 1 can create a backup of this directory.
If this is true, I don’t think it makes sense.
And why is directory netdata not included in /root/logfile_storage/ after running /boot/dietpi/func/dietpi-logclear 0?

Although there could be services that you prefer to control with DietPi, I don’t agree that netdata is belonging to this list.
Question:
How can I consistently remove netdata from the list of services controlled by DietPi?

THX

Jappe · March 11, 2022, 10:29am

This can be done via dietpi-services by changing the mode of the particular service.

cmonty14 · March 11, 2022, 3:03pm

If it would be that simple…

I run this command to disable DietPi service controlling netdata:
dietpi-services systemd_controlled netdata

Then I disabled netdata.service using systemctl:
systemctl disable --now netdata.service

After this I rebooted RPi.

But the error message for starting netdata is still there; this error message shows two things:

netdata is still controlled by DietPi service
netdata fails to start because directory /var/log/netdata does not exist

root@eddie:~# dietpi-services status netdata

 DietPi-Services
─────────────────────────────────────────────────────
 Mode: status netdata

[FAILED] DietPi-Services | ● netdata.service - netdata - Real-time performance monitoring
     Loaded: loaded (/lib/systemd/system/netdata.service; disabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Fri 2022-03-11 15:52:43 CET; 3min 17s ago
       Docs: man:netdata
             file:///usr/share/doc/netdata/html/index.html
             https://github.com/netdata/netdata
    Process: 556 ExecStart=/usr/sbin/netdata -D (code=exited, status=1/FAILURE)
   Main PID: 556 (code=exited, status=1/FAILURE)
        CPU: 140ms

Mar 11 15:52:43 eddie netdata[556]: 2022-03-11 15:52:43: netdata FATAL : netdata : Cannot cd to directory '/var/log/netdata' # : No such file or directory
Mar 11 15:52:43 eddie netdata[556]: 2022-03-11 15:52:43: netdata INFO  : MAIN : EXIT: netdata prepares to exit with code 1...
Mar 11 15:52:43 eddie netdata[556]: 2022-03-11 15:52:43: netdata INFO  : MAIN : EXIT: cleaning up the database...
Mar 11 15:52:43 eddie netdata[556]: 2022-03-11 15:52:43: netdata INFO  : MAIN : Cleaning up database [0 hosts(s)]...
Mar 11 15:52:43 eddie netdata[556]: 2022-03-11 15:52:43: netdata INFO  : MAIN : EXIT: all done - netdata is now exiting - bye bye...
Mar 11 15:52:43 eddie netdata[556]: EXIT: cleaning up the database...
Mar 11 15:52:43 eddie netdata[556]: Cleaning up database [0 hosts(s)]...
Mar 11 15:52:43 eddie netdata[556]: EXIT: all done - netdata is now exiting - bye bye...
Mar 11 15:52:43 eddie systemd[1]: netdata.service: Main process exited, code=exited, status=1/FAILURE
Mar 11 15:52:43 eddie systemd[1]: netdata.service: Failed with result 'exit-code'.

I would prefer to solve issue netdata is still controlled by DietPi service first.

cmonty14 · March 11, 2022, 3:48pm

Update:

Issue “netdata fails to start after reboot” is solved.
The root cause was an entry in /etc/fstab:
After deleting this line
tmpfs /var/tmp tmpfs size=25M,noatime,lazytime,nodev,nosuid,mode=1777
netdata is starting after reboot.

However I still cannot disable DietPi service control of netdata.
With this status before reboot

[code]root@eddie:~# dietpi-services systemd_controlled netdata

 DietPi-Services
─────────────────────────────────────────────────────
 Mode: systemd_controlled netdata

[  OK  ] DietPi-Services | systemd_controlled : netdata

the service is started after reboot automatically.

root@eddie:~# dietpi-services status netdata

 DietPi-Services
─────────────────────────────────────────────────────
 Mode: status netdata

[  OK  ] DietPi-Services | netdata               active (running) since Fri 2022-03-11 16:45:39 CET; 3min 36s ago

This means I have no control of netdata.service at all.

Joulinar · March 11, 2022, 9:37pm

but tmpfs /var/tmp tmpfs is not part of default DietPi if I’m not mistaken. Only /tmp and /var/log should be a tmpfs

To change netdata.service setting, just run dietpi-services, select netdata and adjust settings as follow

Mode                    : [Systemd controlled]
Include/Exclude         : [excluded]

cmonty14 · March 12, 2022, 12:07pm

This is correct; I have entered the line manually, but this makes no sense and causes errors.

With regards to netdata I prefer to install the latest version provide by netdata using netdata’s kickstart script.

In advance I deleted anything that is related to a previous netdata installation, and I modified files in /boot/dietpi that contain netdata.

It looks like DietPi service has now no control of netdata.service anymore.

Joulinar · March 12, 2022, 9:37pm

did you tried my suggestion above, to simply exclude the service using dietpi-services and selecting netdata? There you have the configuration menu to set

Mode                    : [Systemd controlled]
Include/Exclude         : [excluded]

Because changing the shell scripts inside /boot/dietpi is not a good idea as all scripts will be overwritten on next update.

cmonty14 · March 14, 2022, 7:13am

did you tried my suggestion above, to simply exclude the service using dietpi-services and selecting netdata? There you have the configuration menu to set
Mode                    : [Systemd controlled]
Include/Exclude         : [excluded]
Because changing the shell scripts inside /boot/dietpi is not a good idea as all scripts will be overwritten on next update.

Yes, I tried to exclude/disable DietPi service for netdata.
Please check my posting here.

In my opinion the issue with DietPi service is netdata is starting automatically after reboot although I executed dietpi-services systemd_controlled netdata.

Joulinar · March 14, 2022, 9:32am

I guess we still misunderstood. Pls open dietpi-services UI and navigate to your netdata service inside the UI. Once selected, you would need to set following to line as follow

Mode                    : [Systemd controlled]
Include/Exclude         : [excluded]

Setting the excluded value is the important one. Ensure both values are set and reboot your system. Now you should be able to control netdata service by your own and have it disabled to avoid autostart on boot.

Just tested on my test system and it was working this way. netdata stays disabled even after reboot.