Hi, I’m using headless dietpi on my RPi4 and most of the times it is working great. However, sometimes, I’m surprised to found that my services are down (homebridge, plex, etc.) because for some reason the pi frozen — I can’t even access it through ssh when this happens (which is the only way I have to access it at the moment). The only way I have to solve it is by force restarting (power off/on).
How can I see/understand what is causing this issue, since it’s running headless?
Thank you!
You can enable persistent system logs:
dietpi-software uninstall 103 # uninstalls DIetPi-RAMlog
mkdir /var/log/journal # triggers systemd-journald logs to disk
reboot # required to finalise the RAMlog uninstall
Then you can check system logs via:
journalctl
which will then show as well logs from previous boot sessions. To limit the size, you can additionally e.g. apply the following:
mkdir -p /etc/systemd/journald.conf.d
cat << '_EOF_' > /etc/systemd/journald.conf.d/99-custom.conf
[Journal]
SystemMaxFiles=2
MaxFileSec=7day
_EOF_
This will limit logs to 14 days split across two journal files, so that with rotation you will always have between 7 and 14 days of logs available.
Thank you!
By looking into the current session I can see this multiple times:
Dec 21 07:26:15 DietPi systemd[1]: hostapd.service: Scheduled restart job, restart counter is at 15761.
Dec 21 07:26:15 DietPi systemd[1]: Stopped Access point and authentication server for Wi-Fi and Ethernet.
Dec 21 07:26:15 DietPi systemd[1]: Starting Access point and authentication server for Wi-Fi and Ethernet...
Dec 21 07:26:15 DietPi hostapd[52907]: Configuration file: /etc/hostapd/hostapd.conf
Dec 21 07:26:15 DietPi hostapd[52907]: Could not open configuration file '/etc/hostapd/hostapd.conf' for reading.
Dec 21 07:26:15 DietPi hostapd[52907]: Failed to set up interface with /etc/hostapd/hostapd.conf
Dec 21 07:26:15 DietPi hostapd[52907]: Failed to initialize interface
Dec 21 07:26:15 DietPi systemd[1]: hostapd.service: Control process exited, code=exited, status=1/FAILURE
Dec 21 07:26:15 DietPi systemd[1]: hostapd.service: Failed with result 'exit-code'.
Dec 21 07:26:15 DietPi systemd[1]: Failed to start Access point and authentication server for Wi-Fi and Ethernet.
Everything seems to be working property (I’m connected through ETH) so I don’t really understand what the issue is.
looks like you have a HotSpot installed which is not working because of missing configuration files.
Could not open configuration file '/etc/hostapd/hostapd.conf'
I don’t remember installing one. Is this part of dietpi?
Not by default. It would need to be activated/installed within dietpi-software
Joulinar maybe it’s now easier to sort what’s happening. I noticed today that it has freeze again (around 19h) when I manually rebooted and checked the logs for the culprit. This are the logs:
Dec 28 03:17:33 DietPi Radarr[921]: [Info] WebHostController: Listening on the following URLs:
Dec 28 03:17:33 DietPi Radarr[921]: [Info] WebHostController: http://*:7878
Dec 28 03:17:33 DietPi Sonarr[918]: [Info] SonarrBootstrapper: Starting Web Server
Dec 28 03:17:35 DietPi Radarr[921]: [Info] RadarrBootstrapper: Starting Web Server
Dec 28 03:17:35 DietPi systemd[1]: systemd-fsckd.service: Succeeded.
-- Boot c7441da7a2a341f5a2a585b1c9470e71 --
Dec 28 03:20:01 DietPi CRON[32312]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Dec 28 03:20:01 DietPi CRON[32313]: (root) CMD ( PATH="$PATH:/usr/sbin:/usr/local/bin/" pihole updatechecker local)
Dec 28 03:20:01 DietPi CRON[32312]: (CRON) info (No MTA installed, discarding output)
Dec 28 03:20:01 DietPi CRON[32312]: pam_unix(cron:session): session closed for user root
-- Boot 34aeae6b8e514b9184303d7adeea4a22 --
Dec 28 19:38:34 DietPi systemd-timesyncd[387]: Initial synchronization to time server [2001:67c:2e74:1::17]:123 (2.europe.pool.ntp.org).
Dec 28 19:38:34 DietPi systemd[1]: Starting Clean php session files...
Dec 28 19:38:34 DietPi dhclient[496]: DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval 8
Dec 28 19:38:34 DietPi systemd[1]: systemd-hostnamed.service: Succeeded.
I can see that around 3 AM there was some maintenance done by Radarr/Sonarr (this is just the end of the log regarding that, but it started around 3 AM) then I can see a Boot — where does this boot come from!? I actually don’t get it, since I was sleeping.
Then, the 4 lines below that, related with the CRON I don’t know what they are and what they are doing, but as you can see, it could have probably been frozen since that until I manually rebooted when I noticed today at 19:38.
Now, time to understand what’s causing the pi to freeze and become unresponsive until a manual reboot happens. 
can you share the entire log, let’s say for at least a couple of more hours.
Dec 28 03:17 is the last saved time stamp. By default DietPi is saving a time stamp once a hour. It will be used on reboot to have an initial time before time sync succeed. This usually happen on the 17th minute of a hour. Therefore you see Dec 28 03:17. This means the system crashed between 3:17 - 4:17.
As it will a quite some lines, just copy it into a text file and attach it to the post.
The log shows entries from 29th only. But doesn’t matter. There seems to be an issue with your external disk as your system is not able to mount it correctly.
Dec 29 00:55:30 DietPi systemd[1]: dev-disk-by\x2duuid-38DE75CFDE7585C2.device: Job dev-disk-by\x2duuid-38DE75CFDE7585C2.device/start timed out.
Dec 29 00:55:30 DietPi systemd[1]: Timed out waiting for device /dev/disk/by-uuid/38DE75CFDE7585C2.
Dec 29 00:55:30 DietPi systemd[1]: Dependency failed for /mnt/storage.
Dec 29 00:55:30 DietPi systemd[1]: mnt-storage.mount: Job mnt-storage.mount/start failed with result 'dependency'.
Dec 29 00:55:30 DietPi systemd[1]: dev-disk-by\x2duuid-38DE75CFDE7585C2.device: Job dev-disk-by\x2duuid-38DE75CFDE7585C2.device/start failed with result 'timeout'.
Hum, could it be the reason for it to freeze? I’ve one external ssd (with OS) and one media storage hdd (the storage one). Which one is having issues? By the logs seems to be the storage one which shouldn’t result in any freezes for the pi IMO. Am I right?
how do you power your external USB devices. Do they have an own PSU or are both powered via RPi only?
The drive issue looks like it is from boot time. Good to investigate. Indeed the Pi-hole updater cron job is the last log entry from before the crash it seems. It produces some output (not shown, only the info that there is output). Can you run and paste the output of:
pihole updatechecker local
Another thing that caught my attention is that systemd-hostnamed is running. As long as you do not switched to systemd-networkd this is not required. So you may disable it:
systemctl disable --now systemd-hostnamed
Hi! Thank you for helping me out with this MichaIng . This is the output:
root@DietPi:~# pihole updatechecker local
/usr/local/bin/pihole: line 22: /opt/pihole/COL_TABLE: No such file or directory
/usr/local/bin/pihole: line 423: /opt/pihole/updatecheck.sh: No such file or directory
I don’t have the pihole at the moment installed, so I’m wondering why is it updating.
probably /etc/cron.d/pihole still exists. I guess it can be removed if PiHole is not needed anymore.
How did you uninstall Pi-hole? Via dietpi-software or pihole uninstall? As long as there was not abortion due to an error, in both cases the Cron job and pihole executable shouldn’t exist anymore 
. Might be worth to search for other untended remains:
find / -name '*pihole*'
I don’t actually remember. But the search resulted in multiple results which I already removed:
/etc/sudoers.d/pihole
/etc/cron.d/pihole
/etc/bash_completion.d/pihole
/usr/local/bin/pihole
/var/log/pihole-FTL.log
/var/log/pihole.log.1
/var/log/pihole.log
/var/log/pihole_updateGravity.log
Let’s see if it helps from here.
Thank you MichaIng and Joulinar.
hmm not sure if such leftovers are able to freeze a complet system. I guess there is more behind.