HI all, I am having difficulty tracking down why my Pi 3B+ has started consistently crashing at the same time every night.
Diet Pi 3B+ originally set up on an SD card, moved to a USB SSD. Minimal software - Plex Server, Radarr, Sonarr, Jackett - all from the official Dietpi-Software library. Everything up to date.
I’ve enabled persistent Journalctl as per another post, but nothing in the logs seems obvious - last log entry is at 01:17 (the Pi stops responding at around 02:06), then nothing until I force reboot. There are, however, some messages on boot I don;t fully understand - is there a clue there? Have I missed an obvious step, perhaps, when transferring the image across to an SSD?
Any help from the logs, or what to look at next to try and get an idea would be much appreciated! I’d quite like to understand, and fix the issue, rather than flatten it and try and set it all up again if I can help it! Understanding it will also, hopefully, help me prevent similar happening again.
Or, worst case, any tips that will help me flatten/re-image and get back to the same config without too much fuss if that is the only answer!
Hm this is the suspicious stuff I found (see summary):
Summary
Dec 13 01:17:01 Pi-Plex kernel: Booting Linux on physical CPU 0x0000000000 [0x410fd034]
...
...
Dec 13 01:17:01 Pi-Plex systemd-timesyncd[168]: System clock time unset or jumped backwards, restoring from recorded timestamp: Wed 2022-12-14 15:06:18 GMT
Dec 14 15:06:18 Pi-Plex systemd[1]: Started Network Time Synchronization.
Dec 14 15:06:18 Pi-Plex systemd[1]: Reached target System Initialization.
So the last recorded timestamp is Wed 2022-12-14 15:06:18 GMT, it boots and thinks it is Dec 13, 01:17:01, bur realizes it is at least Dec 14, 15:06.
Dec 14 15:06:19 Pi-Plex systemd[1]: Starting DietPi-PreBoot...
Dec 14 15:06:19 Pi-Plex systemd[181]: dietpi-preboot.service: Failed to locate executable /boot/dietpi/preboot: No such file or directory
Dec 14 15:06:19 Pi-Plex systemd[181]: dietpi-preboot.service: Failed at step EXEC spawning /boot/dietpi/preboot: No such file or directory
Dec 14 15:06:19 Pi-Plex systemd[1]: Started Start entropy gathering daemon (rngd).
Dec 14 15:06:19 Pi-Plex systemd[1]: dietpi-preboot.service: Main process exited, code=exited, status=203/EXEC
Dec 14 15:06:19 Pi-Plex systemd[1]: dietpi-preboot.service: Failed with result 'exit-code'.
Dec 14 15:06:19 Pi-Plex systemd[1]: Failed to start DietPi-PreBoot.
Dec 14 15:06:19 Pi-Plex systemd[1]: Dependency failed for DietPi-PostBoot.
Dec 14 15:06:19 Pi-Plex systemd[1]: dietpi-postboot.service: Job dietpi-postboot.service/start failed with result 'dependency'.
Some files are missing, which indicates a corrupted filesystem? (which comes maybe from the pwoercycling when it came unresponsive) More hints to underpin this:
Dec 14 15:06:20 Pi-Plex systemd-fsck[360]: fsck.fat 4.2 (2021-01-31)
Dec 14 15:06:20 Pi-Plex systemd-fsck[360]: There are differences between boot sector and its backup.
Dec 14 15:06:20 Pi-Plex systemd-fsck[360]: This is mostly harmless. Differences: (offset:original/backup)
Dec 14 15:06:20 Pi-Plex systemd-fsck[360]: 14:26/20, 26:ff/40, 32:00/ff, 33:00/ff, 34:04/03, 65:01/00
Dec 14 15:06:20 Pi-Plex systemd-fsck[360]: Not automatically fixing this.
Dec 14 15:06:20 Pi-Plex systemd-fsck[360]: Dirty bit is set. Fs was not properly unmounted and some data may be corrupt.
Dec 14 15:06:20 Pi-Plex systemd-fsck[360]: Automatically removing dirty bit.
Dec 14 15:06:20 Pi-Plex systemd-fsck[360]: *** Filesystem was changed ***
Dec 14 15:06:20 Pi-Plex systemd-fsck[360]: Writing changes.
Also later a timesync fails, which 30 minutes prior succeeded.
Dec 14 15:10:08 Pi-Plex systemd-timesyncd[168]: Initial synchronization to time server 130.159.196.118:123 (2.debian.pool.ntp.org).
Dec 14 15:45:04 Pi-Plex systemd-timesyncd[168]: Timed out waiting for reply from 130.159.196.118:123 (2.debian.pool.ntp.org).
Do you use an extra power supply for the USB SSD?
And how do you moved the system to SSD?
This means system was crashing after our hourly time stamp save job. Every hour at the 17th minute, we save the current time stamp down to disk. This time stamp will be used on early boot to have at least a time/date. The system time will be corrected as soon as network is online and a ntp server could be reached.
Dec 13 01:17:01 Pi-Plex fake-hwclock[137]: Tue Dec 13 01:17:01 UTC 2022
Anyway, let’s have a look to the journal directory