Rpi 3B+ crashes with SSD

Xperimental · 26 May 2022 09:06

I have an issue with my Dietpi. I am unable to get it up and running.
It keeps crashes within a day.

Required Information

DietPi version:

G_DIETPI_VERSION_CORE=8
G_DIETPI_VERSION_SUB=4
G_DIETPI_VERSION_RC=2
G_GITBRANCH=‘master’
G_GITOWNER=‘MichaIng’

Distro version:

bullseye 0

Kernel version:

Linux DietPi 5.15.32-v8+ #1538 SMP PREEMPT Thu Mar 31 19:40:39 BST 2022 aarch64 GNU/Linux

SBC model:

RPi 3 Model B+ (aarch64)

Power supply used:

Power hub Xstorm 5V 2.4A 60W
Sossolution 5V 2.5A
NOW: Raspberry Foundation Official power supply 5V 2.5A

SSD card used:

Kingston A2000 250GB NVMe M.2 80mm in a metal USB-C case

Additional Information:

Software:

Sonarr
Radarr
Prowlarr

Journalctl --dmesg:

May 25 18:13:01 DietPi kernel: cacheinfo: Unable to detect cache hierarchy for CPU 0
May 25 18:13:01 DietPi kernel: brd: module loaded
May 25 18:13:01 DietPi kernel: loop: module loaded
May 25 18:13:01 DietPi kernel: Loading iSCSI transport class v2.0-870.
May 25 18:13:01 DietPi kernel: usbcore: registered new interface driver r8152
May 25 18:13:01 DietPi kernel: usbcore: registered new interface driver lan78xx
May 25 18:13:01 DietPi kernel: usbcore: registered new interface driver smsc95xx
May 25 18:13:01 DietPi kernel: dwc_otg: version 3.00a 10-AUG-2012 (platform bus)
May 25 18:13:01 DietPi kernel: Core Release: 2.80a
May 25 18:13:01 DietPi kernel: Setting default values for core params
May 25 18:13:01 DietPi kernel: Finished setting default values for core params
May 25 18:13:01 DietPi kernel: Using Buffer DMA mode
May 25 18:13:01 DietPi kernel: Periodic Transfer Interrupt Enhancement - disabled
May 25 18:13:01 DietPi kernel: Multiprocessor Interrupt Enhancement - disabled
May 25 18:13:01 DietPi kernel: OTG VER PARAM: 0, OTG VER FLAG: 0
May 25 18:13:01 DietPi kernel: Dedicated Tx FIFOs mode
May 25 18:13:01 DietPi kernel:
May 25 18:13:01 DietPi kernel: WARN::dwc_otg_hcd_init:1074: FIQ DMA bounce buffers: virt = ffffffc009a51000 dma = 0x00000000fa810000 len=9024
May 25 18:13:01 DietPi kernel: FIQ FSM acceleration enabled for :
Non-periodic Split Transactions
Periodic Split Transactions
High-Speed Isochronous Endpoints
Interrupt/Control Split Transaction hack enabled
May 25 18:13:01 DietPi kernel: dwc_otg: Microframe scheduler enabled
May 25 18:13:01 DietPi kernel:
May 25 18:13:01 DietPi kernel: WARN::hcd_init_fiq:497: MPHI regs_base at ffffffc00962d000
May 25 18:13:01 DietPi kernel: dwc_otg 3f980000.usb: DWC OTG Controller
May 25 18:13:01 DietPi kernel: dwc_otg 3f980000.usb: new USB bus registered, assigned bus number 1
May 25 18:13:01 DietPi kernel: dwc_otg 3f980000.usb: irq 74, io mem 0x00000000
May 25 18:13:01 DietPi kernel: Init: Port Power? op_state=1
May 25 18:13:01 DietPi kernel: Init: Power Port (0)
May 25 18:13:01 DietPi kernel: usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 5.15
May 25 18:13:01 DietPi kernel: usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
May 25 18:13:01 DietPi kernel: usb usb1: Product: DWC OTG Controller
May 25 18:13:01 DietPi kernel: usb usb1: Manufacturer: Linux 5.15.32-v8+ dwc_otg_hcd
May 25 18:13:01 DietPi kernel: usb usb1: SerialNumber: 3f980000.usb
May 25 18:13:01 DietPi kernel: hub 1-0:1.0: USB hub found
May 25 18:13:01 DietPi kernel: hub 1-0:1.0: 1 port detected

─────────────────────────────────────────────────────
DietPi CPU Info
Use dietpi-config to change CPU / performance options
─────────────────────────────────────────────────────
Architecture | aarch64
Temperature | 49 °C / 120 °F : Optimal temperature
Governor | conservative
Throttle up | 50% CPU usage
Current Freq Min Freq Max Freq
CPU0 | 1000 MHz 600 MHz 1400 MHz
CPU1 | 1000 MHz 600 MHz 1400 MHz
CPU2 | 1000 MHz 600 MHz 1400 MHz
CPU3 | 1000 MHz 600 MHz 1400 MHz
[ INFO ] DietPi-CPU_info | CPU current frequency, may be affected by this script, due to the processing required to run it.

Steps to reproduce

Flashed DietPi with BalenaEtcher on he SSD (now running ARMv8 64-bit, but trued ARMv7 32-bit)
Dietpi basic config: GPUI off, changed pass, set OpenSSH, set time/date, CPU conservative mode, max cpu temp 65 C
Installed software as listed
Mounted NAS dietpi_drive_manager (NFS)
Inserted Media in Sonarr and Radarr lib

Expected behaviour

A stable DietPi with the current SSD.

Actual behaviour

DietPi clogges and eventually crashes. Unable to connect with SSH or the running apps.

Extra information
Tried serveral power supplies, as posted above. (with 64-bit and 32-bit version of DietPi)
DietPi runs stable with an Samsung USB FIT 32GB
Changed USB-C between RPi and SSD case.
Swapped SSD NVMe for a new one (same type)

Joulinar · 26 May 2022 09:20

Welcome to our community.

You are running the system with an attached USB HDD/SSD? Does this device has an own power supply unit?

Xperimental · 26 May 2022 10:11

Thank you

The device doesn’t have it’s own power supply.
I’m using a JEYI i9 GTR NVMe Aluminium case.
I use this simular case/setup for Home Assistant and it is stable for months.
And a friend of mine has the same setup aswell except he is using a 500GB Kingston SSD.

Joulinar · 26 May 2022 10:17

From our side we do not recommend to run external USB devices without own PSU as we had quite some reports related to power consumption in such cases.

Anyway to be able to have a look to the system logs before the crash happen, you would need to make them permanent. Persistent system logs could be enabled as follows:

dietpi-software uninstall 103 # uninstalls DIetPi-RAMlog
mkdir /var/log/journal # triggers systemd-journald logs to disk
reboot # required to finalise the RAMlog uninstall

Then you can check system logs via:

journalctl

which will then show as well logs from previous boot sessions. To limit the size, you can additionally e.g. apply the following:

mkdir -p /etc/systemd/journald.conf.d
cat << '_EOF_' > /etc/systemd/journald.conf.d/99-custom.conf
[Journal]
SystemMaxFiles=2
MaxFileSec=7day
_EOF_

This will limit logs to 14 days split across two journal files, so that with rotation you will always have between 7 and 14 days of logs available.

Xperimental · 26 May 2022 10:53

Thanks for the help.

I am aware of the power issues that may come if a SSD hasn’t its own power source.
Though I find it strange my friend doesn’t have this issue with almost the same setup.
And just build a new setup where the Pi and the SSD is powered by POE++.

I have applied the follwing:
dietpi_help

I will get back to you when I have more info.

Joulinar · 26 May 2022 10:57

btw, according your screen print, there is some interesting character in front of the [Journal] tag. Just double check if it is as well present in /etc/systemd/journald.conf.d/99-custom.conf

Xperimental · 26 May 2022 11:01

Thanks, wasn’t sure myself either.
EOF was missing in 99-custom.conf

What’s the best journalctl command to get a clear report of the crash?

Joulinar · 26 May 2022 11:13

That’s fine. _EOF_ means end of file and is basically the trigger for the command to indicate that the input is done.

The whole content of the file should looks like this

[Journal]
SystemMaxFiles=2
MaxFileSec=7day

Best is to simply run journalctl. It will be a long output but might give the best view on what happen. It might be that there is a mixture with the time stamps after the crash. This usually happen because the system is starting with a fixed/saved time stamp. By default we save it every hour at the 17th minute. This fixed time stamp is used until the system finished booting and was able to sync time with a NTP time server. Something to keep in mind and not to get confused.

Xperimental · 26 May 2022 17:33

It looks like it hasn’t logged anything before crash, right?

log.txt (36.6 KB)

Joulinar · 26 May 2022 19:39

I copied all the logs you have posted together in a single file and attached it as log.txt to your last message. Hope this was ok.

Yes correct, it just logged the boot process. Looks like the log was not persisted and did not survive the reboot. Could retry all steps described above again?

You could verify to log being persistent by performing a reboot and check if there are message from before the reboot.

giuliomagnifico · 28 May 2022 19:05

Maybe a stupid question, but is your EEPROM already updated to the latest release?

Xperimental · 29 May 2022 07:54

No such thing as stupid querstions I appreciate the interest in my problem.

rpi-eeprom-update

I cannot check if it’s up to date, command not found.
Is there another command? I am using a RPi 3B+

So far, the pi is running for 2 days now…

giuliomagnifico · 29 May 2022 08:44

Oh, I missed that you are on a 3B, the 3B doesn’t have the EEPROM, I’m on a 4B and I’m using it via USB 3 SSD, works fine, never have troubles/locks.

Xperimental · 31 May 2022 20:05

Yes, that’s what I thought aswell.

The Pi is still running btw.

Xperimental · 12 June 2022 07:01

The pi eventually crashed, here is the crash log.

log_220612.txt (47.4 KB)

I noticed the temperature of the Pi was 60 degrees when it just booted.

Do you see something unusual ?

Joulinar · 12 June 2022 14:46

The system seems to have crashed on the 8th of June somewhere between 11pm and midnight. At least there are no other information in the log except the system boot.

Jun 08 23:17:32 DietPi systemd[1]: systemd-fsckd.service: Succeeded.
Jun 12 08:45:40 DietPi systemd-timesyncd[284]: Initial synchronization to time server 188.166.67.25:123 (2.debian.pool.ntp.org).

You see how the time is jumping from the last saved time point at Jun 08 23:17 to the real time Jun 12 08:45.

Usually the time is saved down to disk every hour at the 17th minute. I assume the system stopped working after Jun 08 23:17, as there was no time point synced afterwards. Usually the next one should have been done on Jun 09 00:17.

Xperimental · 12 June 2022 18:08

Yes, I found out today I wasn’t able to connect to the pi anymore. I rebooted the device @ 08:45.

What about these warnings?

Jun 08 23:17:01 DietPi kernel:
Jun 08 23:17:01 DietPi kernel: WARN::dwc_otg_hcd_init:1074: FIQ DMA bounce buffers: virt = ffffffc009a51000 dma = 0x00000000fa810000 len=9024
Jun 08 23:17:01 DietPi kernel: FIQ FSM acceleration enabled for :
                               Non-periodic Split Transactions
                               Periodic Split Transactions
                               High-Speed Isochronous Endpoints
                               Interrupt/Control Split Transaction hack enabled
Jun 08 23:17:01 DietPi kernel: dwc_otg: Microframe scheduler enabled
Jun 08 23:17:01 DietPi kernel:
Jun 08 23:17:01 DietPi kernel: WARN::hcd_init_fiq:497: MPHI regs_base at ffffffc00962d000
Jun 08 23:17:01 DietPi kernel: dwc_otg 3f980000.usb: DWC OTG Controller
Jun 08 23:17:01 DietPi kernel: dwc_otg 3f980000.usb: new USB bus registered, assigned bus number 1
Jun 08 23:17:01 DietPi kernel: dwc_otg 3f980000.usb: irq 74, io mem 0x00000000
Jun 08 23:17:01 DietPi kernel: Init: Port Power? op_state=1
Jun 08 23:17:01 DietPi kernel: Init: Power Port (0)

And at some point it keeps detecting new USB devices:

Jun 08 23:17:01 DietPi kernel: usb 1-1.1.1: new high-speed USB device number 4 using dwc_otg
Jun 08 23:17:01 DietPi kernel: usb 1-1.1.1: New USB device found, idVendor=0424, idProduct=7800, bcdDevice= 3.00
Jun 08 23:17:01 DietPi kernel: usb 1-1.1.1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
Jun 08 23:17:01 DietPi kernel: lan78xx 1-1.1.1:1.0 (unnamed net_device) (uninitialized): No External EEPROM. Setting MAC Speed
Jun 08 23:17:01 DietPi kernel: lan78xx 1-1.1.1:1.0 (unnamed net_device) (uninitialized): int urb period 64
Jun 08 23:17:01 DietPi kernel: usb 1-1.1.3: new high-speed USB device number 5 using dwc_otg
Jun 08 23:17:01 DietPi kernel: usb 1-1.1.3: New USB device found, idVendor=0bda, idProduct=9210, bcdDevice=20.01
Jun 08 23:17:01 DietPi kernel: usb 1-1.1.3: New USB device strings: Mfr=1, Product=2, SerialNumber=3
Jun 08 23:17:01 DietPi kernel: usb 1-1.1.3: Product: RTL9210
Jun 08 23:17:01 DietPi kernel: usb 1-1.1.3: Manufacturer: Realtek
Jun 08 23:17:01 DietPi kernel: usb 1-1.1.3: SerialNumber: 012345678918
Jun 08 23:17:01 DietPi kernel: usb-storage 1-1.1.3:1.0: USB Mass Storage device detected

And

Jun 08 23:17:03 DietPi kernel: lan78xx 1-1.1.1:1.0 eth0: kevent 4 may have been dropped
Jun 08 23:17:03 DietPi kernel: lan78xx 1-1.1.1:1.0 eth0: kevent 4 may have been dropped
Jun 08 23:17:03 DietPi kernel: IPv6: ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready

When I restarted the device (unplug & plug power back in) The Pi was 60 degrees C. Could the Pi running to hot at some point?

Joulinar · 12 June 2022 20:13

These messages are generated during the boot process you initiated today morning.