XU4 no longer boots/connects properly since update; root partition has fsck issues after every restart

[andoru]

Creating a bug report/issue

I have searched the existing open and closed issues

Required Information

• DietPi version | v8.25.1 (MichaIng/master)
• Architecture | armv7l
• SBC model | Odroid XU3/XU4/MC1/HC1/HC2
• Kernel version | Linux Server 6.1.63-current-odroidxu4 #1 SMP PREEMPT Tue Nov 7 20:10:43 UTC 2023 armv7l GNU/Linux
• Distro | bullseye (ID=6)
• Power supply used | 6V 2A
• SD card used | SanDisk HighEndurance

Steps to reproduce

1. Updated DietPi to the latest version using “dietpi-update” script
2. Finished update, don’t remember anything going awry
3. Update suggested to do a restart as new kernel was installed, pressed OK which automatically restarted the SBC (this should be mentioned on the OK button; ie. “OK (this will restart the device)”; as I did not intend to restart right away and I didn’t read the notice carefully)
4. Device restarted, but would not connect to LAN, could not SSH into it,could not ping it
5. Turned off device, moved it close to a HDMI monitor so I could diagnose

Expected behaviour

• Boot as normal, connect with correct DHCP settings, boot into DE, all unattended / not requiring display

Actual behaviour

1. After connecting the SBC to the monitor, it boots into initramfs prompt, stating that the root partition needs to be scanned with fsck manually:
   
   

(sorry for the shaky image)
2. I wasn’t able to do the fsck at the prompt, so I turned off the SBC, and did the fsck on a PC running linux on the partition on the uSD
3. Replaced the uSD back into the SBC, turned on the device again
4. Booted into DietPi, but seems if was having boot issues:

(apologies again for the shaky image)

5. Still could not ping to the SBC, the router listed the SBC connecting on the wrong IP address (I’ve configured the correct IP address assigned to it in the DHCP settings on the router), while the ifup service was not starting, the SBC would constantly change it’s IP address
6. After the ifup line either quit or failed, it asked me to log in; after that I was able to ping and ssh into the SBC.
7. Tried to run dietpi-config to see if I can fix the DHCP issue, got this message:

[FAILED] DietPi-Config | RootFS is currently Read Only (R/O) mounted. Aborting...
[ INFO ] DietPi-Config | DietPi requires RootFS to be Read/Write (R/W) mounted. Please run "dietpi-drive_manager" to re-enable.

8. Ran “dietpi-drive_manager”, after which I got this error:

[ INFO ] DietPi-Drive_Manager | Executing alternative command: mount -v -o remount,rw /
mount: /: can't read superblock on /dev/mmcblk1p1.
[FAILED] Alternative command execution | Exited with error

9. Tried restarting the SBC (“sudo shutdown -r now”), but got the same fsck error as above on step 1 (fsck exited with status error 12)

[Joulinar]

Hmm that sounds like a dying SD card.

[andoru]

I’m not sure if I would arrive to the same conclusion. This seems too much of a coincidence to have happened right after a system update.

I tried checking the uSD card if it was failing by analysing bad blocks, also checked with f3. Both came back with negative results.

So I tried saving all the config files off the uSD I knew that were relevant, then flashed the DietPi image on to the card with Balena Etcher, and turned on the SBC. It started as normal, however it wouldn’t show up as a connected device on the router. Waited for 10-15 min, still nothing.

Flashed the image again, turned the SBC back on, same issue.

Took another uSD card, flashed on that one, same issue.

Disabled the “Trim unallocated space” option in Balena in case it was causing issues, and tried again with another uSD card, again same issue.

Anything else that I can try?

[Joulinar]

It always resulted in I/O errors? And yes it could be a coincidence with the update because the update will involve package updates causing I/O operation.

[andoru]

What about the other attempts with a fresh install and other uSD cards?

[Joulinar]

To be able to further analyse, it would be needed to have serial console access, to see what happens during boot and where it stuck.

[andoru]

Alright, having no serial/UART cable available to be able to access the console, I’m stuck with hooking up the XU4 to a monitor and keyboard again.

Flashing the image onto a uSD card it boots into DietPi, with the initial login screen showing up. (the one where it prompts you to hit enter to login).
The SBC still does not show up as connected on the router panel.

If I restart the SBC, it kicks me into the initramfs prompt. I checked this with three different uSD cards, they all do the same. I think this confirms that the uSD card I used previously did not have any issues.

Once I do a fsck on the partitions on the card on my Linux setup, it boots into DietPi, with the initial login screen showing up. (the one where it prompts you to hit enter to login)

After I login, I get this screen:

Checking the uSD card on the Linux setup, the file does exist, and AUTO_SETUP_AUTOMATED is set to 0.

The file permissions on the /boot/ folder:

[...]/boot$ ls -la
total 29588
drwxr-xr-x  4 root root    4096 Dec 31 03:46 .
drwxr-xr-x 18 root root    4096 Dec 31 03:47 ..
-rw-r--r--  1 root root   10803 Dec 31 03:38 boot.ini
-rw-r--r--  1 root root  192738 Nov 27 13:12 config-6.1.63-current-odroidxu4
drwxr-xr-x  4 root root    4096 Dec 31 03:46 dietpi
-rw-r--r--  1 root root   18092 Dec 18 21:26 dietpi-LICENSE.txt
-rw-r--r--  1 root root   16059 Dec 18 21:26 dietpi-README.md
-rw-r--r--  1 root root   17431 Dec 31 03:46 dietpi.txt
-rw-------  1 root root    3950 Dec 31 03:46 dietpi-wifi.txt
lrwxrwxrwx  1 root root      28 Dec 31 03:42 dtb -> dtb-6.1.63-current-odroidxu4
drwxr-xr-x  3 root root    4096 Dec 31 03:41 dtb-6.1.63-current-odroidxu4
-rw-r--r--  1 root root 9175219 Dec 31 03:42 initrd.img-6.1.63-current-odroidxu4
-rw-r--r--  1 root root 3168868 Nov 27 13:12 System.map-6.1.63-current-odroidxu4
lrwxrwxrwx  1 root root      32 Dec 31 03:42 uInitrd -> uInitrd-6.1.63-current-odroidxu4
-rw-r--r--  1 root root 9175283 Dec 31 03:42 uInitrd-6.1.63-current-odroidxu4
-rw-r--r--  1 root root 8476584 Nov 27 13:12 vmlinuz-6.1.63-current-odroidxu4
lrwxrwxrwx  1 root root      32 Dec 31 03:42 zImage -> vmlinuz-6.1.63-current-odroidxu4

Not sure if the permissions above are correct, but I would assume they need to have read/write permissions only for root.

After I select OK, I get this:

After selecting OK again, I go back to the previous screen about missing or non-writable dietpi.txt.

No file is present under /var/tmp/dietpi/logs/

Anything else I could try?

[Joulinar]

if you reboot your system and hit by the issue, are you able to exit the initial setup and drop back to CLI?

[andoru]

Yes, if I press Cancel on the second message, I’m back to the root prompt.

[Joulinar]

@andoru

if I press Cancel on the second message, I’m back to the root prompt.

can you try to edit dietpi.txt via nano

nano /boot/dietpi.txt

Try to change stuff and save the file.

Well have a look to system log and kernel messages

journalctl
dmesg

[andoru]

So I tried to edit dietpi.txt, after saving it I get notified that the volume is mounted as read-only:

Output of dmesg and journalctl:

dmesg.log (39.4 KB)
journalctl.log (61.2 KB)

[Joulinar]

Hmm both logs contain file system I/O errors. As a result, the system is putting your RootFS into r/o mode. I see a couple of USB disk connected. Could you try to create a new image and start the system without additional disk connected.

[andoru]

Alright, downloaded the DietPi image again just to make sure.
Wiped the uSD card completely.
Flashed using Balena Etcher.
Unplugged all USB devices from the XU4.
Booted again with the flashed uSD card.

After all that I’m sorry to report but I still have the same issue. Read-only fs, no LAN connection, not able to run setup.

Could it be then that the XU4 got bricked in some way?

It makes absolutely no sense that it would have I/O errors considering that I’ve tried multiple uSD cards

Not sure if dmesg and journalctl logs would help, but I’ve attached them to this message.

dmesg.1.log (38.5 KB)
journalctl.1.log (65.0 KB)

[Joulinar]

hmm did you already tried a different os? Like Armbian? Odroid XU4 / HCx – Armbian
Would be interesting to see if it behaves differently.

[andoru]

Tried that now, gives me the same I/O error.

Tried again with multiple uSD cards and I get the same error message.

[Joulinar]

at least we could exclude DietPi as cause because you have same issue with plain Armbian. Which unfortunately leads to the conclusion that it is possibly a hardware error? Do you agree?

I mean, you could try one of the other options for the XU4 to see if that will change anything. https://wiki.odroid.com/odroid-xu4/os_images/os_images

[andoru]

Tried both Ubuntu and Android images provided by Hardkernel, and I got this:

ubuntu-18.04.3-4.14-minimal-odroid-xu4-20190910 → Booted this one without any errors, the XU4 connected to LAN with the correct IP according to DHCP, I could SSH into it. On the root partition I was able to create files, create folders, delete them, etc. As such I assume the partition is writeable.
On the screen I only had the TTY show up, no DE. Not sure if this was supposed to happen, where I downloaded the image, it was labeled as CLI. I expected to have the Mate desktop on it.
Giving it a restart from the SSH and after it booted got no mounting or I/O errors.

android-4.4.4-alpha-7.1-sd_installer-odroidxu4-20210228 → Also tried this one, at first it seemed like it wouldn’t boot, the blue LED flashed a few times, then it would be steady for a few seconds, then flash again. Left it for 5-10 minutes, and after that I saw the Android home screen. I don’t think this system comes with SSH by default, so I could not SSH into it, but the SBC did show up on the router panel with the correct IP set under DHCP.

So to me this sounds like the problem lies within DietPi and Armbian.

Anything else I could try? Could you provide me with an older version of DietPi for this device so I could test that?

[Joulinar]

in this case it might be related to kernel version. I guess we use the same one as Armbian because we don’t do own kernel development from our side.

Maybe @MichaIng has an idea?

[andoru]

Until @Michalng is able to provide a reply to this, would you (or anyone else) happen to have an older version image of the OS?

That way at least I could resume using the SBC until I can fix this issue.

[Joulinar]

hmm you could try the images from August

• https://dietpi.com/downloads/images/DietPi_OdroidXU4-ARMv7-Bookworm.7z
• https://dietpi.com/downloads/images/DietPi_OdroidXU4-ARMv7-Bullseye.7z

But probably you need to start them without network connection to avoid any kernel update during first install. But at least you can check if it is booting.