Odroid XU4 - Kernel panic after update

Hello,
First off all thanks so much for these distribution!
In the past I builded 2 nas with Odroid XU4 and DietPI. They works very well since few years and this morning I saw a new update. After apt update, I reboot and nothing more… I connected a screen and I discovered the logs with “end Kernel panic - not syncing: VFS: Unable to mount root fs unknow-block(0,0)”.
I rebuilded a new card with the last version and the XU4 restart but with the new installation there is an automatic update then the system reboot and I obtained the same error message…
What can I do??

1 Like

Hi! The exact same thing just happened to me. With the new update on an XU4.

Can you have a look @MichaIng

Same here!
Did an update on Odroid HC1 and no boot anymore!

Fortunately found a working backup:

visible UART-output after buggy update NO BOOT ANYMORE

and by comparsion:

UART-output before buggy update BOOT OK

I assume the problem is here:

[    4.879383] Disabling rootwait; root= is invalid.
[    4.883718] RAMDISK: Couldn't find valid RAM disk image starting at 0.
[    4.891991] /dev/root: Can't open blockdev
[    4.894661] VFS: Cannot open root device "UUID=a3af1683-ebad-4def-b14b-3bdb314411d2" or unknown-block(0,0): error -6

Not even a recovery shell is available - can do absolutely nothing…

For reference a related GitHub issue New install on odroid X4U (HC2) fails to boot after first start · Issue #7341 · MichaIng/DietPi · GitHub

Probably @MichaIng will answer there when the problem has been looked at.

1 Like

Ok, thanks! Looking forward to a solution to the problem. Regards!

I took the new kernel off the APT repo for now. It seems to have some issues with the initramfs. The root=UUID support is something the initramfs should provide, and apart of the fact that it detects this as invalid, there is another difference in the log:

[    1.861869] Trying to unpack rootfs image as initramfs...
[    1.862068] rootfs image is not initramfs (invalid magic at start of compressed archive); looks like an initrd

The 2nd line is not present in the successful boot.

The kernel upgrade was minor only, hence I need to check possible config changes at Armbian end. Looks like we strictly need to test even minor kernel upgrades manually first.

That kernel contained only Hardkernel updates, and there were no changes Armbian end or in the config etc. The nighly smoke tests at Armbian fail since a while, so we do not see whether it worked their end: Smoke tests (cronjob) · Workflow runs · armbian/os · GitHub
… it doesn’t: Debian Trixie brick after kernel update - Odroid XU4 - Armbian Community Forums

@GerryK as you seem to have the failing image with the new kernel on a dedicated SD card. Could you check/compare the content of old and new initramfs?

lsinitramfs /boot/initrd-*

Maybe there is a way to mount the failing image to the Odroid XU4 as well, or to another Linux system.

And does someone have a log from the kernel upgrade itself, respectively the initramfs generation?

I have access to my Odroid XU4 only after 2 weeks.

Hello;
I provide the requested logs:

dietpi with 6.6.43-current-odroidxu4:
# lsinitramfs /boot/initrd*

broken dietpi with 6.6.68-current-odroidxu4
# lsinitramfs /boot/initrd*

1 Like

My goodness, I may have found it: conf/conf.d/zz-resume-auto at DuckDuckGo
And I even saw a changelog entry somewhere within the last days, that support for resume from hibernation was added, but I don’t remember where exactly, as I did not pay attention for this, which seemed unrelated to what I was actually looking for.

@GerryK can you show:

ls -l /etc/initramfs-tools/conf.d
ls -l /etc/initramfs-tools/hooks
cat /etc/initramfs-tools/conf.d/zz-resume-auto

There should be a zz-resume-auto inside (or maybe just a resume). The conf/conf.d/zz-resume-auto within the initramfs is generated by mkinitramfs, if its config defines the RESUME variable, which should have a value like UUID=..., defining a swap partition, used for suspent-to-disk. We have no swap partition, hence this cannot work anyway like that, so that file should not exist in the first place. But that RESUME variable is not defined in the default config, nor passed from /etc/kernel/postinst.d/initramfs-tools, nor set in Armbian’s kernel package maintainer script. So I am puzzled where it is suddenly coming from. Also I could not find the changelog or what it was anymore, where something with enabling hibernation support was mentioned. No idea if it even was related to the Odroid XU4 at all, as I was checking a lot of changelogs today, for various other SBCs :thinking:. Maybe just a coincidence. Also weird that the vast amount of finds for this configs are about a syntax error in the hook script, which looks more like an old bug in initramfs-tools to me, which should be fixed long ago. But that this appears for various platforms during a long range of years is strange.

Many questions, let’s try to get to the bottom of this.

Hello;
Seems like none of the directories / files does exist:

On update-failed dietpi bookworm sd-card:

ls -l etc/initramfs-tools/conf.d
total 0

ls -l etc/initramfs-tools/hooks
total 0

cat etc/initramfs-tools/conf.d/zz-resume-auto
cat: etc/initramfs-tools/conf.d/zz-resume-auto: No such file or directory

Can you check the content of this zz-resume-auto within the initramfs?

mkdir /boot/initramfs
unmkinitramfs /boot/initrd-* /boot/initramfs
cat /boot/initramfs/conf/conf.d/zz-resume-auto

Searching around, this should work to disable its generation. Let’s hope it allows this system to reboot:

echo 'RESUME=none' > /etc/initramfs-tools/conf.d/resume 
update-initramfs -u

Hi,
Sorry I can’t test because I rebuilded a new card.

Regards.

Okay no problem. If you or anyone else finds time and mood to try it again, to investigate this further, it would be great. Otherwise I can do so from January 17th on.

It can cause the kernel panic again, so at best do with a test system, or have another Linux system where you can insert the SD card to downgrade the kernel. This is possible by downloading the older/functional kernel package and use dpkg -i --root=/path/to/mount ..., I can give detailed instructions, if needed.

To test/investigate:

lsinitramfs /boot/initrd-*
# verify the current/functional initramfs does not contain conf/conf.d/zz-resume-auto
update-initramfs -u
lsinitramfs /boot/initrd-*
# verify it does neither when freshly re-generating it on up-to-date system
cd /tmp
wget https://dietpi.com/downloads/binaries/linux-{dtb,image}-current-odroidxu4.deb
dpkg -i linux-{dtb,image}-current-odroidxu4.deb
# the part of the output where the initramfs is generated would be interesting
lsinitramfs /boot/initrd-*
# check whether it now contains conf/conf.d/zz-resume-auto
mkdir /boot/initramfs
unmkinitramfs /boot/initrd-* /boot/initramfs
cat /boot/initramfs/conf/conf.d/zz-resume-auto
# see content of that script, since there is actually nothing "correct" it could contain
echo 'RESUME=none' > /etc/initramfs-tools/conf.d/resume 
update-initramfs -u
lsinitramfs /boot/initrd-*
# see whether above initramfs-tools config prevents it from generating the faulty script
reboot
# and whether this solves the kernel panic