Odroid N2 V8.24 doesn't boot

Troubleshooting
22

I can replicate it. Seems to be a regression with Linux 6.6:

[    9.988715] device offline error, dev sda, sector 1273856 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 2
[    9.994123] device offline error, dev sda, sector 626688 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 2
[    9.994198] Buffer I/O error on device sda1, logical block 158208
[   10.009654] Buffer I/O error on device sda1, logical block 77312
[   10.009658] device offline error, dev sda, sector 622608 op 0x1:(WRITE) flags 0x0 phys_seg 1 prio class 2
[   10.025164] Buffer I/O error on device sda1, logical block 76802
[   10.031359] device offline error, dev sda, sector 1056856 op 0x1:(WRITE) flags 0x9800 phys_seg 29 prio class 2
[   10.041938] Aborting journal on device sda1-8.
[   10.045534] device offline error, dev sda, sector 1056768 op 0x1:(WRITE) flags 0x9800 phys_seg 1 prio class 2
[   10.056190] Buffer I/O error on dev sda1, logical block 131072, lost sync page write
[   10.063887] JBD2: I/O error when updating journal superblock for sda1-8.
[   10.085168] EXT4-fs (sda1): shut down requested (2)

Iā€™ll check whether a more recent or older build from us works.

1 Like
23

Nasty, it does not work with any newer or older Linux 6.6 build, so this is broken since a longer time already. Interesting that no one else reported it. Maybe it also depends on the used USB device, though you tested multiple ones.

sudo apt-mark hold linux-image-current-meson64

Do the same with the device tree package:

sudo apt-mark hold linux-dtb-current-meson64

I guess the kernel is Linux 6.1 then?

uname -a

Those have been removed from Armbianā€™s APT repository already, so it is not easy to recover them. The ā€œlegacyā€ branch points to Linux 6.1, but it is disabled for Odroid N2. Iā€™ll re-enable it and do builds based on it, so we can test whether those work and in case at least have Linux 6.1 builds with more recent patch versions.

24

It would be very interesting, which kernel works on the older Armbian and DietPi v8.24 image. I just tested a Linux 6.1.92 build, and it fails the same way:

[    0.000000] Linux version 6.1.92-legacy-meson64 (armbian@next) (aarch64-linux-gnu-gcc (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0, GNU ld (GNU Binutils for Ubuntu) 2.38) #1 SMP PREEMPT Sat May 25 14:21:36 UTC 2024
...
[    7.852982] systemd[1]: Started systemd-journald.service - Journal Service.
[    9.011374] panfrost ffe40000.gpu: error -ENODEV: _opp_set_regulators: no regulator (mali) found
[    9.403354] device offline error, dev sda, sector 8192 op 0x1:(WRITE) flags 0x103000 phys_seg 1 prio class 2
[    9.408311] Buffer I/O error on dev sda1, logical block 0, lost async page write
...

It always happens ~0.5s after the panfrost error, but I donā€™t think it is related. Iā€™m currently doing another build with all Armbian kernel patches removed. If it still does not work, Iā€™ll try to disable the panfrost driver.

1 Like
25

Thank you for your response.

Itā€™s indeed strange that no one else has reported this issueā€¦ I guess no one has done a fresh install on Odroid since November :thinking:

Since the images are not bootable, Iā€™m unable to run the commands to hold the kernel and device tree package. When booting from an SD card, the system works fine even with the meson64 update, but the issue arises specifically with USB and HDD booting.

Given this, do you have any other recommendations for steps I can take, or should I wait for the new builds you mentioned? Iā€™ll be ready to test those once they are available and report back on whether they resolve the issue.

Thanks again for your support!

26

There were dozens of new Odroid N2 systems according to our statistics, maybe a hundreds when extrapolating to the numbers of users who do not participate the dietpi-survey. Most of them will however use eMMC or SD card, which do not have issues. And not everyone who tried and failed to boot form USB, does also report it here.

I just tested with Linux 6.1.92 with all Armbian patches removed. It still fails the same way. I am sure that it worked with Linux 6.1, when I implemented petitboot support. So unless it is a specific newly added driver, it is maybe a mainline Linux change, which was backported to Linux 6.1 as well.

EDIT: Blacklisting panfrost doesnā€™t help either, so this is unrelated.

27

I came across something in the release notes of a recent Home Assistant update that might be related:

:warning: For ODROID-N2 (and Home Assistant Blue), this release reverts the U-Boot update which is believed to be the cause of some intermittent issues when booting from eMMC. If you had any trouble previously after an update to OS 11.5 or newer, please see #3227 and help us with tracing this issue. Also note that some new eMMC modules with Kingston chips are not supported yet - this is tracked in #3250.

Additionally, I have a functional backup that I would like to restore. However, since I cannot boot the system with the current images, Iā€™m unable to reach the dietpi-restore tool. Is there a way to transform this backup into a bootable ISO so I can restore my system?

Thanks again for your support!

28

In our case, however, booting from eMMC and SD card works fine. The issue here is only when booting from USB (at least on my tests). And it appeared after a kernel upgrade, which does not imply a U-Boot upgrade. And it happened with petitboot on the SPI storage in my case, so U-Boot is basically ruled out.

Also reviewing the linked issues, HA OS indeed fails at U-Boot stage, while in our case, when booting from USB, it fails much later in kernel stage, when the init system is starting to cause load on the drive.

So nope, not related.

That is awesome. Which kernel version is it? Please check the content of the data/boot directory of the backup, when attaching the drive to another Linux system. From there, you can also copy the old/functional kernel version back in place. Iā€™ll give you further instructions when you found it.

29

Thank you for clarifying the situation regarding U-Boot and kernel stages.

Regarding the functional backup, I checked the content of the data/boot directory on another Linux system. The kernel version appears to be 6.1.50, based on the filenames:

  • vmlinuz-6.1.50-current-meson64
  • uInitrd-6.1.50-current-meson64

What steps should I take next to copy this old/functional kernel version back in place? Any guidance on this process would be greatly appreciated.

Thanks again for your support!

30

Okay do the following: Copy all files directories from /boot with 6.1.50-current-meson64 ending, as well as the /lib/modules/6.1.50-current-meson64 directory to respective dirs on the USB drive. Then there are 3 symlinks in /boot, which are pointing to vmlinuz-*, dtb-* and uInitrd-*. Re-create them to point to the respective *-6.1.50-current-meson64 variants, like so:

cd /mnt/usb/boot
rm Image uInitrd dtb
ln -s vmlinuz-6.1.50-current-meson64 Image
ln -s uInitrd-6.1.50-current-meson64 uInitrd
ln -s dtb-6.1.50-current-meson64 dtb
31

Interesting, I generated a build based on Linux 6.1.50, and indeed it works fine here as well: meson64 legacy: downgrade to Linux 6.1.50 Ā· MichaIng/build@8e1b6f4 Ā· GitHub
Linux 6.1.92 did not work. No other changes. So it is related to an upstream change. Iā€™ll test some versions in between to narrow it down.

EDIT: I spoke too soon. After another reboot, the same error shows up again, and as well the next 4 power cycles. Seems to be some race condition which succeeds in rare cases but fails most of the time. But pretty weird that even with the Linux version which matches the functional one from your backup, it does not work now. I did also remove all Armbian patches, to rule that out. Doing another build with Armbian patches back in. But I have the feeling that we cannot find a working state anymore ā€¦

For completeness, here the matching Armbian forum topic: https://forum.armbian.com/topic/32593-startups-fail-after-root-file-system-moved-to-ssd/
Blacklisting UAS does not help here either. Makes sense since it is a USB stick which does not support UAS anyway.

EDIT2: Stranger, now it works, respectively it worked for all 4 reboots I did: I did another build with same kernel version but Armbian patches re-applied. Additionally, not sure whether it makes a difference, I flashed a U-Boot build with one Armbian patch removed, which switched the boot order to try booting from USB first. This causes issues when booting from SD card or eMMC in some cases, where USB devices take too long to initialize, resulting in a reset when trying to read from them: Odroid N2 Plus: Boot loop when attached USB storage NOT ready Ā· Issue #7168 Ā· MichaIng/DietPi Ā· GitHub
This could indirectly have an effect here, since the USB device is loaded a little later in bootloader stage. Maybe loading it too early works for our case with the bootable USB sticks, since obviously kernel and init system load well, but maybe leaves it sometimes in a different state or what, so that we run into this disconnect later.

But what does not fit into the picture, is that I ran into the same error when booting with petitboot, which tries eMMC/SD card first, and generally takes MUCH longer, since it is a two stage boot process, first loading petitboot, which then loads an embedded U-Boot in a 2nd stage. And of course it does work for you with your backup Linux 6.1.50, without the (un)patched bootloader.

Iā€™m doing another kernel build with latest Linux 6.1.y again. If this works as well, try it with Linux 6.6. Letā€™s see.