Boot issues on latest FW / Kernel with Odroid C4

Hi,
this morning dietpi prompted me with 4 updates. It turned out that these where a Firmware and kernel updates.
So I updated my system just to find out that half way there was an error fetching the update packages (404), after several retries all packages where downloaded and installed. but…

when rebooting my system it ‘hangs’ where only a cursor is shown.

I connected a monitor (running headless) and see that while booting everything starts okay, except for all (!) diepi services (starting with the ramdisk.service.

I am running dietpi from a USB disk where the boot parttition comes from the sdcard (odroid c4).
I wrote this setup here, so this is how I am using it: [Tutorial] Odroid C4 rootfs on USB drive

when doing a fresh install of dietpi on the sdcard everything works, but when changing the rootpartition in dietpiEnv.txt back to my USB drive, then all dietpi services fail.

I currently have no access to any log files (as the system doesn’t boot properly), how can I best troubleshoot this?

any help appreciated :slight_smile:

Ping @MichaIng

Thanks @Joulinar just tried to setup a new sdcard and usb diver followin my own (above linked) tutorial. This doesn’t work anymore, what normally used to happen was that the boot was waiting for 60 seconds (configured bootdelay) and then it would boot from the usb drive, now it boots without the delay (I think)…

Maybe Armbian changed something on kernel/firmware

I do not know, what I did find out is that having the root on an usb drive doesn’t work anymore, so now I have (by trial and error) moved directories from my usb drive back to the sdcard.

currently I have the following directories on the usb drive via a mount and symlinked directory:

  • home
  • srv

so /usr/ etc / var / root need to be on the sdcard as these are used before the usbdrive is mounting

So I now have a system that starts and I can login, but not all services are starting.
Trying to reinstall them (to see if that fixes anything gives me the following error:

APT update                                                                                                             
                          │  - Command: apt-get -y -eany update                                                                                    
                          │  - Exit code: 100                                                                                                      
                          │  - DietPi version: v9.1.1 (MichaIng/master) | HW_MODEL: 16 | HW_ARCH: 3 | DISTRO: 7                                    
                          │  - Error log:                                                                                                          
                          │ Hit:1 https://deb.debian.org/debian bookworm InRelease                                                                 
                          │ Get:2 https://deb.debian.org/debian bookworm-updates InRelease [55.4 kB]                                               
                          │ Get:3 https://deb.debian.org/debian-security bookworm-security InRelease [48.0 kB]                                     
                          │ Get:4 https://deb.debian.org/debian bookworm-backports InRelease [56.5 kB]                                             
                          │ Hit:5 https://repo.ebusd.eu/apt/default/bookworm bookworm InRelease                                                    
                          │ Err:1 https://deb.debian.org/debian bookworm InRelease                                                                 
                          │   The following signatures couldn't be verified because the public key is not available: NO_PUBKEY 0E98404D386FA1D9    
                          │ NO_PUBKEY 6ED0E7B82643E131 NO_PUBKEY F8D2585B8783D481

How do I get the correct keys as without them I can’t do anything…
Or am I missing something else… been working on this nonstop from this morning so my head is having trouble keeping up …

I wonder how you lost the PGP key for global Debian repository.

You can try following

sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv-keys 6ED0E7B82643E131
apt update 

one step further… still no joy… services will not start. when reinstalling them system is complaining about missing shared libs… calling it a day, shutdown now
tomorrow is another day…
Thanks for looking over my shoulder and for the help: really appreciate it!

Out of curiosity, will my firmware build work for you?

  • Put the firmware with: dd bs=512 seek=1 conv=notrunc,fsync if=odroid-c4/u-boot-meson.bin of=/dev/${entire-device-to-be-used} on a microSD card.

  • Put an unmodified OS image on a USB disk.

  • Boot up your ODROID-C4 with these prepared storages plugged in.

Hi,

i’ve the same issue and so i’ve tried your firmware, but it doesn’t help

The same errors appears:

Need to get 76.1 MB/121 MB of archives.
After this operation, 25.2 MB of additional disk space will be used.
Get:1 ..://xogium.performanceservers.nl/apt bookworm/main arm64 armbian-firmware all 24.2.1 [75.2 MB]
Ign:1 ..://xogium.performanceservers.nl/apt bookworm/main arm64 armbian-firmware all 24.2.1
Err:1 ..://apt.armbian.com bookworm/main arm64 armbian-firmware all 24.2.1
File has unexpected size (75168144 != 75168140). Mirror sync in progress? [IP: 198.140.141.60 443]
Err:2 ..://armbian.hosthatch.com/apt bookworm/main arm64 linux-dtb-current-meson64 arm64 24.2.1
404 Not Found [IP: 153.92.127.3 443]
Get:3 ..://mirrors.xtom.de/armbian bookworm/main arm64 linux-u-boot-odroidc4-current arm64 24.2.1 [762 kB]
Fetched 762 kB in 3s (262 kB/s)
E: Failed to fetch ..://apt.armbian.com/pool/main/a/armbian-firmware/armbian-firmware_24.2.1_all__1-SA6b6f-B90f5-R448a.deb File has unexpected size (75168144 != 75168140). Mirror sync in progress? [IP: 198.140.141.60 443]
E: Failed to fetch ..://armbian.hosthatch.com/apt/pool/main/l/linux-dtb-current-meson64/linux-dtb-current-meson64_24.2.1_arm64__6.6.16-Seb3e-D6b4a-P8ee5-C1d33H61a9-HK01ba-Ve377-B067e-R448a.deb 404 Not Found [IP: 153.92.127.3 443]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

I’m a new forum user and limited for link sharing, so i’ve replaced https with …

Thanks

Hi,

please be aware that now is working

thanks

simply use code fences. I updated your post above accordingly.

You have a different error, more like this Dietpi radxa zero package upgrade error

Okay,

so using my sdcard again copied all files / directories from my usb drive (that was working as rootfs, except /home and /srv (as these have terabytes of data) to the sdcard

mounted /home and /srv to two partitions om my usb drive: working correct

now I run into the following errors, the one below as an example:
/usr/sbin/apache2: error while loading shared libraries: libaprutil-1.so.0: cannot open shared object file: No such file or directory

So somewhere in the copy process something didn’t transfer okay, the files are all transfered, but I suspect a symlink is not there or not working correct?

All dietpi services start except for the software that is installed like openvpn, home-assistant, mosquito, mariadb, apache, etc. All fail will references to these libraries errors…

How can I best troubleshoot this?

can you check following

ls -la /usr/lib/aarch64-linux-gnu/ | grep libaprutil-1

Should looks like this

lrwxrwxrwx  1 root root       21 Feb  3  2023 libaprutil-1.so.0 -> libaprutil-1.so.0.6.3
-rw-r--r--  1 root root   201304 Feb  3  2023 libaprutil-1.so.0.6.3

nope… but it is on my backup disk. Backup from before the firmware and kernel update, so likely the halfway failing of the update ‘borked’ more then anticipated…

Will copy that whole directory from the backup disk and see what I run into next.
Thanks again @Joulinar appreciate your help!

Look here >>>> ヽ(•‿•)ノ

That is my doing a happy dance after two days sweating like I have never sweated before :upside_down_face:

Thanks @Joulinar for sticking with me on this one, will do some final checks to see if nothing else is broken / borked. Once done will do a donation to the project!

1 Like

APT package upgrades won’t update the bootloader, so at least that part did not change when the USB boot issue appeared.

@Ruud
How did you actually apply bootdelay? This is a U-Boot environment variable to make U-Boot starting the bootcmd delayed. However, /boot/dietpiEnv.txt is loaded and /boot/boot.scr executed as part of the bootcmd, hence applying bootdelay anywhere there should not have any effect anymore. You would need to apply it via U-Boot environment at defined drive offset, but last time I checked, this is actually entirely disabled in Armbian’s U-Boot builds, hence you cannot delay the execution of bootcmd with these bootloader builds at all.

Reading your guide, you actually applied the rootdelay kernel command-line parameter, which is a different thing, instructing the kernel to wait for the root device defined via root= parameter. This however should not make any difference, since our /boot/boot.scr hardcodes rootwait. This waits for the root device indefinitely instead and proceeds as fast as it appears. Not sure which one would take priority, but rootdelay should not be able to make anything better, but either is overridden, adds an unnecessarily long delay, or will make boot fail if the defined seconds were too short.

There was a significant kernel upgrade yesterday or two days ago: All “current” kernel builds were bumped from Linux 6.1 to Linux 6.6, at least in those cases where we did not install Linux 6.6 previously from builds hosted on our server. So quite possible that something broke due to this. There is usually no dedicated binary firmware involved in USB drive access, so I am 99% sure that it would be the kernel.

However, as I understand your last post, it does now actually work, i.e. there is no general issue with the kernel and rootfs on USB? Probably there was some voltage issue or other reason for some I/O error during the kernel upgrade initially, which broke something :thinking:.

@MichaIng I think I mixed up the term. I am using extraargs=rootdelay=60 net.ifnames=0 and that does what it says it does: wait (in this case) 60 seconds when booting.

So as a recap of what happened:
I logged into ssh and dietpi told me there where 4 updates that could be applied.
I ran apt upgrade but the new packages where unavailable because the repository server was ‘down’ > 404. Normally it then switches to a mirror so I kept trying the apt update / apt upgrade with no result.
I then did a apt-get upgrade and behold, the packages where downloaded and installed correct.

The issue occurred when rebooting my server: in the boot screen there where a lot of ‘failed’ messages ending in the display of only a blinking cursor: The server didn’t recover from this.

So this could be one of two things:

  1. The installation / update of the kernel / firmware packages didn’t fully work due to the repository server being down
  2. The new firmware / kernel didn’t support having the rootfs on the external usb drive

After putting on fresh underpants, I tried to install dietpi (new dietpi image downloaded), installed that on my sdcard, and that booted just fine. I then tried to follow my own instructions of moving the rootfs onto the usb drive and that failed the same way my server failed: a lot of errors and failed resulting in a blinking cursor.

So then I guessed that in the kernel moving the rootfs wasn’t supported anymore.

I recovered my server by moving all files back to the sdcard (basically reverting the rootfs change): trying (and succeeding) to keep as much as data / settings / configurations as possible.

My server is now running again but this time it runs from the sdcard, where my /home, /srv and /var/www (the data directories) are on usb drive partitions.

So it is working, not optimal (as i am not a big fan of sdcards . speed / wear / unstable) and would rather have all ‘moving’ parts on my usb drive.

So my guess is that indeed the 6.6 kernel was causing this issue and preventing having the rootfs on the usb drive.

And that’s exactly why I offered my firmware build. It doesn’t require a clunky bootflow and should work with the standard bootflow, even if it’s on a USB device.
So only the firmware needs to be on the microSD card, and since it doesn’t use a persistent environment, you could even switch the microSD card to read-only mode after the firmware has been copied to it. Yes, microSD cards have such a flag, but hardly anyone knows and uses it.
And your used bootflow could very well have been the cause of your problem, as the default update routines are probably not prepared to deal with a split partition environment that you have set up.
So I suggest, if you have a spare USB and microSD card, do some experiments with my firmware build and report on its result.

Hi @usual-user my first priority was to get my system back up with as minimal changes as possible: this server has been running production for some time now and is the center of a lot of things running in my private / work life.
So I didn’t have the time to test your proposed solution: sorry.

I currently do not have any spare (C4 / sdcard / etc.) lying around, but after this weeks adventure I came to realize how dependent I am on this server so will do some purchasing of spares. This will then also give me the opportunity to test other setups etc.

Anyway, thanks for thinking along, appreciate it!

Ah okay. If it adds the 60 seconds delay, then it should be just unnecessary: As said, the rootwait you should see in /boot/boot.scr means an infinite wait for the rootfs already, making rootdelay obsolete.

  1. If something with the download fails, the package would not extract in the first place, hence APT would throw errors and not even start replacing files. So the only thing where something could go wrong is when there are I/O errors when the files are replaced. I/O errors could also happen later, when the initramfs is generated, but similarly, it would result in an APT error, and the very last invoked script which replaces the /boot/initrd symlink would not run, hence there would be still a valid initramfs. So as long as there are no kernel errors thrown during the package upgrades, it is unlikely that something went wrong with its installation.
  2. The bootloader definitely supports loading the rootfs (boot scripts/env, kernel, dtb, initramfs) from anywhere, including USB. But the bootloader is not flashed on package upgrade anyway, hence did not change. Do you have some exact examples of those errors you see? It could be the initramfs not detecting or failing to mount the rootfs, before the kernel is launched, or the kernel at some point. Best would be of course the whole output taken from a serial console.

And did you check whether all files/links are there as intended? Do you still have this /mnt/sdcard mount and /mnt/sdcard/boot symlink (or bind mount), or do you in the meantime mount /boot directly? Check whether in /boot, you see the symlinks Image, uInitrd and dtb all pointing to the same new kernel version, matching the the kernel modules below /lib/modules.

In theory, power usage could have changed, or a higher peak caused in boot stage. So what could be tested is using a more powerful PSU or powering the USB drive with a dedicated PSU (if not the case already).

What exactly is not working with our bootloader in this regards? It supports all boot media, directory structures, filesystems etc just fine as well. I am not aware of anything which would prevent the setup Ruud uses, or any other with smaller or larger parts on the SD card (like bootloader binary and script and/or env only). And again, the bootloader has not been updated, neither has the boot script or environment been touched, hence it has already proven to work.

Ruud’s setup has the whole /boot directory on the SD card, hence it must not be R/O, otherwise kernel upgrades and initramfs rebuilds would fail. But of course with a different setup this can be done. I never found the time to start with this, but a tiny image with only bootloader and a FAT partition with a boot script which does nothing else then searching for and running another boot script, is what I am aiming for: USB boot | Provide tiny SD card image as USB boot relay · Issue #6186 · MichaIng/DietPi · GitHub
So the SD card never needs to be mounted, but just serves as minimal relay to boot any other attached image.

Now that I think about it, if I am not mistaken, it could even work to really only flash the bootloader binary to the SD card, and no partition table or filesystem at all. If I am not mistaken, the default U-Boot environment loops through all attached drives by itself, not only the one it was loaded from. Will test when I find time.

1 Like