Dietpi is running fine
It’s the u-boot that causes problems. Somehow related maybe: [Tutorial] Odroid C4 rootfs on USB drive [deprecated as of kernel 6.6] - #29 by pyavitz
Sorry, not used to forum rules yet…
I am using a Nanopi Neo3 as a home server for about a year now.
In different phases of “learning on the job” I created a configuration where the rootfs is on a usb drive. I used this item:
to create that configuration.
But some time ago the warning for kernel >6.6 appeared.
I don’t know if I am lucky or that this warning is only ment for Odroid C4 boards, but as far I know my system is still running OK:
root@DietPi:~# uname -a
Linux DietPi 6.6.34-current-rockchip64 #2 SMP PREEMPT Sun Jun 16 11:47:49 UTC 2024 aarch64 GNU/Linux
and:
root@DietPi:~# df -h
Filesystem Size Used Avail Use% Mounted on
udev 918M 0 918M 0% /dev
tmpfs 198M 1.1M 197M 1% /run
/dev/sda1 20G 1.2G 18G 6% /
tmpfs 990M 0 990M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 990M 0 990M 0% /tmp
/dev/sda2 200G 24G 166G 13% /mnt/data
/dev/mmcblk0p1 30G 745M 28G 3% /mnt/sdcard
overlay 200G 24G 166G 13% /mnt/data/docker-data/overlay2/6a754abd0709679f86d040774399b5647a8806cfb456d7d09136e3b55eaeb8d0/merged
Because I don’t like unpleasant surprises I tried to create a different solution as sort of “explained” here: [Tutorial] Odroid C4 rootfs on USB drive [deprecated as of kernel 6.6] - #29 by pyavitz
I call this the “pyatz” method…
A new phase of “learning on the job” enrolled itself. I re-learned a lot about u-boot etc. but after a lot of trial an error I concluded in the end that probably due to incomplete u-boot implementations it appeared to be impossible to use this “pyatz method”.
I already knew that the proper DTB file for the Nanopi neo3 was not used in the u-boot. Instead it uses the NanoPi R2S DTB.
At first I thought that this would be the cause why I could not use the USB3 device for the rootfs, but I could not verify that. (NP RS2 has no USB3 interface…)
Because in the u-boot tty log I could see that 3 interfaces for USB where used:
starting USB…
Bus usb@ff5c0000: USB EHCI 1.00
Bus usb@ff5d0000: USB OHCI 1.0
Bus usb@ff580000: USB DWC2
The 2nd interface @ff5d0000 is the USB3 interface. But no storage-device is recognised by u-boot when connected to that interface.
However when I connected the USB drive with a valid Dietpi image on it to the ff5c0000: USB EHCI 1.00 (USB2 0) interface the “pyatz” method worked as expected.
Dietpi booted and ran flawlessly from this USB 2.0 interface.
When I tried the same with the USB drive connected on the other USB 2.0 interface (with the DWG2 protocol) the system booted, but derailed with the following u-boot output:
–many previous messages deleted
Begin: Running /scripts/local-block … done.
done.
Gave up waiting for root file system device. Common problems:
- Boot args (cat /proc/cmdline)
- Check rootdelay= (did the system wait long enough?)
- Missing modules (cat /proc/modules; ls /dev)
ALERT! UUID=8a079158-7be5-41b9-afa6-52248797b314 does not exist. Dropping to a shell!
(initramfs)
I had a shell available at this point but did not investigate this any further because I wanted to be able to boot from the USB OHCI (usb3) interface.
And that appeared to be impossible thus far.
It may possible that the difference in behaviour is because of differences between “host” or “OTG” usb, but I have no idea what those differences are and why they would cause this. I suspect that the difference might be caused by the device ID assigned by u-boot of the interfaces. Something pyatz mentions.
From this I assumed the following:
Because the “pyatz” method worked flawlessly from the USB EHCI interface and partially from the usb DWG2 interface the OHCI implementation of the current u-boot image based on the NanoPi R2S DTB is flawed.
In an attempt to improve things I managed to have the u-boot to use the nanopi neo3 dtb using the boot.scr, but that made no difference. Still nothing detected on the @ff5d0000: USB OHCI 1.0 USB3 interface. Also using a usb stick instead of a USB drive or use a self powered USB drive did no make any difference.
I may be able to modify the DTB ( or actually the DTS file) when I knew what to modify. And then create a new u-boot. But I have no clue what to modify.
I also looked in the DTB’s from other arm boards using USB3 and could not find any lead that would explain why the current u-boot does not recognise a device on the interface. Apparently the OHCI driver is loaded. I have no idea if it is working as it should. But it is not missing. Only not function properly or not as expected or otherwise.
Searching the internet for u-boot USB OHCI/ USB3 problems showed nothing that I could use or even recognised remotely as a similar problem.
It seems that I have stumbled on the hardware/software interface of this particular board and information about this is scarce and comprehensible information about the connection between the 2 layers is nowhere to be found. I know that basically it is not complicated. A map of hardware specs is linked to software functions. But I only found a multitude of versions of the Nanopi Neo 2 DTB and variants floating around. Some filled with hardcoded content, some with more modular contents importing other *.c or *.h files etc.
I have written similar software years ago but with so many different versions I got stuck finding out which version to use as a start. If that would even be necessary.
To summarise it all:
- I would like to be able to use the “pyatz” method: Put only u-boot on SD an boot and run rootfs from the USB3 interface of the Nanopi Neo3. Just the same as when the USB drive is connected on the USB 2.0 interface with the EHCI protocol…
- I have not mentioned the usb quirks issue yet but I assume that this may cause problems, but not that a storage device is not detected at all…
Any help appreciated!
I use this method…boot from the SD card (the /boot partition [can use a cheap smaller SD card…only used for the uboot/boot partition]), but the entire OS / is on an external USB drive that can read and write faster and has MUCH larger storage capability
That’s essentially what I want. And to be more specific Using the USB3.0 interface. Because if I use the USB2.0 interface it works OK. I have seen the solution you refer to in my search but dismissed it because I have a NanoPi and not a Raspberry pi.
But when it will work on other architectures I can try it out of course. Can anybody confirm this? Does it work on non Raspberry SBC’s?
Thanks!
Here is my OrangePiZero2 w/ armbian
ssh warhawk@192.168.0.107
___ ____ _ _____ ____
/ _ \| _ \(_) |__ /___ _ __ ___|___ \
| | | | |_) | | / // _ \ '__/ _ \ __) |
| |_| | __/| | / /| __/ | | (_) / __/
\___/|_| |_| /____\___|_| \___/_____|
Welcome to Armbian 24.5.1 Jammy with Linux 6.6.36-current-sunxi64
System load: 4% Up time: 6 days 15:04
Memory usage: 33% of 918M Zram usage: 49% of 459M IP: 172.21.0.1 172.20.0.1 172.19.0.1 172.17.0.1 192.168.0.107
CPU temp: 48°C Usage of /: 28% of 58G
storage/: 2% of 932G storage temp: 41°C
RX today: 331.0 MiB
cat /etc/fstab
# <file system> <mount point> <type> <options> <dump> <pass>
tmpfs /tmp tmpfs defaults,nosuid 0 0
UUID=b5110915-cb65-4723-8ada-a154419be620 /media/mmcboot ext4 defaults,noatime,commit=600,errors=remount-ro,x-gvfs-hide 0 1
/media/mmcboot/boot /boot none bind 0 0
UUID=ff497056-7e4a-4bc8-af35-52e785417fb1 / btrfs defaults,noatime,commit=600,compress=lzo,x-gvfs-hide 0 2
UUID=a3eea7d7-c51e-41af-8f83-e4ac218483de /home/warhawk/hdd btrfs defaults,noatime,commit=600,compress=lzo,x-gvfs-hide 0 0
lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 0 931.5G 0 disk
└─sda1 8:1 0 931.5G 0 part /home/warhawk/hdd
sdb 8:16 1 57.3G 0 disk
└─sdb1 8:17 1 57.3G 0 part /var/log.hdd
/
mtdblock0 31:0 0 2M 0 disk
mmcblk0 179:0 0 7.3G 0 disk
└─mmcblk0p1 179:1 0 7.1G 0 part /boot
/media/mmcboot
zram0 252:0 0 459.4M 0 disk [SWAP]
zram1 252:1 0 150M 0 disk /var/log
zram2 252:2 0 0B 0 disk
The OpiZero2 only has USB2.0 though…
Thanks for your info on this.
Your config seems similar to what I have on my NPNeo3.
root@DietPi:~# cat /etc/fstab
- some lines deleted
#----------------------------------------------------------------PHYSICAL DRIVES
#----------------------------------------------------------------
UUID=a1e61736-4963-444e-8fdd-9cf78dab8eff / ext4 noatime,lazytime,rw 0 1
UUID=55e73fb1-a348-4bcf-81e2-186c4145a937 /mnt/sdcard ext4 noatime,lazytime,rw,nofail,noauto,x-systemd.automount
UUID=73d03cd2-248f-4987-861e-47227f602d37 /mnt/data auto defaults 0 1root@DietPi:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 0 223.6G 0 disk
|-sda1 8:1 0 20G 0 part /
-sda2 8:2 0 203.6G 0 part /mnt/data mmcblk0 179:0 0 29.7G 0 disk-mmcblk0p1 179:1 0 29.7G 0 part /mnt/sdcardroot@DietPi:~# ls -l /
total 56
lrwxrwxrwx 1 root root 7 Sep 24 2023 bin → usr/bin
lrwxrwxrwx 1 root root 16 Sep 30 2023 boot → /mnt/sdcard/boot
drwxr-xr-x 15 root root 3400 Jul 24 19:13 dev
- some lines deleted
The differences between your and mine configuration seem minor:
In your config a bind mount is used, I mine it is a link. But the result is functionally the same I guess.
These difference have been discussed here: [Tutorial] Odroid C4 rootfs on USB drive [deprecated as of kernel 6.6] - #2 by Ruud
What I tried to realize using the “pyatz” method:
so that there is no need for any reference to the SD card any more.
As can be seen from the setup on my 2nd NPNeo3, with a usb drive connected to the USB 2.0 interface:
root@DietPi:~# cat /etc/fstab
- some lines deleted
#----------------------------------------------------------------PHYSICAL DRIVES
#----------------------------------------------------------------
UUID=8a079158-7be5-41b9-afa6-52248797b314 / ext4 noatime,lazytime,rw 0 1
root@DietPi:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
sda 8:0 0 29.8G 0 disk
└─sda1 8:1 0 6.7G 0 part /
mmcblk0 179:0 0 14.5G 0 disk
└─mmcblk0p1 179:1 0 14.5G 0 part
root@DietPi:~# ls -l /
total 8816
lrwxrwxrwx 1 root root 7 Jun 10 18:04 bin → usr/bin
drwxr-xr-x 4 root root 4096 Jul 21 10:23 boot
-rwxr-xr-x 1 root root 138 Jul 19 11:46 checknode.sh
drwxr-xr-x 15 root root 3380 Jul 23 11:54 dev
drwxr-xr-x 58 root root 4096 Jul 18 20:09 etc
drwxr-xr-x 3 root root 4096 Jun 10 18:16 home
- some lines deleted
As you can see : there is no reference any more to files or directories on the SD card, so I assume that whatever update will be performed on this filesystem including possible modification of the content of the /boot directory ( e.g. updated DTB or new items in config files) will function as intended.
Both the bind or link to the /boot directory on the SD card try to functionally do the same I guess. But you never know beforehand if all changes will function as expected.
Ruud marked his tutorial as depreciated because it does not work with a kernel > 6.6
I have no problems thus far using a kernel > 6.6 but as said before: I don’t like unpleasant surprises.
With the “pyatz” method you only need to have a SBC specific u-boot version on the SD image to boot from. And all config info coming from the USB device.
So you are not dependant any more of any info on the SD drive.
And it works with the USB 2.0 interface, but not with the USB3.0 interface.
I suspect that the use of incomplete coding used in the u-boot source is causing this. But I cannot verify this.
I got lost in the “woods” so to speak. I found many different u-boot sources for the nanopi neo2/3 family, but no apparent differences between USB2.0 and USB3.0 DTB’s that could explain why there is a difference between using a USB 2.0 interface or a USB3.0 interface. I know that one is using the EHCI protocol (USB2) and the other OHCI. But what I found about that protocol was that it is quite common so I would not expect it not to function properly.
That leaves only a few options:
- it is not implemented properly
- My expectations about how it should work are not correct
- Additional functionality is needed for it to operate properly in de u-boot environment
- Something else?
So still hoping that someone can shed a light on this…
But I might be wrong on that of course…
I found some additional information I think:
I also have an Orangpi 3 LTS, it has USB3.0 interfaces and when I connect the same USB drive to one of those interfaces in the u-boot terminal log I see this:
1 Hub (5 Gb/s, 0mA)
| U-Boot XHCI Host Controller
And more important: the drive is recognised as a storage device
I have never seen this on the nanopi neo3.
Whatever I connected to the USB3.0 interface, nothing was recognised.
Also: the usb interface was shown, but without the proper specifications of the speed.
So my belief that the u-boot for the neo3 is incorrect or incomplete becomes more likely
Unfortunately no simple solution using the nanopi neo3 DTB instead of the neo2 rs DTB. It made no difference for the USB divices.
Even if the u-boot is limited to USB2.0 OHCI, when it then loads the kernel from the usb drive with usb2.0 speed getting the XHCI driver avaiable for the higher usb3.0 speed would be ok for me. That’s what I see all the time when a debian or other linux kernel is loaded.
To be continued…
The reason I like the /boot on the SD and the root / on USB/external is that if the main root partition get’s borked…I can still boot into minimal and do a fsck on the borked drive…but I see what you are saying…minimal uboot to get things rolling, then everything runs from the external drive at faster (even USB2.0 is faster than most SD card speeds)
Plus with booting from a cheapo SD, the system ALWAYS starts…for some reason if I put the /boot on a EMMC on occasion it gets borked and I have to use a SD OS to fsck the partitions to fix em.
It would be nice to be able to use the entire SD card for the /boot partition once the root / is copied/moved over to external though…but as long as uboot and the starting kernel is small enough I guess it doesn’t matter
I think I solved the problem myself.
After a lot of trial, error, half baked solutions, modifying device trees, configs etc I almost gave up. In a final effort I decided to compile a version of u-boot myself from: Build U-Boot — Das U-Boot unknown version documentation
First I installed a Ubuntu jammy VM in Virtualbox to prevent cluttering my normal system, also running Ubuntu jammy.
Then in that VM I followed these steps:
I installed the needed dependencies: Building with GCC — Das U-Boot unknown version documentation
and additional packages.
Then I followed: ROCKCHIP — Das U-Boot unknown version documentation
with:
git clone --depth 1 https://github.com/ARM-software/arm-trusted-firmware.git
cd arm-trusted-firmware
make realclean
make CROSS_COMPILE=aarch64-linux-gnu- PLAT=rk3328
cd ..
then: ROCKCHIP — Das U-Boot unknown version documentation
git clone --depth 1 https://source.denx.de/u-boot/u-boot.git
cd u-boot
and to build rk3328 boards:
export BL31=../arm-trusted-firmware/build/rk3328/release/bl31/bl31.elf
make nanopi-r2s-rk3328_defconfig
make CROSS_COMPILE=aarch64-linux-gnu-
Mind, I changed the PLAT and defconfig name to the one I would expect to be the proper one for my u-boot. I know there does not exist a specific nanopi-neo3 defconfig.
Then I copied the u-boot to the SD card: ROCKCHIP — Das U-Boot unknown version documentation
with:
sudo dd if=u-boot-rockchip.bin of=/dev/sda seek=64
sync
That was all!!
I put this SD in my npn3 board with a working dietpi image installed on a USB ssd connected with the usb3 port.
I booted this and the USB ssd was used to boot from.
I have not investigated in detail if there are things missing, but from what I could see from the u-boot console and from dmesg was that the proper dtb is used by the kernel, Machine model: FriendlyElec NanoPi NEO3, and also the information from /boot/dietpiEnv.txt is passed to the kernel cmdline.
Altogether this u-boot does not even use the /boot/boot.cmd script with the usb-boot script, the method I tried to use originally, to boot from usb. For some reason the SD partition is not recognised or set properly from within this u-boot version.
But for me that does not matter. The u-boot scans for bootable devices and recognises the USB ssd with the Dietpi rootfs and boots from it.
For now the problem seems to be solved.
Some additional information:
I have a few types of ssd/usb3 converters. All cheap stuff from alixpress. One type uses a jmicron controller. The other one I forgot. Both use different USB3 connectors. De unknown type uses a usb-c connector, the jmicron device uses a usb3 a connector (rectangle format). I discovered that not every combination cabling/converter works reliably. To my surprise when I connected the jmicron device directly to the usb3 connector of the nanopi the boot from usb runs, but there are a lot of retries and sometime it even takes to long and a successful boot fails. When I connected the same device using a short usb3 extension cable it boots flawlessly and I can verify the speed is indeed high with the command: lsusb -t
Port 1: Dev 2, If 0, Class=Mass Storage, Driver=uas, 5000M
This is not always the case, many times the speed is only 480M, so just usb2 speed.
Most times you won’t notice that difference, but it seems that it is related to the quality of the connection cable used. Why it performs best with an extension cable added, I really don’t know.
I have encountered similar problems with the other device that is using a usb-c connector on the device. Many cheap usb-c cables don’t function at all, but I have a few that finally boot, but only at a low speed. So in order to get the best performance: check out various cables and verify the indicated speed…
This topic was automatically closed 90 days after the last reply. New replies are no longer allowed.