NanoPi R5C eth0 and eth1 swapping

Troubleshooting
1

Creating a bug report/issue

I have searched the existing open and closed issues

Required Information

  • DietPi version | 9.19.2
  • Distro version | trixie
  • Kernel version | Linux 6.12.57-current-rockchip64 #1 SMP PREEMPT Sun Nov 2 13:15:23 UTC 2025 aarch64 GNU/Linux
  • Architecture | arm64
  • SBC model | NanoPi R5C
  • Power supply used | Meanwell HDR-15-5 5V 2.4A
  • SD card used | N/A - Onboard eMMC

Additional Information (if applicable)

  • Software title | N/A
  • Was the software title installed freshly or updated/migrated? N/A
  • Can this issue be replicated on a fresh installation of DietPi? Yes

Steps to reproduce

  1. Install DietPi image based off Debian Trixie
  2. Reboot PC
  3. Ports randomly change without modifying anything in /etc/networking/
  4. Have to physically swap LAN and WAN ports to get correct behaviour for eth0 and eh1

Expected behaviour

eth0 should be LAN
eth1 should be WAN

Actual behaviour

eth0 is sometimes WAN
eth1 is sometimes LAN

Extra details

# Stable Bullseye based DietPi v9.7.1 with Kernel 5.10.160 on NanoPi R5C

# Physical port = LAN
$ readlink -f /sys/class/net/eth0/device/
/sys/devices/platform/3c0400000.pcie/pci0001:00/0001:00:00.0/0001:01:00.0

# Physical port = WAN
$ readlink -f /sys/class/net/eth1/device/
/sys/devices/platform/3c0800000.pcie/pci0002:00/0002:00:00.0/0002:01:00.0

# Unstalbe Trixe based DietPi v9.19.2 with Kernel 6.12.57-current-rockchip64 on NanoPi R5C

# Physical port = WAN
$ readlink -f /sys/class/net/eth0/device/
/sys/devices/platform/3c0800000.pcie/pci0002:00/0002:00:00.0/0002:01:00.0

# Physical port = LAN
$ readlink -f /sys/class/net/eth1/device/
/sys/devices/platform/3c0400000.pcie/pci0001:00/0001:00:00.0/0001:01:00.0

# Both device drivers on both systems are the same

$ readlink -f /sys/class/net/eth0/device/driver
/sys/bus/pci/drivers/r8169
$ readlink -f /sys/class/net/eth1/device/driver
/sys/bus/pci/drivers/r8169

$ cat /etc/network/interfaces.d/eth0.cnf 
# LAN interface (static)
# Physical port: eth0
auto eth0
iface eth0 inet static
    address 192.168.100.1
    netmask 255.255.255.0

$ cat /etc/network/interfaces.d/eth1.cnf 
# WAN interface (DHCP)
# Physical port: eth1
allow-hotplug eth1
iface eth1 inet manual
    pre-up sleep 3
    post-up dhclient -nw eth1 || true

I have the LAN port setup with a static IP and the WAN port using DHCP to get an IP from the local network. Same setup on both devices yet the Trixie PC currently is getting its DHCP lease from the LAN port and serving out a staitc IP on the WAN port.

After seeing the ports inexplicably switch I tried the following udev rules which seemed to make no difference:

$ cat /etc/systemd/network/10-nanopi-r5c-lan.link
[Match]
Path=platform-3c0400000*.pcie

[Link]
Name=eth1

$ cat > /etc/systemd/network/11-nanopi-r5c-wan.link
[Match]
Path=platform-3c0800000*.pcie

[Link]
Name=eth0

I found some useful information from How to associate eth0/1/2 with Lan1/2/3 but even after udev rules the issue persists.

eth0 should map to the physical LAN port as suggested by MichaIng in Nanopi R5C led lights not staying on and it did in bullseye and bookworm but trixie is not working correctly.

Could be related to a similar issue on a NanoPi R76S: NanoPi R76S - LAN ports switching on boot · Issue #7844 · MichaIng/DietPi · GitHub

Any help would be greatly appreciated!

2

In other cases, the one of the LAN ports is not available at all by times. There is some issue with the PCIe implementation in mainline Linux for the RK356x boards, which I am investigating since some months (whenever I find time) but without any progress so far.

udev rules and other interface renaming can work only partially, since the kernel device name cannot be changed, so some inconsistency remains at best. But we do that on the Orange Pi 5 Plus already. A single rule however cannot work reliably, since e.g. eth1 cannot be renamed to eth0 if eth1 exists already. So it needs to be more complex:

cat << '_EOF_' > /etc/udev/rules.d/99-dietpi-nanopir5c.rules
SUBSYSTEM=="net", KERNEL=="eth0", KERNELS=="0002:01:00.0", RUN:="/bin/true"
SUBSYSTEM=="net", KERNEL=="eth1", KERNELS=="0001:01:00.0", NAME="to_eth0", RUN:="/bin/true"
SUBSYSTEM=="net", KERNEL=="to_eth0", RUN="/bin/ip l s dev eth0 name eth1", RUN+="/bin/ip l s dev to_eth0 name eth0", RUN+="/bin/udevadm trigger -c add /sys/class/net/eth0 /sys/class/net/eth1"
_EOF_

So 0001:01:00.0 is the LAN port and 0002:01:00.0 is the WAN port, right? Otherwise the two identifiers need to be switched. What the rules do:

  • If the LAN port is detected as eth1 (hence after the WAN port), it is renamed to to_eth0 in a first step, since eth0 is in use by the WAN port already.
  • Once to_eth0 is detected, hence eth1 is not in use anymore, the WAN port is first renamed from eth0 to eth1, then the LAN port is renamed from to_eth0 to eth0.
  • The RUN:="/bin/true" is needed to assure no other udev rules run commands for the still falsely named interfaces. The to_eth0 rule instead triggers the udev event for both interfaces, after they have been renamed as intended.

Let me know whether this works reliable for you. I’ll add it to our images then. I hope it does not cause any interference with the Ethernet LED rules.

EDIT: And this is unrelated to the DietPi or Debian version, but a matter of the kernel version only. If you upgrade the kernel package on Bullseye to the same version, you will face the same issue.

1 Like
3

Hello,

I have a nano pi r5c how do I use this??

4

Just paste the lines in the code box I posted above into your console/SSH client, it is a shell command which creates /etc/udev/rules.d/99-dietpi-nanopir5c.rules with this content:

SUBSYSTEM=="net", KERNEL=="eth0", KERNELS=="0002:01:00.0", RUN:="/bin/true"
SUBSYSTEM=="net", KERNEL=="eth1", KERNELS=="0001:01:00.0", NAME="to_eth0", RUN:="/bin/true"
SUBSYSTEM=="net", KERNEL=="to_eth0", RUN="/bin/ip l s dev eth0 name eth1", RUN+="/bin/ip l s dev to_eth0 name eth0", RUN+="/bin/udevadm trigger -c add /sys/class/net/eth0 /sys/class/net/eth1"
5

I don’t need to modify this?

 KERNELS=="0002:01:00.0"
6

It should be the correct way round. If unexpectedly WAN port is eth0 and LAN port eth1 fixed/throughout multiple reboots, report back, then the values need to be swapped.

7

@MichaIng Thanks for coming up with that fix. I will test it on a few devices over the coming days and report back.

For anyone else interested I have knocked up a quick script which implements the fix with a few safety checks here: DietPi-Scripts/Nanopi_R5C_R5S-Rockchip-fix.sh at main · MDBInd/DietPi-Scripts · GitHub

You can run it as follows:

bash -c "$(curl -fsSL https://raw.githubusercontent.com/MDBInd/DietPi-Scripts/refs/heads/main/Nanopi_R5C_R5S-Rockchip-fix.sh)"

1 Like
8

For me this works

9

I have some strange results from initial testing…

I could connect to the NanoPi R5C directly from my laptop but when I unplugged the WAN cable I lost my connection.

Setup

WAN port connected to local network
LAN port connected to my laptop

$ cat /etc/network/interfaces.d/eth0.cnf 
# LAN interface (static)
# Physical port: eth0
auto eth0
iface eth0 inet static
    hwaddress ether CA:CA:0F:66:49:D3
    address 192.168.17.1
    netmask 255.255.255.0

$ cat /etc/network/interfaces.d/eth1.cnf 
# WAN interface (DHCP)
# Physical port: eth1
allow-hotplug eth1
iface eth1 inet dhcp
    hwaddress ether CA:CA:0F:66:49:D4

# Debian: trixie
# DietPi: v9.20.1
# Kernel: 6.18.5-current-rockchip64

$ readlink -f /sys/class/net/eth0/device/
/sys/devices/platform/3c0800000.pcie/pci0002:20/0002:20:00.0/0002:21:00.0
$ readlink -f /sys/class/net/eth1/device/
/sys/devices/platform/3c0400000.pcie/pci0001:10/0001:10:00.0/0001:11:00.0

Note the path has changed after updating the kernel from 6.18.5-current-rockchip64 to 6.12.57-current-rockchip64.
eth0 (LAN) from pci0002:00 to pci0002:20
eth1 (WAN) remained the same pci0001:00

Further testing on another PC with a non Rockchip kernel revealed this:

# Debian: bullseye
# DietPi: v9.0.2
# Kernel: 5.10.160
# eth1 is LAN
# eth0 is WAN

$ readlink -f /sys/class/net/eth0/device/
/sys/devices/platform/3c0800000.pcie/pci0002:20/0002:20:00.0/0002:21:00.0
$ readlink -f /sys/class/net/eth1/device/
/sys/devices/platform/3c0400000.pcie/pci0001:10/0001:10:00.0/0001:11:00.0

Note: 3c0400000 is kernel agnostic for LAN and 3c0800000 for WAN.

Can we use those identifiers in the udev rules instead?

Something like this:

tee /etc/udev/rules.d/99-dietpi-nanopir5c.rules > /dev/null <<'EOT'
SUBSYSTEM=="net", KERNEL=="eth0", ENV{DEVPATH}==*/3c0800000.pcie/*, RUN:="/bin/true"
SUBSYSTEM=="net", KERNEL=="eth1", ENV{DEVPATH}==*/3c0400000.pcie/*, NAME="to_eth0", RUN:="/bin/true"
SUBSYSTEM=="net", KERNEL=="to_eth0", RUN="/bin/ip l s dev eth0 name eth1", RUN+="/bin/ip l s dev to_eth0 name eth0", RUN+="/bin/udevadm trigger -c add /sys/class/net/eth0 /sys/class/net/eth1"
EOT

Initial testing of the above ENV{DEVPATH} udev rules has resulted in a boot wait on Job ifupdown-pre.service/start running … Where the timer keeps on ticking so the rule must need more work.

1 Like
10

Ah right the bus ranges have been separated between Linux 6.12 and Linux 6.18 for RK356x SBCs, I forgot about that. They were the same on both PCIe nodes before, which did not cause any issues (AFAIK) but related kernel log entries which looked like an error, but had just “info” severity.

The correct rules are hence these (or your ENV{DEVPATH} solution, which looks fine as well):

cat << '_EOF_' > /etc/udev/rules.d/99-dietpi-nanopir5c.rules
SUBSYSTEM=="net", KERNEL=="eth0", KERNELS=="0002:21:00.0", RUN:="/bin/true"
SUBSYSTEM=="net", KERNEL=="eth1", KERNELS=="0001:11:00.0", NAME="to_eth0", RUN:="/bin/true"
SUBSYSTEM=="net", KERNEL=="to_eth0", RUN="/bin/ip l s dev eth0 name eth1", RUN+="/bin/ip l s dev to_eth0 name eth0", RUN+="/bin/udevadm trigger -c add /sys/class/net/eth0 /sys/class/net/eth1"
_EOF_

The ifupdown-pre.service does nothing but an udevadm settle, i.e. waits for udev to finish all rules. If hence one of the rules (its RUN commands) hang, that service will hang. It is however important to wait for finished udev, otherwise the network interfaces may not exist yet when network targets are reached, or ifup@eth0 may run while the renaming did not happen yet in this case. But ENV{DEVPATH} is unlikely to cause this. If the rule matches, the commands are the same. In /etc/udev/udev.conf, commenting out udev_log=info enables (very) verbose udev logs in journalctl, which can be used to check what udev is doing when, and which command takes how long. But it is a bit tedious to read due to the huge amount of log lines :sweat_smile:.

The interface name swap method of the rules does work well on Orange Pi 5 Plus, but no guarantee that it works as good on RK3568. As said, that chip has general PCIe issues, so possible that the Ethernet adapters are not that ready yet when the renaming shall happen, and then there are the additional Ethernet LED rules, which bring up and down the interface once, to put the LED into correct initial state (else it is lid even without any cable connected and interface down). Those rules are in /etc/modules-load.d/dietpi-eth-leds.conf. Commenting them out (or moving that file elsewhere) could be tested, to see whether this fixes the hanging udev. We should probably remove that anyway (and set the WiFi LED trigger separately), in which case the LEDs fall back to the driver’s own trigger, which means static lid LEDs if a cable is connected, but not blinking on activity.

11

Ok so after the “softbrick” a couple of days ago where I was stuck in a boot wait. I just booted the system fine and eth0 was LAN and eth1 WAN as they should be.

For reference I kept the udev rules unchanged trhoughout this testing post using my kernal agnostic method:

tee /etc/udev/rules.d/99-dietpi-nanopir5c.rules > /dev/null <<'EOT'
SUBSYSTEM=="net", KERNEL=="eth0", ENV{DEVPATH}==*/3c0800000.pcie/*, RUN:="/bin/true"
SUBSYSTEM=="net", KERNEL=="eth1", ENV{DEVPATH}==*/3c0400000.pcie/*, NAME="to_eth0", RUN:="/bin/true"
SUBSYSTEM=="net", KERNEL=="to_eth0", RUN="/bin/ip l s dev eth0 name eth1", RUN+="/bin/ip l s dev to_eth0 name eth0", RUN+="/bin/udevadm trigger -c add /sys/class/net/eth0 /sys/class/net/eth1"
EOT

So I didn’t touch the physical cables. I moved the led rules out of the way for testing sudo mv rules.d/dietpi-eth-leds.rules ~/ and uncommented udev_log=info in /etc/udev/udev.conf for debugging. I then reloaded the rules sudo udevadm control --reload-rules && sudo udevadm trigger and rebooted.

After rebooting eth0 is WAN and eth1 is LAN.

After a 2nd reboot they swap back to how they should be mapped.

I ran sudo journalctl -k > ~/boot.log in both caes and compared the data.

Apologies for the length of this post @MichaIng, but as you say there is a log of logging so I have tried to break it down the best I can. Note: The datetime, hostname and Kernel: have been omitted for readability.

The first 270 lines are identical up to this section:

Serial: 8250/16550 driver, 8 ports, IRQ sharing disabled
fe660000.serial: ttyS2 at MMIO 0xfe660000 (irq = 38, base_baud = 1500000) is a 16550A
Serial: AMBA driver
platform fdea0000.video-codec: Adding to iommu group 0
platform fdee0000.video-codec: Adding to iommu group 1
platform fe040000.vop: Adding to iommu group 2
loop: module loaded
Key type psk registered
thunder_xcv, ver 1.0
thunder_bgx, ver 1.0
nicpf, ver 1.0

Then on the faulty (eth0 is WAN) boot I get this from line 271:

usbcore: registered new interface driver usb-storage
mousedev: PS/2 mouse device common for all mice
i2c_dev: i2c /dev entries driver
fan53555-regulator 0-001c: FAN53555 Option[12] Rev[15] Detected!

Whereas on the correct (eh0 is LAN) boot I get this from line 271:

xhci-hcd xhci-hcd.0.auto: xHCI Host Controller
xhci-hcd xhci-hcd.0.auto: new USB bus registered, assigned bus number 1
xhci-hcd xhci-hcd.0.auto: hcc params 0x0220fe64 hci version 0x110 quirks 0x0000808002000010
xhci-hcd xhci-hcd.0.auto: irq 42, io mem 0xfcc00000
xhci-hcd xhci-hcd.0.auto: xHCI Host Controller
xhci-hcd xhci-hcd.0.auto: new USB bus registered, assigned bus number 2
xhci-hcd xhci-hcd.0.auto: Host supports USB 3.0 SuperSpeed
usb usb1: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 6.18
usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb1: Product: xHCI Host Controller
usb usb1: Manufacturer: Linux 6.18.5-current-rockchip64 xhci-hcd
usb usb1: SerialNumber: xhci-hcd.0.auto
ehci-platform fd880000.usb: EHCI Host Controller
ohci-platform fd8c0000.usb: Generic Platform OHCI controller
ehci-platform fd800000.usb: EHCI Host Controller
ohci-platform fd840000.usb: Generic Platform OHCI controller
hub 1-0:1.0: USB hub found
hub 1-0:1.0: 1 port detected
usb usb2: We don't know the algorithms for LPM for this host, disabling LPM.
usb usb2: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 6.18
usb usb2: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb2: Product: xHCI Host Controller
usb usb2: Manufacturer: Linux 6.18.5-current-rockchip64 xhci-hcd
usb usb2: SerialNumber: xhci-hcd.0.auto
hub 2-0:1.0: USB hub found
hub 2-0:1.0: 1 port detected
ehci-platform fd880000.usb: new USB bus registered, assigned bus number 3
ohci-platform fd8c0000.usb: new USB bus registered, assigned bus number 4
ehci-platform fd800000.usb: new USB bus registered, assigned bus number 5
xhci-hcd xhci-hcd.1.auto: xHCI Host Controller
ehci-platform fd880000.usb: irq 45, io mem 0xfd880000
ohci-platform fd8c0000.usb: irq 47, io mem 0xfd8c0000
ehci-platform fd800000.usb: irq 44, io mem 0xfd800000
xhci-hcd xhci-hcd.1.auto: new USB bus registered, assigned bus number 6
ohci-platform fd840000.usb: new USB bus registered, assigned bus number 7
ohci-platform fd840000.usb: irq 46, io mem 0xfd840000
xhci-hcd xhci-hcd.1.auto: hcc params 0x0220fe64 hci version 0x110 quirks 0x0000808002000010
xhci-hcd xhci-hcd.1.auto: irq 43, io mem 0xfd000000
xhci-hcd xhci-hcd.1.auto: xHCI Host Controller
xhci-hcd xhci-hcd.1.auto: new USB bus registered, assigned bus number 8
xhci-hcd xhci-hcd.1.auto: Host supports USB 3.0 SuperSpeed
usb usb6: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 6.18
usb usb6: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb6: Product: xHCI Host Controller
usb usb6: Manufacturer: Linux 6.18.5-current-rockchip64 xhci-hcd
usb usb6: SerialNumber: xhci-hcd.1.auto
hub 6-0:1.0: USB hub found
hub 6-0:1.0: 1 port detected
usb usb8: We don't know the algorithms for LPM for this host, disabling LPM.
usb usb8: New USB device found, idVendor=1d6b, idProduct=0003, bcdDevice= 6.18
usb usb8: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb8: Product: xHCI Host Controller
usb usb8: Manufacturer: Linux 6.18.5-current-rockchip64 xhci-hcd
usb usb8: SerialNumber: xhci-hcd.1.auto
hub 8-0:1.0: USB hub found
hub 8-0:1.0: 1 port detected
usbcore: registered new interface driver usb-storage
mousedev: PS/2 mouse device common for all mice
i2c_dev: i2c /dev entries driver
ehci-platform fd880000.usb: USB 2.0 started, EHCI 1.00
usb usb3: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 6.18
usb usb3: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb3: Product: EHCI Host Controller
usb usb3: Manufacturer: Linux 6.18.5-current-rockchip64 ehci_hcd
usb usb3: SerialNumber: fd880000.usb
fan53555-regulator 0-001c: FAN53555 Option[12] Rev[15] Detected!
hub 3-0:1.0: USB hub found
hub 3-0:1.0: 1 port detected
ehci-platform fd800000.usb: USB 2.0 started, EHCI 1.00
usb usb5: New USB device found, idVendor=1d6b, idProduct=0002, bcdDevice= 6.18
usb usb5: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb5: Product: EHCI Host Controller
usb usb5: Manufacturer: Linux 6.18.5-current-rockchip64 ehci_hcd
usb usb5: SerialNumber: fd800000.usb
hub 5-0:1.0: USB hub found
hub 5-0:1.0: 1 port detected
usb usb4: New USB device found, idVendor=1d6b, idProduct=0001, bcdDevice= 6.18
usb usb4: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb4: Product: Generic Platform OHCI controller
usb usb4: Manufacturer: Linux 6.18.5-current-rockchip64 ohci_hcd
usb usb4: SerialNumber: fd8c0000.usb
hub 4-0:1.0: USB hub found
hub 4-0:1.0: 1 port detected
usb usb7: New USB device found, idVendor=1d6b, idProduct=0001, bcdDevice= 6.18
usb usb7: New USB device strings: Mfr=3, Product=2, SerialNumber=1
usb usb7: Product: Generic Platform OHCI controller
usb usb7: Manufacturer: Linux 6.18.5-current-rockchip64 ohci_hcd
usb usb7: SerialNumber: fd840000.usb
hub 7-0:1.0: USB hub found
hub 7-0:1.0: 1 port detected

Then on the faulty boot I get the DW MMC controller on irw 71 and arm-scmi on 6:

dwmmc_rockchip fe2b0000.mmc: DW MMC controller at irq 71,32 bit host data width,256 deep fifo
arm-scmi arm-scmi.6.auto: Using scmi_smc_transport
arm-scmi arm-scmi.6.auto: SCMI max-rx-timeout: 30ms / max-msg-size: 104bytes / max-msg: 20
scmi_protocol scmi_dev.1: Enabled polling mode TX channel - prot_id:16
arm-scmi arm-scmi.6.auto: SCMI Notifications - Core Enabled.
arm-scmi arm-scmi.6.auto: SCMI Protocol v2.0 'rockchip:' Firmware version 0x0
arm-scmi arm-scmi.6.auto: Enabling SCMI Quirk [quirk_clock_rates_triplet_out_of_spec]

Whereas on the correct boot I get the DW MMC controller on irw 77 and arm-scmi on 8:

dwmmc_rockchip fe2b0000.mmc: DW MMC controller at irq 77,32 bit host data width,256 deep fifo
arm-scmi arm-scmi.8.auto: Using scmi_smc_transport
arm-scmi arm-scmi.8.auto: SCMI max-rx-timeout: 30ms / max-msg-size: 104bytes / max-msg: 20
scmi_protocol scmi_dev.1: Enabled polling mode TX channel - prot_id:16
arm-scmi arm-scmi.8.auto: SCMI Notifications - Core Enabled.
arm-scmi arm-scmi.8.auto: SCMI Protocol v2.0 'rockchip:' Firmware version 0x0
arm-scmi arm-scmi.8.auto: Enabling SCMI Quirk [quirk_clock_rates_triplet_out_of_spec]

Then both boots run through USB HDU, SDHCI controller through to AppArmor the same:

SMCCC: SOC_ID: ARCH_SOC_ID not implemented, skipping ....
hid: raw HID events driver (C) Jiri Kosina
usbcore: registered new interface driver usbhid
usbhid: USB HID core driver
rockchip-dfi fe230000.dfi: dfi initialized, dram type: 0x8, channels: 1
hw perfevents: enabled with armv8_cortex_a55 PMU driver, 7 (0,8000003f) counters available
mmc_host mmc0: Bus speed (slot 0) = 375000Hz (slot req 400000Hz, actual 375000HZ div = 0)
NET: Registered PF_INET6 protocol family
Segment Routing with IPv6
In-situ OAM (IOAM) with IPv6
NET: Registered PF_PACKET protocol family
bridge: filtering via arp/ip/ip6tables is no longer available by default. Update your scripts to load br_netfilter if you need this.
Key type dns_resolver registered
mmc1: SDHCI controller on fe310000.mmc [fe310000.mmc] using ADMA
registered taskstats version 1
Loading compiled-in X.509 certificates
zswap: loaded using pool zstd
Demotion targets for Node 0: null
Key type .fscrypt registered
Key type fscrypt-provisioning registered
Key type encrypted registered
AppArmor: AppArmor sha256 policy hashing enabled

At this point the faulty boot loads xhci-hcd and USB like the correct boot did back on line 271.

Then by line 437 both systems are essentially the same (only difference is rk808-rtc.3.auto on the correct boot and rk808-rtc.5.auto on the faulty one) the with this block:

input: adc-keys as /devices/platform/adc-keys/input/input6
rockchip-drm display-subsystem: bound fe040000.vop (ops vop2_component_ops [rockchipdrm])
dwhdmi-rockchip fe0a0000.hdmi: Detected HDMI TX controller v2.11a with HDCP (DWC HDMI 2.0 TX PHY)
dwhdmi-rockchip fe0a0000.hdmi: registered DesignWare HDMI I2C bus driver
rockchip-drm display-subsystem: bound fe0a0000.hdmi (ops dw_hdmi_rockchip_ops [rockchipdrm])
[drm] Initialized rockchip 1.0.0 for display-subsystem on minor 0
rk808-rtc rk808-rtc.3.auto: registered as rtc0
rk808-rtc rk808-rtc.3.auto: setting system clock to 2026-01-14T17:25:25 UTC (1768411525)
rtc-hym8563 5-0051: /aliases ID 0 not available
rtc-hym8563 5-0051: no valid clock/calendar values available
rtc-hym8563 5-0051: registered as rtc1
Console: switching to colour frame buffer device 240x67
rockchip-drm display-subsystem: [drm] fb0: rockchipdrmfb frame buffer device
phy phy-fe8c0000.phy.7: lane number 0, val 1
phy phy-fe8c0000.phy.7: lane number 1, val 2
phy phy-fe8c0000.phy.7: bifurcation enabled

At this point the 3c0800000 device (WAN) loads first on the faulty boot and 3c0400000 (LAN) loads on the correct boot.

Faulty boot:

rockchip-dw-pcie 3c0400000.pcie: host bridge /pcie@fe270000 ranges:
rockchip-dw-pcie 3c0400000.pcie:       IO 0x00f2100000..0x00f21fffff -> 0x00f2100000
rockchip-dw-pcie 3c0400000.pcie:      MEM 0x00f2200000..0x00f3ffffff -> 0x00f2200000
rockchip-dw-pcie 3c0400000.pcie:      MEM 0x0340000000..0x037fffffff -> 0x0040000000
rockchip-dw-pcie 3c0400000.pcie: iATU: unroll T, 8 ob, 8 ib, align 64K, limit 8G
rockchip-dw-pcie 3c0400000.pcie: PCI host bridge to bus 0001:10
pci_bus 0001:10: root bus resource [bus 10-1f]
pci_bus 0001:10: root bus resource [io  0x100000-0x1fffff] (bus address [0xf2100000-0xf21fffff])
pci_bus 0001:10: root bus resource [mem 0xf2200000-0xf3ffffff]
pci_bus 0001:10: root bus resource [mem 0x340000000-0x37fffffff] (bus address [0x40000000-0x7fffffff])
pci 0001:10:00.0: [1d87:3566] type 01 class 0x060400 PCIe Root Port
pci 0001:10:00.0: ROM [mem 0x00000000-0x0000ffff pref]
pci 0001:10:00.0: PCI bridge to [bus 01-ff]
pci 0001:10:00.0:   bridge window [io  0x0000-0x0fff]
pci 0001:10:00.0:   bridge window [mem 0x00000000-0x000fffff]
pci 0001:10:00.0:   bridge window [mem 0x00000000-0x000fffff 64bit pref]
pci 0001:10:00.0: supports D1 D2
pci 0001:10:00.0: PME# supported from D0 D1 D3hot
pci 0001:10:00.0: Primary bus is hard wired to 0
pci 0001:10:00.0: bridge configuration invalid ([bus 01-ff]), reconfiguring
pci 0001:11:00.0: [10ec:8125] type 00 class 0x020000 PCIe Endpoint
pci 0001:11:00.0: BAR 0 [io  0x0000-0x00ff]
pci 0001:11:00.0: BAR 2 [mem 0x00000000-0x0000ffff 64bit]
pci 0001:11:00.0: BAR 4 [mem 0x00000000-0x00003fff 64bit]
pci 0001:11:00.0: supports D1 D2
pci 0001:11:00.0: PME# supported from D0 D1 D2 D3hot D3cold
pci 0001:11:00.0: ASPM: default states L0s L1
pci_bus 0001:11: busn_res: [bus 11-1f] end is updated to 11
pci 0001:10:00.0: bridge window [mem 0xf2200000-0xf22fffff]: assigned
pci 0001:10:00.0: ROM [mem 0xf2300000-0xf230ffff pref]: assigned
pci 0001:10:00.0: bridge window [io  0x100000-0x100fff]: assigned
pci 0001:11:00.0: BAR 2 [mem 0xf2200000-0xf220ffff 64bit]: assigned
pci 0001:11:00.0: BAR 4 [mem 0xf2210000-0xf2213fff 64bit]: assigned
pci 0001:11:00.0: BAR 0 [io  0x100000-0x1000ff]: assigned
pci 0001:10:00.0: PCI bridge to [bus 11]
pci 0001:10:00.0:   bridge window [io  0x100000-0x100fff]
pci 0001:10:00.0:   bridge window [mem 0xf2200000-0xf22fffff]
pci_bus 0001:10: resource 4 [io  0x100000-0x1fffff]
pci_bus 0001:10: resource 5 [mem 0xf2200000-0xf3ffffff]
pci_bus 0001:10: resource 6 [mem 0x340000000-0x37fffffff]
pci_bus 0001:11: resource 0 [io  0x100000-0x100fff]
pci_bus 0001:11: resource 1 [mem 0xf2200000-0xf22fffff]
pcieport 0001:10:00.0: PME: Signaling with IRQ 98
pcieport 0001:10:00.0: AER: enabled with IRQ 99
r8169 0001:11:00.0: enabling device (0000 -> 0003)
r8169 0001:11:00.0 eth0: RTL8125B, ##:##:##:##:##:58, XID 641, IRQ 100
r8169 0001:11:00.0 eth0: jumbo features [frames: 16362 bytes, tx checksumming: ko]
rockchip-dw-pcie 3c0800000.pcie: host bridge /pcie@fe280000 ranges:
rockchip-dw-pcie 3c0800000.pcie:       IO 0x00f0100000..0x00f01fffff -> 0x00f0100000
rockchip-dw-pcie 3c0800000.pcie:      MEM 0x00f0200000..0x00f1ffffff -> 0x00f0200000
rockchip-dw-pcie 3c0800000.pcie:      MEM 0x0380000000..0x03bfffffff -> 0x0040000000
rockchip-dw-pcie 3c0800000.pcie: iATU: unroll T, 8 ob, 8 ib, align 64K, limit 8G
rockchip-dw-pcie 3c0800000.pcie: PCI host bridge to bus 0002:20
pci_bus 0002:20: root bus resource [bus 20-2f]
pci_bus 0002:20: root bus resource [io  0x200000-0x2fffff] (bus address [0xf0100000-0xf01fffff])
EXT4-fs (mmcblk1p1): mounted filesystem f4e38404-a5ad-429a-8345-bd4607c449db ro with ordered data mode. Quota mode: none.
pci_bus 0002:20: root bus resource [mem 0xf0200000-0xf1ffffff]
pci_bus 0002:20: root bus resource [mem 0x380000000-0x3bfffffff] (bus address [0x40000000-0x7fffffff])
pci 0002:20:00.0: [1d87:3566] type 01 class 0x060400 PCIe Root Port
pci 0002:20:00.0: BAR 0 [mem 0x00000000-0x3fffffff]
pci 0002:20:00.0: BAR 1 [mem 0x00000000-0x3fffffff]
pci 0002:20:00.0: ROM [mem 0x00000000-0x0000ffff pref]
pci 0002:20:00.0: PCI bridge to [bus 01-ff]
pci 0002:20:00.0:   bridge window [io  0x0000-0x0fff]
pci 0002:20:00.0:   bridge window [mem 0x00000000-0x000fffff]
pci 0002:20:00.0:   bridge window [mem 0x00000000-0x000fffff 64bit pref]
pci 0002:20:00.0: supports D1 D2
pci 0002:20:00.0: PME# supported from D0 D1 D3hot
pci 0002:20:00.0: Primary bus is hard wired to 0
pci 0002:20:00.0: bridge configuration invalid ([bus 01-ff]), reconfiguring
pci 0002:21:00.0: [10ec:8125] type 00 class 0x020000 PCIe Endpoint
pci 0002:21:00.0: BAR 0 [io  0x0000-0x00ff]
pci 0002:21:00.0: BAR 2 [mem 0x00000000-0x0000ffff 64bit]
pci 0002:21:00.0: BAR 4 [mem 0x00000000-0x00003fff 64bit]
pci 0002:21:00.0: supports D1 D2
pci 0002:21:00.0: PME# supported from D0 D1 D2 D3hot D3cold
pci 0002:21:00.0: ASPM: default states L0s L1
pci_bus 0002:21: busn_res: [bus 21-2f] end is updated to 21
pci 0002:20:00.0: BAR 0 [mem 0x380000000-0x3bfffffff]: assigned
pci 0002:20:00.0: BAR 1 [mem size 0x40000000]: can't assign; no space
pci 0002:20:00.0: BAR 1 [mem size 0x40000000]: failed to assign
pci 0002:20:00.0: bridge window [mem 0xf0200000-0xf02fffff]: assigned
pci 0002:20:00.0: ROM [mem 0xf0300000-0xf030ffff pref]: assigned
pci 0002:20:00.0: bridge window [io  0x200000-0x200fff]: assigned
pci 0002:20:00.0: BAR 0 [mem 0x380000000-0x3bfffffff]: releasing
pci 0002:20:00.0: bridge window [mem 0xf0200000-0xf02fffff]: releasing
pci 0002:20:00.0: ROM [mem 0xf0300000-0xf030ffff pref]: releasing
pci 0002:20:00.0: BAR 0 [mem 0x380000000-0x3bfffffff]: assigned
pci 0002:20:00.0: BAR 1 [mem size 0x40000000]: can't assign; no space
pci 0002:20:00.0: BAR 1 [mem size 0x40000000]: failed to assign
pci 0002:20:00.0: bridge window [mem 0xf0200000-0xf02fffff]: assigned
pci 0002:21:00.0: BAR 2 [mem 0xf0200000-0xf020ffff 64bit]: assigned
pci 0002:21:00.0: BAR 4 [mem 0xf0210000-0xf0213fff 64bit]: assigned
pci 0002:21:00.0: BAR 0 [io  0x200000-0x2000ff]: assigned
pci 0002:20:00.0: PCI bridge to [bus 21]
pci 0002:20:00.0:   bridge window [io  0x200000-0x200fff]
pci 0002:20:00.0:   bridge window [mem 0xf0200000-0xf02fffff]
pci_bus 0002:20: Some PCI device resources are unassigned, try booting with pci=realloc
pci_bus 0002:20: resource 4 [io  0x200000-0x2fffff]
pci_bus 0002:20: resource 5 [mem 0xf0200000-0xf1ffffff]
pci_bus 0002:20: resource 6 [mem 0x380000000-0x3bfffffff]
pci_bus 0002:21: resource 0 [io  0x200000-0x200fff]
pci_bus 0002:21: resource 1 [mem 0xf0200000-0xf02fffff]
pcieport 0002:20:00.0: PME: Signaling with IRQ 112
pcieport 0002:20:00.0: AER: enabled with IRQ 113
r8169 0002:21:00.0: enabling device (0000 -> 0003)
r8169 0002:21:00.0 eth1: RTL8125B, ##:##:##:##:##:59, XID 641, IRQ 114

Correct boot:

rockchip-dw-pcie 3c0800000.pcie: host bridge /pcie@fe280000 ranges:
rockchip-dw-pcie 3c0800000.pcie:       IO 0x00f0100000..0x00f01fffff -> 0x00f0100000
rockchip-dw-pcie 3c0800000.pcie:      MEM 0x00f0200000..0x00f1ffffff -> 0x00f0200000
rockchip-dw-pcie 3c0800000.pcie:      MEM 0x0380000000..0x03bfffffff -> 0x0040000000
rockchip-dw-pcie 3c0800000.pcie: iATU: unroll T, 8 ob, 8 ib, align 64K, limit 8G
rockchip-dw-pcie 3c0800000.pcie: PCI host bridge to bus 0002:20
pci_bus 0002:20: root bus resource [bus 20-2f]
pci_bus 0002:20: root bus resource [io  0x100000-0x1fffff] (bus address [0xf0100000-0xf01fffff])
pci_bus 0002:20: root bus resource [mem 0xf0200000-0xf1ffffff]
pci_bus 0002:20: root bus resource [mem 0x380000000-0x3bfffffff] (bus address [0x40000000-0x7fffffff])
pci 0002:20:00.0: [1d87:3566] type 01 class 0x060400 PCIe Root Port
pci 0002:20:00.0: BAR 0 [mem 0x00000000-0x3fffffff]
pci 0002:20:00.0: BAR 1 [mem 0x00000000-0x3fffffff]
pci 0002:20:00.0: ROM [mem 0x00000000-0x0000ffff pref]
pci 0002:20:00.0: PCI bridge to [bus 01-ff]
pci 0002:20:00.0:   bridge window [io  0x0000-0x0fff]
pci 0002:20:00.0:   bridge window [mem 0x00000000-0x000fffff]
pci 0002:20:00.0:   bridge window [mem 0x00000000-0x000fffff 64bit pref]
pci 0002:20:00.0: supports D1 D2
pci 0002:20:00.0: PME# supported from D0 D1 D3hot
pci 0002:20:00.0: Primary bus is hard wired to 0
pci 0002:20:00.0: bridge configuration invalid ([bus 01-ff]), reconfiguring
pci 0002:21:00.0: [10ec:8125] type 00 class 0x020000 PCIe Endpoint
pci 0002:21:00.0: BAR 0 [io  0x0000-0x00ff]
pci 0002:21:00.0: BAR 2 [mem 0x00000000-0x0000ffff 64bit]
pci 0002:21:00.0: BAR 4 [mem 0x00000000-0x00003fff 64bit]
pci 0002:21:00.0: supports D1 D2
pci 0002:21:00.0: PME# supported from D0 D1 D2 D3hot D3cold
pci 0002:21:00.0: ASPM: default states L0s L1
pci_bus 0002:21: busn_res: [bus 21-2f] end is updated to 21
pci 0002:20:00.0: BAR 0 [mem 0x380000000-0x3bfffffff]: assigned
pci 0002:20:00.0: BAR 1 [mem size 0x40000000]: can't assign; no space
pci 0002:20:00.0: BAR 1 [mem size 0x40000000]: failed to assign
pci 0002:20:00.0: bridge window [mem 0xf0200000-0xf02fffff]: assigned
pci 0002:20:00.0: ROM [mem 0xf0300000-0xf030ffff pref]: assigned
pci 0002:20:00.0: bridge window [io  0x100000-0x100fff]: assigned
pci 0002:20:00.0: BAR 0 [mem 0x380000000-0x3bfffffff]: releasing
pci 0002:20:00.0: bridge window [mem 0xf0200000-0xf02fffff]: releasing
pci 0002:20:00.0: ROM [mem 0xf0300000-0xf030ffff pref]: releasing
pci 0002:20:00.0: BAR 0 [mem 0x380000000-0x3bfffffff]: assigned
pci 0002:20:00.0: BAR 1 [mem size 0x40000000]: can't assign; no space
pci 0002:20:00.0: BAR 1 [mem size 0x40000000]: failed to assign
pci 0002:20:00.0: bridge window [mem 0xf0200000-0xf02fffff]: assigned
pci 0002:21:00.0: BAR 2 [mem 0xf0200000-0xf020ffff 64bit]: assigned
pci 0002:21:00.0: BAR 4 [mem 0xf0210000-0xf0213fff 64bit]: assigned
pci 0002:21:00.0: BAR 0 [io  0x100000-0x1000ff]: assigned
pci 0002:20:00.0: PCI bridge to [bus 21]
pci 0002:20:00.0:   bridge window [io  0x100000-0x100fff]
pci 0002:20:00.0:   bridge window [mem 0xf0200000-0xf02fffff]
pci_bus 0002:20: Some PCI device resources are unassigned, try booting with pci=realloc
pci_bus 0002:20: resource 4 [io  0x100000-0x1fffff]
pci_bus 0002:20: resource 5 [mem 0xf0200000-0xf1ffffff]
pci_bus 0002:20: resource 6 [mem 0x380000000-0x3bfffffff]
pci_bus 0002:21: resource 0 [io  0x100000-0x100fff]
pci_bus 0002:21: resource 1 [mem 0xf0200000-0xf02fffff]
pcieport 0002:20:00.0: PME: Signaling with IRQ 98
pcieport 0002:20:00.0: AER: enabled with IRQ 99
r8169 0002:21:00.0: enabling device (0000 -> 0003)
r8169 0002:21:00.0 eth0: RTL8125B, ##:##:##:##:##:59, XID 641, IRQ 100
r8169 0002:21:00.0 eth0: jumbo features [frames: 16362 bytes, tx checksumming: ko]

To help make sense of this I ran the diff output through ChatGPT and got the following:

On the faulty boot we get:

xhci-hcd xhci-hcd.0.auto
xhci-hcd xhci-hcd.1.auto
...
pci_bus 0001:10: root bus resource [io  0x200000-0x2fffff]

On the correct boot we get:

xhci-hcd xhci-hcd.7.auto
xhci-hcd xhci-hcd.8.auto
..
pci_bus 0001:10: root bus resource [io  0x100000-0x1fffff]

I really hope that helps narrow things down a little. There really isn’t much indication as to why usb starts on one boot sooner than the other. If there is any more tested I can do I am happy to help out!

12

Apologies in advance for the long post, I hit the character limit in my previous post due to all the logs.

Further to this I ran sudo udevadm monitor on the correct system and then ran sudo udevadm control --reload-rules && sudo udevadm trigger in another shell and filtered the output:

$ grep -E "3c0400000|3c0800000|eth0|eth1" udev-correct.log
KERNEL[5024.970811] change   /devices/platform/3c0400000.pcie (platform)
KERNEL[5024.970958] change   /devices/platform/3c0400000.pcie/pci0001:10/0001:10:00.0 (pci)
KERNEL[5024.971035] change   /devices/platform/3c0400000.pcie/pci0001:10/0001:10:00.0/0001:10:00.0:pcie001 (pci_express)
KERNEL[5024.971099] change   /devices/platform/3c0400000.pcie/pci0001:10/0001:10:00.0/0001:10:00.0:pcie002 (pci_express)
KERNEL[5024.971179] change   /devices/platform/3c0400000.pcie/pci0001:10/0001:10:00.0/0001:10:00.0:pcie010 (pci_express)
KERNEL[5024.971261] change   /devices/platform/3c0400000.pcie/pci0001:10/0001:10:00.0/0001:11:00.0 (pci)
KERNEL[5024.971325] change   /devices/platform/3c0400000.pcie/pci0001:10/0001:10:00.0/0001:11:00.0/mdio_bus/r8169-1-1100 (mdio_bus)
KERNEL[5024.971387] change   /devices/platform/3c0400000.pcie/pci0001:10/0001:10:00.0/0001:11:00.0/mdio_bus/r8169-1-1100/r8169-1-1100:00 (mdio_bus)
KERNEL[5024.971454] change   /devices/platform/3c0400000.pcie/pci0001:10/0001:10:00.0/0001:11:00.0/net/eth0 (net)
KERNEL[5024.971519] change   /devices/platform/3c0400000.pcie/pci0001:10/0001:10:00.0/pci_bus/0001:11 (pci_bus)
KERNEL[5024.971588] change   /devices/platform/3c0400000.pcie/pci0001:10/pci_bus/0001:10 (pci_bus)
KERNEL[5024.971660] change   /devices/platform/3c0800000.pcie (platform)
KERNEL[5024.971737] change   /devices/platform/3c0800000.pcie/pci0002:20/0002:20:00.0 (pci)
KERNEL[5024.971796] change   /devices/platform/3c0800000.pcie/pci0002:20/0002:20:00.0/0002:20:00.0:pcie001 (pci_express)
KERNEL[5024.971851] change   /devices/platform/3c0800000.pcie/pci0002:20/0002:20:00.0/0002:20:00.0:pcie002 (pci_express)
KERNEL[5024.971926] change   /devices/platform/3c0800000.pcie/pci0002:20/0002:20:00.0/0002:21:00.0 (pci)
KERNEL[5024.971984] change   /devices/platform/3c0800000.pcie/pci0002:20/0002:20:00.0/0002:21:00.0/mdio_bus/r8169-2-2100 (mdio_bus)
KERNEL[5024.973396] change   /devices/platform/3c0800000.pcie/pci0002:20/0002:20:00.0/0002:21:00.0/mdio_bus/r8169-2-2100/r8169-2-2100:00 (mdio_bus)
KERNEL[5024.974358] change   /devices/platform/3c0800000.pcie/pci0002:20/0002:20:00.0/0002:21:00.0/net/eth1 (net)
KERNEL[5024.974578] change   /devices/platform/3c0800000.pcie/pci0002:20/0002:20:00.0/pci_bus/0002:21 (pci_bus)
KERNEL[5024.975142] change   /devices/platform/3c0800000.pcie/pci0002:20/pci_bus/0002:20 (pci_bus)
UDEV  [5024.996416] change   /devices/platform/3c0400000.pcie (platform)
UDEV  [5025.003404] change   /devices/platform/3c0800000.pcie (platform)
UDEV  [5025.004892] change   /devices/platform/3c0400000.pcie/pci0001:10/0001:10:00.0 (pci)
UDEV  [5025.006337] change   /devices/platform/3c0400000.pcie/pci0001:10/pci_bus/0001:10 (pci_bus)
UDEV  [5025.016709] change   /devices/platform/3c0400000.pcie/pci0001:10/0001:10:00.0/0001:10:00.0:pcie002 (pci_express)
UDEV  [5025.017159] change   /devices/platform/3c0400000.pcie/pci0001:10/0001:10:00.0/0001:10:00.0:pcie001 (pci_express)
UDEV  [5025.017527] change   /devices/platform/3c0800000.pcie/pci0002:20/0002:20:00.0 (pci)
UDEV  [5025.020212] change   /devices/platform/3c0400000.pcie/pci0001:10/0001:10:00.0/0001:11:00.0 (pci)
UDEV  [5025.021974] change   /devices/platform/3c0400000.pcie/pci0001:10/0001:10:00.0/0001:10:00.0:pcie010 (pci_express)
UDEV  [5025.022899] change   /devices/platform/3c0400000.pcie/pci0001:10/0001:10:00.0/pci_bus/0001:11 (pci_bus)
UDEV  [5025.030363] change   /devices/platform/3c0800000.pcie/pci0002:20/0002:20:00.0/0002:20:00.0:pcie001 (pci_express)
UDEV  [5025.033413] change   /devices/platform/3c0800000.pcie/pci0002:20/0002:20:00.0/0002:20:00.0:pcie002 (pci_express)
UDEV  [5025.038445] change   /devices/platform/3c0400000.pcie/pci0001:10/0001:10:00.0/0001:11:00.0/mdio_bus/r8169-1-1100 (mdio_bus)
UDEV  [5025.049567] change   /devices/platform/3c0400000.pcie/pci0001:10/0001:10:00.0/0001:11:00.0/net/eth0 (net)
UDEV  [5025.050830] change   /devices/platform/3c0800000.pcie/pci0002:20/pci_bus/0002:20 (pci_bus)
UDEV  [5025.055117] change   /devices/platform/3c0800000.pcie/pci0002:20/0002:20:00.0/0002:21:00.0 (pci)
UDEV  [5025.067427] change   /devices/platform/3c0800000.pcie/pci0002:20/0002:20:00.0/pci_bus/0002:21 (pci_bus)
UDEV  [5025.072987] change   /devices/platform/3c0400000.pcie/pci0001:10/0001:10:00.0/0001:11:00.0/mdio_bus/r8169-1-1100/r8169-1-1100:00 (mdio_bus)
UDEV  [5025.076924] change   /devices/platform/3c0800000.pcie/pci0002:20/0002:20:00.0/0002:21:00.0/mdio_bus/r8169-2-2100 (mdio_bus)
UDEV  [5025.078831] change   /devices/platform/3c0800000.pcie/pci0002:20/0002:20:00.0/0002:21:00.0/mdio_bus/r8169-2-2100/r8169-2-2100:00 (mdio_bus)
KERNEL[5025.092311] change   /devices/virtual/devlink/phy:phy-fe8c0000.phy.7--platform:3c0400000.pcie (devlink)
KERNEL[5025.092630] change   /devices/virtual/devlink/phy:phy-fe8c0000.phy.7--platform:3c0800000.pcie (devlink)
KERNEL[5025.094441] change   /devices/virtual/devlink/platform:fdd60000.gpio--platform:3c0400000.pcie (devlink)
KERNEL[5025.094981] change   /devices/virtual/devlink/platform:fdd60000.gpio--platform:3c0800000.pcie (devlink)
KERNEL[5025.096301] change   /devices/virtual/devlink/platform:fdd90000.power-management:power-controller--platform:3c0400000.pcie (devlink)
KERNEL[5025.096508] change   /devices/virtual/devlink/platform:fdd90000.power-management:power-controller--platform:3c0800000.pcie (devlink)
KERNEL[5025.103189] change   /devices/virtual/devlink/platform:fe8c0000.phy--platform:3c0400000.pcie (devlink)
KERNEL[5025.103471] change   /devices/virtual/devlink/platform:fe8c0000.phy--platform:3c0800000.pcie (devlink)
KERNEL[5025.104762] change   /devices/virtual/devlink/platform:regulator-vcc3v3-pcie--platform:3c0400000.pcie (devlink)
KERNEL[5025.104813] change   /devices/virtual/devlink/platform:regulator-vcc3v3-pcie--platform:3c0800000.pcie (devlink)
UDEV  [5025.105202] change   /devices/platform/3c0800000.pcie/pci0002:20/0002:20:00.0/0002:21:00.0/net/eth1 (net)
KERNEL[5025.111343] change   /devices/virtual/devlink/regulator:regulator.4--platform:3c0400000.pcie (devlink)
KERNEL[5025.111393] change   /devices/virtual/devlink/regulator:regulator.4--platform:3c0800000.pcie (devlink)
KERNEL[5025.113539] change   /devices/virtual/net/veth00cae65 (net)
UDEV  [5025.663067] change   /devices/virtual/devlink/phy:phy-fe8c0000.phy.7--platform:3c0400000.pcie (devlink)
UDEV  [5025.663129] change   /devices/virtual/devlink/platform:fdd60000.gpio--platform:3c0800000.pcie (devlink)
UDEV  [5025.679691] change   /devices/virtual/devlink/platform:fdd60000.gpio--platform:3c0400000.pcie (devlink)
UDEV  [5025.682608] change   /devices/virtual/devlink/phy:phy-fe8c0000.phy.7--platform:3c0800000.pcie (devlink)
UDEV  [5025.684968] change   /devices/virtual/devlink/platform:fdd90000.power-management:power-controller--platform:3c0400000.pcie (devlink)
UDEV  [5025.685930] change   /devices/virtual/devlink/platform:fdd90000.power-management:power-controller--platform:3c0800000.pcie (devlink)
UDEV  [5025.686736] change   /devices/virtual/devlink/platform:fe8c0000.phy--platform:3c0400000.pcie (devlink)
UDEV  [5025.693423] change   /devices/virtual/devlink/platform:regulator-vcc3v3-pcie--platform:3c0400000.pcie (devlink)
UDEV  [5025.693764] change   /devices/virtual/devlink/platform:regulator-vcc3v3-pcie--platform:3c0800000.pcie (devlink)
UDEV  [5025.694637] change   /devices/virtual/devlink/platform:fe8c0000.phy--platform:3c0800000.pcie (devlink)
UDEV  [5025.705724] change   /devices/virtual/devlink/regulator:regulator.4--platform:3c0400000.pcie (devlink)
UDEV  [5025.736424] change   /devices/virtual/net/veth00cae65 (net)
UDEV  [5025.769177] change   /devices/virtual/devlink/regulator:regulator.4--platform:3c0800000.pcie (devlink)

After > 10 reboots having the correct mappings I finally got a faulty boot. And ran the same commands with the below output.

$ grep -E "3c0400000|3c0800000|eth0|eth1" udev-faulty.log
KERNEL[801.963679] change   /devices/platform/3c0400000.pcie (platform)
KERNEL[801.963754] change   /devices/platform/3c0400000.pcie/pci0001:10/0001:10:00.0 (pci)
KERNEL[801.963808] change   /devices/platform/3c0400000.pcie/pci0001:10/0001:10:00.0/0001:10:00.0:pcie001 (pci_express)
KERNEL[801.963857] change   /devices/platform/3c0400000.pcie/pci0001:10/0001:10:00.0/0001:10:00.0:pcie002 (pci_express)
KERNEL[801.963907] change   /devices/platform/3c0400000.pcie/pci0001:10/0001:10:00.0/0001:10:00.0:pcie010 (pci_express)
KERNEL[801.963977] change   /devices/platform/3c0400000.pcie/pci0001:10/0001:10:00.0/0001:11:00.0 (pci)
KERNEL[801.964158] change   /devices/platform/3c0400000.pcie/pci0001:10/0001:10:00.0/0001:11:00.0/mdio_bus/r8169-1-1100 (mdio_bus)
KERNEL[801.964232] change   /devices/platform/3c0400000.pcie/pci0001:10/0001:10:00.0/0001:11:00.0/mdio_bus/r8169-1-1100/r8169-1-1100:00 (mdio_bus)
KERNEL[801.964303] change   /devices/platform/3c0400000.pcie/pci0001:10/0001:10:00.0/0001:11:00.0/net/eth1 (net)
KERNEL[801.964363] change   /devices/platform/3c0400000.pcie/pci0001:10/0001:10:00.0/pci_bus/0001:11 (pci_bus)
KERNEL[801.964427] change   /devices/platform/3c0400000.pcie/pci0001:10/pci_bus/0001:10 (pci_bus)
KERNEL[801.964499] change   /devices/platform/3c0800000.pcie (platform)
KERNEL[801.964771] change   /devices/platform/3c0800000.pcie/pci0002:20/0002:20:00.0 (pci)
KERNEL[801.964854] change   /devices/platform/3c0800000.pcie/pci0002:20/0002:20:00.0/0002:20:00.0:pcie001 (pci_express)
KERNEL[801.964931] change   /devices/platform/3c0800000.pcie/pci0002:20/0002:20:00.0/0002:20:00.0:pcie002 (pci_express)
KERNEL[801.965003] change   /devices/platform/3c0800000.pcie/pci0002:20/0002:20:00.0/0002:21:00.0 (pci)
KERNEL[801.965057] change   /devices/platform/3c0800000.pcie/pci0002:20/0002:20:00.0/0002:21:00.0/mdio_bus/r8169-2-2100 (mdio_bus)
KERNEL[801.965113] change   /devices/platform/3c0800000.pcie/pci0002:20/0002:20:00.0/0002:21:00.0/mdio_bus/r8169-2-2100/r8169-2-2100:00 (mdio_bus)
KERNEL[801.965170] change   /devices/platform/3c0800000.pcie/pci0002:20/0002:20:00.0/0002:21:00.0/net/eth0 (net)
KERNEL[801.965224] change   /devices/platform/3c0800000.pcie/pci0002:20/0002:20:00.0/pci_bus/0002:21 (pci_bus)
KERNEL[801.965293] change   /devices/platform/3c0800000.pcie/pci0002:20/pci_bus/0002:20 (pci_bus)
KERNEL[802.010796] change   /devices/virtual/devlink/phy:phy-fe8c0000.phy.7--platform:3c0400000.pcie (devlink)
KERNEL[802.010846] change   /devices/virtual/devlink/phy:phy-fe8c0000.phy.7--platform:3c0800000.pcie (devlink)
KERNEL[802.011070] change   /devices/virtual/devlink/platform:fdd60000.gpio--platform:3c0400000.pcie (devlink)
KERNEL[802.011127] change   /devices/virtual/devlink/platform:fdd60000.gpio--platform:3c0800000.pcie (devlink)
KERNEL[802.011410] change   /devices/virtual/devlink/platform:fdd90000.power-management:power-controller--platform:3c0400000.pcie (devlink)
UDEV  [802.011506] change   /devices/platform/3c0800000.pcie (platform)
KERNEL[802.011564] change   /devices/virtual/devlink/platform:fdd90000.power-management:power-controller--platform:3c0800000.pcie (devlink)
UDEV  [802.013701] change   /devices/platform/3c0400000.pcie (platform)
KERNEL[802.014788] change   /devices/virtual/devlink/platform:fe8c0000.phy--platform:3c0400000.pcie (devlink)
KERNEL[802.014855] change   /devices/virtual/devlink/platform:fe8c0000.phy--platform:3c0800000.pcie (devlink)
KERNEL[802.015894] change   /devices/virtual/devlink/platform:regulator-vcc3v3-pcie--platform:3c0400000.pcie (devlink)
KERNEL[802.015954] change   /devices/virtual/devlink/platform:regulator-vcc3v3-pcie--platform:3c0800000.pcie (devlink)
KERNEL[802.020457] change   /devices/virtual/devlink/regulator:regulator.24--platform:3c0400000.pcie (devlink)
KERNEL[802.020520] change   /devices/virtual/devlink/regulator:regulator.24--platform:3c0800000.pcie (devlink)
UDEV  [802.022512] change   /devices/platform/3c0800000.pcie/pci0002:20/pci_bus/0002:20 (pci_bus)
KERNEL[802.022858] change   /devices/virtual/net/veth09c5da5 (net)
KERNEL[802.022913] change   /devices/virtual/net/veth0cb2fc3 (net)
UDEV  [802.031416] change   /devices/platform/3c0800000.pcie/pci0002:20/0002:20:00.0 (pci)
UDEV  [802.041507] change   /devices/platform/3c0400000.pcie/pci0001:10/0001:10:00.0 (pci)
UDEV  [802.042286] change   /devices/platform/3c0400000.pcie/pci0001:10/pci_bus/0001:10 (pci_bus)
UDEV  [802.048209] change   /devices/platform/3c0800000.pcie/pci0002:20/0002:20:00.0/0002:20:00.0:pcie002 (pci_express)
UDEV  [802.051331] change   /devices/platform/3c0800000.pcie/pci0002:20/0002:20:00.0/0002:20:00.0:pcie001 (pci_express)
UDEV  [802.058776] change   /devices/platform/3c0800000.pcie/pci0002:20/0002:20:00.0/0002:21:00.0 (pci)
UDEV  [802.066671] change   /devices/platform/3c0800000.pcie/pci0002:20/0002:20:00.0/pci_bus/0002:21 (pci_bus)
UDEV  [802.068169] change   /devices/platform/3c0400000.pcie/pci0001:10/0001:10:00.0/pci_bus/0001:11 (pci_bus)
UDEV  [802.069067] change   /devices/platform/3c0400000.pcie/pci0001:10/0001:10:00.0/0001:10:00.0:pcie001 (pci_express)
UDEV  [802.073027] change   /devices/platform/3c0400000.pcie/pci0001:10/0001:10:00.0/0001:10:00.0:pcie002 (pci_express)
UDEV  [802.073119] change   /devices/platform/3c0400000.pcie/pci0001:10/0001:10:00.0/0001:10:00.0:pcie010 (pci_express)
UDEV  [802.090814] change   /devices/platform/3c0800000.pcie/pci0002:20/0002:20:00.0/0002:21:00.0/mdio_bus/r8169-2-2100 (mdio_bus)
UDEV  [802.092140] change   /devices/platform/3c0400000.pcie/pci0001:10/0001:10:00.0/0001:11:00.0 (pci)
UDEV  [802.114813] change   /devices/platform/3c0400000.pcie/pci0001:10/0001:10:00.0/0001:11:00.0/mdio_bus/r8169-1-1100 (mdio_bus)
UDEV  [802.124502] change   /devices/platform/3c0800000.pcie/pci0002:20/0002:20:00.0/0002:21:00.0/net/eth0 (net)
UDEV  [802.143561] change   /devices/platform/3c0800000.pcie/pci0002:20/0002:20:00.0/0002:21:00.0/mdio_bus/r8169-2-2100/r8169-2-2100:00 (mdio_bus)
UDEV  [802.151191] change   /devices/platform/3c0400000.pcie/pci0001:10/0001:10:00.0/0001:11:00.0/mdio_bus/r8169-1-1100/r8169-1-1100:00 (mdio_bus)
UDEV  [802.207369] change   /devices/platform/3c0400000.pcie/pci0001:10/0001:10:00.0/0001:11:00.0/net/eth1 (net)
UDEV  [803.594560] change   /devices/virtual/devlink/phy:phy-fe8c0000.phy.7--platform:3c0800000.pcie (devlink)
UDEV  [803.595284] change   /devices/virtual/devlink/platform:fdd60000.gpio--platform:3c0400000.pcie (devlink)
UDEV  [803.638906] change   /devices/virtual/devlink/platform:fdd90000.power-management:power-controller--platform:3c0800000.pcie (devlink)
UDEV  [803.648575] change   /devices/virtual/devlink/phy:phy-fe8c0000.phy.7--platform:3c0400000.pcie (devlink)
UDEV  [803.744967] change   /devices/virtual/devlink/platform:fdd60000.gpio--platform:3c0800000.pcie (devlink)
UDEV  [803.745137] change   /devices/virtual/devlink/platform:fdd90000.power-management:power-controller--platform:3c0400000.pcie (devlink)
UDEV  [803.745467] change   /devices/virtual/devlink/platform:fe8c0000.phy--platform:3c0400000.pcie (devlink)
UDEV  [803.745722] change   /devices/virtual/devlink/platform:fe8c0000.phy--platform:3c0800000.pcie (devlink)
UDEV  [803.746573] change   /devices/virtual/devlink/platform:regulator-vcc3v3-pcie--platform:3c0400000.pcie (devlink)
UDEV  [803.756874] change   /devices/virtual/devlink/platform:regulator-vcc3v3-pcie--platform:3c0800000.pcie (devlink)
UDEV  [803.844882] change   /devices/virtual/devlink/regulator:regulator.24--platform:3c0800000.pcie (devlink)
UDEV  [803.845757] change   /devices/virtual/devlink/regulator:regulator.24--platform:3c0400000.pcie (devlink)
UDEV  [803.847358] change   /devices/virtual/net/veth09c5da5 (net)
UDEV  [803.852303] change   /devices/virtual/net/veth0cb2fc3 (net)

When I remove the timings and diff them you can see the faulty boot outputs:

/devices/platform/3c0400000.pcie/pci0001:10/0001:10:00.0/0001:11:00.0/net/eth1 (net)
/devices/platform/3c0800000.pcie/pci0002:20/0002:20:00.0/0002:21:00.0/net/eth0 (net)

And the correct boot outputs:

/devices/platform/3c0400000.pcie/pci0001:10/0001:10:00.0/0001:11:00.0/net/eth0 (net)
/devices/platform/3c0800000.pcie/pci0002:20/0002:20:00.0/0002:21:00.0/net/eth1 (net)

But the lead up to these lines is identical on both boots so I am at a bit of a loss as to why this is happening and the udev does not seem to swap the port mappings consistently.

Any ideas?

13

Any news here?

14

I have not solved this yet. I am looking into downgrading the kernel to v5.10.160 until this
6.*-current-rockchip64 kernel issue can be fixed.

When you said this @temp, was the fix only temporary?

Older v5.* kernels are no longer available in the repo so I am trying to extract the data from a stable system. Another option is to download the rk3568-XYZ-debian-bullseye-minimal-6.1-arm64-YYYYMMDD.img.gz image directly from FriendlyElec then try the dietpi conversion script. Presumably the official images do not suffer from this eth0/eth1 problem even though they are using a 6.1* kernel.

15

For me I have no issues anymore

16

That’s great. Can you clarify the steps you have taken to resolve the problem?

In my testing the eth0 and eth1 ports can stay the same for up to ~10 reboots and then the next boot they have swapped.

17

I have only done this

I have now done 20 reboots or more everything is fine

1 Like
18

@firebox
So even with your udev rules, the interfaces swap? If 3c0400000.pcie gets assigned to eth1 (and if the ENV{DEVPATH} check works like that), you should see the /bin/true commands in udev info logs. They happen at a later stage, i.e. all those RUN commands run after initial device detection and e.g. attribute changes. And then you should also see the to_eth0 handling at some point.

As for the underlying reason: It is not necessarily an issue that devices are detected in inconsistent order. This does not only happen on the R5C but also in some other cases, and as you can see also not only for PCIe but also USB (where it does not cause issues as long as no multiple USB network adapters are attached). When things are parallelized, there are race conditions, just usually for onboard devices the power-up/initializing duration is sufficiently different to keep it in consistent order. But obviously this is not always the case.

19

Thanks for the information @MichaIng and the feedback @temp. I dig a little digging on the udev rules and found that ENV{DEVPATH} is not permitted - systemd/src/udev/udev-rules.c at dde8bb32b12c855509777ce52ff59a835155ac78 · systemd/systemd · GitHub

With this in mind I will update my rules as you originally suggested @MichaIng and report back after further testing. Based on my observations with older Kernels this method is not kernel agnostic so I have updated my script to take this into account and use the valid device paths for the target system.

bash -c "$(curl -fsSL https://raw.githubusercontent.com/MDBInd/DietPi-Scripts/refs/heads/main/Nanopi_R5C_R5S-Rockchip-fix.sh)"

After a couple of reboots I am pleased to report this is working! The ports were swapped when I ran the updated script, after the first reboot the ports were correct and subsequent reboots seem stable.

1 Like
20

Great find about the blacklisted variables. Good idea to match the MMIO bus address via KERNELS. Shouldn’t KERNELS=="*3c0400000*" work as well, so you don’t need to print and parse the exact device path element?

And do the LED rules interfere, or does it still work perfectly for you when you re-enable them?