NanoPi R6S poor iperf3 performance (vs. friendlyelec bullseye img)

Troubleshooting
1

Creating a bug report/issue

Required Information

  • DietPi version | cat /boot/dietpi/.version
    G_DIETPI_VERSION_CORE=8
    G_DIETPI_VERSION_SUB=18
    G_DIETPI_VERSION_RC=2
    G_GITBRANCH=‘master’
    G_GITOWNER=‘MichaIng’
    G_LIVE_PATCH_STATUS[0]=‘not applicable’
  • Distro version | echo $G_DISTRO_NAME $G_RASPBIAN
    bookworm
  • Kernel version | uname -a
    Linux in 5.10.110 #1 SMP Wed May 24 13:40:15 CST 2023 aarch64 GNU/Linux
  • Architecture | dpkg --print-architecture
    arm64
  • SBC model | echo $G_HW_MODEL_NAME or (EG: RPi3)
    NanoPi R6S/R6C (aarch64)
  • Power supply used | (EG: 5V 1A RAVpower)
    5V 1A
  • SD card used | (EG: SanDisk ultra)
    emmc 32gb

Additional Information (if applicable)

  • Software title | iperf3
  • Was the software title installed freshly or updated/migrated? fresh
  • Can this issue be replicated on a fresh installation of DietPi?
    401bc4a6-6cc9-4380-a877-796f39072bc2
  • Bug report ID | echo $G_HW_UUID
    401bc4a6-6cc9-4380-a877-796f39072bc2

This test result is from friendlyelec bullseye image, you can see 2.35Gbits/sec is achieved.

# iperf3 -c 192.168.20.254 -P4
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   706 MBytes   592 Mbits/sec    0             sender
[  5]   0.00-10.00  sec   703 MBytes   589 Mbits/sec                  receiver
[  7]   0.00-10.00  sec   737 MBytes   618 Mbits/sec    0             sender
[  7]   0.00-10.00  sec   734 MBytes   616 Mbits/sec                  receiver
[  9]   0.00-10.00  sec   725 MBytes   608 Mbits/sec    0             sender
[  9]   0.00-10.00  sec   722 MBytes   605 Mbits/sec                  receiver
[ 11]   0.00-10.00  sec   651 MBytes   546 Mbits/sec    0             sender
[ 11]   0.00-10.00  sec   647 MBytes   543 Mbits/sec                  receiver
[SUM]   0.00-10.00  sec  2.75 GBytes  2.36 Gbits/sec    0             sender
[SUM]   0.00-10.00  sec  2.74 GBytes  2.35 Gbits/sec                  receiver

iperf Done.
root@NanoPi-R6S:/home/pi# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
root@NanoPi-R6S:/home/pi#

Steps to reproduce

  1. install iperf3 - have a server on same vlan on same switch
  2. .run iperf3 -s on server and iperf3 -c 192.168.20.254 -P4 on dietpi

Results show a performance issue where 15-20% of network performance is lost or lower than friendlyelec image.

- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   453 MBytes   380 Mbits/sec    0             sender
[  5]   0.00-10.00  sec   450 MBytes   378 Mbits/sec                  receiver
[  7]   0.00-10.00  sec   680 MBytes   570 Mbits/sec    0             sender
[  7]   0.00-10.00  sec   676 MBytes   567 Mbits/sec                  receiver
[  9]   0.00-10.00  sec   677 MBytes   568 Mbits/sec    0             sender
[  9]   0.00-10.00  sec   673 MBytes   565 Mbits/sec                  receiver
[ 11]   0.00-10.00  sec   665 MBytes   558 Mbits/sec    0             sender
[ 11]   0.00-10.00  sec   662 MBytes   555 Mbits/sec                  receiver
[SUM]   0.00-10.00  sec  2.42 GBytes  2.08 Gbits/sec    0             sender
[SUM]   0.00-10.00  sec  2.41 GBytes  2.06 Gbits/sec                  receiver

Expected behaviour

  • iperf3 should operate close to maximum line speeds between 2 devices on the same switch/vlan at 2.5Gbps

Actual behaviour

  • ~20% less performance on iperf3 results (vs. friendlyelec vanilla image)

Extra details

  • …
2

I just flashed another SD card with dietpi vanilla bookworm. It seems speeds are better (2.06gbps original, 10% more).

[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-20.00  sec  1.27 GBytes   547 Mbits/sec    0             sender
[  5]   0.00-20.01  sec  1.27 GBytes   546 Mbits/sec                  receiver
[  7]   0.00-20.00  sec  1.28 GBytes   549 Mbits/sec    0             sender
[  7]   0.00-20.01  sec  1.28 GBytes   547 Mbits/sec                  receiver
[  9]   0.00-20.00  sec  1.28 GBytes   549 Mbits/sec    0             sender
[  9]   0.00-20.01  sec  1.27 GBytes   547 Mbits/sec                  receiver
[ 11]   0.00-20.00  sec  1.33 GBytes   573 Mbits/sec    1             sender
[ 11]   0.00-20.01  sec  1.33 GBytes   572 Mbits/sec                  receiver
[SUM]   0.00-20.00  sec  5.16 GBytes  2.22 Gbits/sec    1             sender
[SUM]   0.00-20.01  sec  5.15 GBytes  2.21 Gbits/sec                  receiver

Still a bit less than the other image. I wonder if there’s a network driver tweak or something that’s missing?

3

We use the exact same kernel and driver as vanilla FriendlyELEC. The only different could be in the firmware. We use Debian’s firmware packages, while FriendlyELEC ships additional firmware files scattered and unpackaged across their image.

We should check /lib/firmware for the exact firmware loaded by the Ethernet driver, and compare it on both images. Also the kernel log might contain this info:

dmesg

Ah, I do not think that it can have an impact on this parcitular benchmark, but did you test our Bullseye image? Index of /downloads/images

4

It seems that the loaded firmware is different - see below.

Dietpi bookworm (current install)

[    5.536433] r8169 0004:41:00.0: can't read MAC address, setting random one
[    5.542964] r8169 0004:41:00.0 eth1: RTL8125B, ee:30:4c:9f:d4:ac, XID 641, IRQ 127
[    5.542972] r8169 0004:41:00.0 eth1: jumbo features [frames: 9194 bytes, tx checksumming: ko]
[    5.543148] r8169 0003:31:00.0: enabling device (0000 -> 0003)
[    5.556701] rtc-hym8563 6-0051: rtc information is invalid
[    5.563419] rtc-hym8563 6-0051: registered as rtc0
[    5.563971] rtc-hym8563 6-0051: setting system clock to 2021-01-01T12:00:00 UTC (1609502400)
[    5.566226] systemd-journald[373]: Time jumped backwards, rotating.
[    5.605837] rk_gmac-dwmac fe1c0000.ethernet eth0: PHY [stmmac-1:01] driver [RTL8211F Gigabit Ethernet] (irq=POLL)
[    5.608386] dwmac4: Master AXI performs any burst length
[    5.608399] rk_gmac-dwmac fe1c0000.ethernet eth0: No Safety Features support found
[    5.608411] rk_gmac-dwmac fe1c0000.ethernet eth0: IEEE 1588-2008 Advanced Timestamp supported
[    5.608503] rk_gmac-dwmac fe1c0000.ethernet eth0: registered PTP clock
[    5.608631] rk_gmac-dwmac fe1c0000.ethernet eth0: configuring for phy/rgmii-rxid link mode
[    5.647058] r8169 0003:31:00.0 eth2: RTL8125B, a2:eb:0b:fd:fb:9e, XID 641, IRQ 128
[    5.647062] r8169 0003:31:00.0 eth2: jumbo features [frames: 9194 bytes, tx checksumming: ko]
[    5.651791] r8125: loading out-of-tree module taints kernel.
[    6.006263] RTL8226B_RTL8221B 2.5Gbps PHY r8169-3-3100:00: attached PHY driver [RTL8226B_RTL8221B 2.5Gbps PHY] (mii_bus:phy_addr=r8169-3-3100:00, irq=IGNORE)
[    6.286510] r8169 0003:31:00.0 eth2: Link is Down
[    6.325956] RTL8226B_RTL8221B 2.5Gbps PHY r8169-4-4100:00: attached PHY driver [RTL8226B_RTL8221B 2.5Gbps PHY] (mii_bus:phy_addr=r8169-4-4100:00, irq=IGNORE)
[    6.606585] r8169 0004:41:00.0 eth1: Link is Down
[    9.697721] r8169 0003:31:00.0 eth2: Link is Up - 2.5Gbps/Full - flow control rx/tx

root@in:~# ethtool -i eth2
driver: r8169
version: 5.10.110
firmware-version: rtl8125b-2_0.0.2 07/13/20
expansion-rom-version:
bus-info: 0003:31:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no

root@in:~# modinfo r8169
filename:       /lib/modules/5.10.110/r8169.ko
firmware:       rtl_nic/rtl8125b-2.fw
firmware:       rtl_nic/rtl8125a-3.fw
firmware:       rtl_nic/rtl8107e-2.fw
firmware:       rtl_nic/rtl8107e-1.fw
firmware:       rtl_nic/rtl8168fp-3.fw
firmware:       rtl_nic/rtl8168h-2.fw
firmware:       rtl_nic/rtl8168h-1.fw
firmware:       rtl_nic/rtl8168g-3.fw
firmware:       rtl_nic/rtl8168g-2.fw
firmware:       rtl_nic/rtl8106e-2.fw
firmware:       rtl_nic/rtl8106e-1.fw
firmware:       rtl_nic/rtl8411-2.fw
firmware:       rtl_nic/rtl8411-1.fw
firmware:       rtl_nic/rtl8402-1.fw
firmware:       rtl_nic/rtl8168f-2.fw
firmware:       rtl_nic/rtl8168f-1.fw
firmware:       rtl_nic/rtl8105e-1.fw
firmware:       rtl_nic/rtl8168e-3.fw
firmware:       rtl_nic/rtl8168e-2.fw
firmware:       rtl_nic/rtl8168e-1.fw
firmware:       rtl_nic/rtl8168d-2.fw
firmware:       rtl_nic/rtl8168d-1.fw
license:        GPL
softdep:        pre: realtek
description:    RealTek RTL-8169 Gigabit Ethernet driver
author:         Realtek and the Linux r8169 crew <netdev@vger.kernel.org>
srcversion:     E2CCA0977608DE3402B936D
alias:          pci:v000010ECd00003000sv*sd*bc*sc*i*
alias:          pci:v000010ECd00008125sv*sd*bc*sc*i*
alias:          pci:v00000001d00008168sv*sd00002410bc*sc*i*
alias:          pci:v00001737d00001032sv*sd00000024bc*sc*i*
alias:          pci:v000016ECd00000116sv*sd*bc*sc*i*
alias:          pci:v00001259d0000C107sv*sd*bc*sc*i*
alias:          pci:v00001186d00004302sv*sd*bc*sc*i*
alias:          pci:v00001186d00004300sv*sd*bc*sc*i*
alias:          pci:v00001186d00004300sv00001186sd00004B10bc*sc*i*
alias:          pci:v000010ECd00008169sv*sd*bc*sc*i*
alias:          pci:v000010FFd00008168sv*sd*bc*sc*i*
alias:          pci:v000010ECd00008168sv*sd*bc*sc*i*
alias:          pci:v000010ECd00008167sv*sd*bc*sc*i*
alias:          pci:v000010ECd00008162sv*sd*bc*sc*i*
alias:          pci:v000010ECd00008161sv*sd*bc*sc*i*
alias:          pci:v000010ECd00008136sv*sd*bc*sc*i*
alias:          pci:v000010ECd00008129sv*sd*bc*sc*i*
alias:          pci:v000010ECd00002600sv*sd*bc*sc*i*
alias:          pci:v000010ECd00002502sv*sd*bc*sc*i*
alias:          pci:v000010ECd00000000sv*sd*bc*sc*i*
depends:
intree:         Y
name:           r8169
vermagic:       5.10.110 SMP mod_unload modversions aarch64

and now friendlyelec img

nanopi-r6s friendlyelec img

[    8.989471] r8125: loading out-of-tree module taints kernel.
[    8.991466] r8125 2.5Gigabit Ethernet driver 9.010.01-NAPI loaded
[    8.991541] r8125 0004:41:00.0: enabling device (0000 -> 0003)
[    9.008856] r8125 0004:41:00.0 (unnamed net_device) (uninitialized): Invalid ether addr 00:00:00:00:00:00
[    9.008872] r8125 0004:41:00.0 (unnamed net_device) (uninitialized): Random ether addr 0a:36:fb:22:ee:da
[    9.009313] r8125: This product is covered by one or more of the following patents: US6,570,884, US6,115,776, and US6,327,625.
[    9.011333] r8125  Copyright (C) 2022 Realtek NIC software team <nicfae@realtek.com>
                This program comes with ABSOLUTELY NO WARRANTY; for details, please see <http://www.gnu.org/licenses/>.
                This is free software, and you are welcome to redistribute it under certain conditions; see <http://www.gnu.org/licenses/>.
[    9.011499] r8125 2.5Gigabit Ethernet driver 9.010.01-NAPI loaded
[    9.011559] r8125 0003:31:00.0: enabling device (0000 -> 0003)
[    9.029174] r8125: This product is covered by one or more of the following patents: US6,570,884, US6,115,776, and US6,327,625.
[    9.031193] r8125  Copyright (C) 2022 Realtek NIC software team <nicfae@realtek.com>
                This program comes with ABSOLUTELY NO WARRANTY; for details, please see <http://www.gnu.org/licenses/>.
                This is free software, and you are welcome to redistribute it under certain conditions; see <http://www.gnu.org/licenses/>.

root@NanoPi-R6S:/home/pi# ethtool -i eth2
driver: r8125
version: 9.010.01-NAPI
firmware-version:
expansion-rom-version:
bus-info: 0003:31:00.0
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no

root@NanoPi-R6S:/home/pi# modinfo r8125
filename:       /lib/modules/5.10.110/extra/r8125.ko
version:        9.010.01-NAPI
license:        GPL
description:    Realtek RTL8125 2.5Gigabit Ethernet driver
author:         Realtek and the Linux r8125 crew <netdev@vger.kernel.org>
srcversion:     05603FAA2BFC944987305A4
alias:          pci:v000010ECd00003000sv*sd*bc*sc*i*
alias:          pci:v000010ECd00008162sv*sd*bc*sc*i*
alias:          pci:v000010ECd00008125sv*sd*bc*sc*i*
depends:
name:           r8125
vermagic:       5.10.110 SMP mod_unload modversions aarch64
parm:           speed_mode:force phy operation. Deprecated by ethtool (8). (uint)
parm:           duplex_mode:force phy operation. Deprecated by ethtool (8). (uint)
parm:           autoneg_mode:force phy operation. Deprecated by ethtool (8). (uint)
parm:           advertising_mode:force phy operation. Deprecated by ethtool (8). (uint)
parm:           aspm:Enable ASPM. (int)
parm:           s5wol:Enable Shutdown Wake On Lan. (int)
parm:           s5_keep_curr_mac:Enable Shutdown Keep Current MAC Address. (int)
parm:           rx_copybreak:Copy breakpoint for copy-only-tiny-frames (int)
parm:           use_dac:Enable PCI DAC. Unsafe on 32 bit PCI slot. (int)
parm:           timer_count:Timer Interrupt Interval. (int)
parm:           eee_enable:Enable Energy Efficient Ethernet. (int)
parm:           hwoptimize:Enable HW optimization function. (ulong)
parm:           s0_magic_packet:Enable S0 Magic Packet. (int)
parm:           tx_no_close_enable:Enable TX No Close. (int)
parm:           enable_ptp_master_mode:Enable PTP Master Mode. (int)
parm:           disable_pm_support:Disable PM support. (int)
parm:           debug:Debug verbosity level (0=none, ..., 16=all) (int)

root@NanoPi-R6S:/home/pi#
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-20.00  sec  1.40 GBytes   600 Mbits/sec                  receiver
[  8]   0.00-20.00  sec  1.37 GBytes   590 Mbits/sec                  receiver
[ 10]   0.00-20.00  sec  1.39 GBytes   595 Mbits/sec                  receiver
[ 12]   0.00-20.00  sec  1.32 GBytes   568 Mbits/sec                  receiver
[SUM]   0.00-20.00  sec  5.48 GBytes  2.35 Gbits/sec                  receiver
-----------------------------------------------------------

do you still want me to try vanilla bullseye dietpi image too? I think with the data above its clear there’s different firmwares used

This firmware seems to be available from immortalwrt (who often provide optimized openwrt images for nanopi) https://sources.immortalwrt.org/:

5

Any pointers or suggestions, could we get the dietpi image to use the better driver by default than the current driver (seems dated from 2020)?

6

@TheLinuxGuy - Did you resolve this? I am looking at possibly building a dietpi image with a different driver for the R2S Plus.

7

R2S and R6S are a completely different thing, zero relation, zero similarities :wink:. If you have bad Ethernet performance with our R2S image compared to FriendlyELEC’s, please open a new topic about this. We use a mainline kernel instead of FriendlyELEC’s ancient but tailored vendor kernel. So quite possible that some tweaks have not been upstreamed.

8

correct - just looking for the same cookbook of changing the driver of a dietpi image over to one that works in the friendlyelec build.

From a working friendlywrt build:

root@r2splus:~# ethtool -i eth0
driver: st_gmac
version: 6.1.63
firmware-version: 
expansion-rom-version: 
bus-info: 
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: yes
supports-priv-flags: no

root@r2splus:~# ethtool -i eth1
driver: r8152
version: v1.12.13
firmware-version: rtl8153b-2 v1 10/23/19
expansion-rom-version: 
bus-info: usb-xhci-hcd.0.auto-1
supports-statistics: yes
supports-test: no
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: no

root@r2splus:~# modinfo st_gmac
modinfo: ERROR: Module st_gmac not found.

root@r2splus:~# modinfo r8152
filename:       /lib/modules/6.1.63/r8152.ko
version:        v1.12.13
license:        GPL
description:    Realtek RTL8152/RTL8153 Based USB Ethernet Adapters
author:         Realtek linux nic maintainers <nic_swsd@realtek.com>
firmware:       rtl_nic/rtl8156b-2.fw
firmware:       rtl_nic/rtl8156a-2.fw
firmware:       rtl_nic/rtl8153c-1.fw
firmware:       rtl_nic/rtl8153b-2.fw
firmware:       rtl_nic/rtl8153a-4.fw
firmware:       rtl_nic/rtl8153a-3.fw
firmware:       rtl_nic/rtl8153a-2.fw

Maybe the other advice I was getting about trying to use eth1 as the WAN port will get me started.