Quartz64 Model A: Very slow eth0 network

Troubleshooting
21

Hi everyone,

I did some more digging and tried to understand when the problem is occuring. So I wrote a script to bring eth0 down and up again and then test the connection:

for I in {1..30}
do
	dateActuelle=$(date)
	echo "$I at $dateActuelle" >> $logfile
	echo "--- IFDOWN ---" >> $logfile
	ifdown eth0 >> $logfile 2>> $logfile
	echo "--- IFUP ---" >> $logfile
	ifup eth0 >> $logfile 2>> $logfile
	echo "--- SPEEDTEST ---" >> $logfile
	speedtest-cli --secure >> $logfile 2>> $logfile
	echo "--- ETHTOOL ---" >> $logfile
	ethtool eth0 >> $logfile 2>> $logfile

	sleep $((random % 60000))
done

And for the good order, I replaced my current router (EdgeRouter X) with the standard router of my provider. So I had 30 runs with my own router and 30 with the provider’s one.

Results:

  • 60 attempts total
  • 26 with bad speeds
  • 18 with good speeds
  • 16 with errors

Lessons learned:

  • Replacing the router didn’t change anything;
  • Almost every time (25 of the 26) I had bad speeds, the master-slave status was “slave”;
  • Everytime I had decent speeds (18), the master-slave status was “master”;
  • 14 times the “ifup” command gave “No DHCPOFFERS received” error and the interface was up but unusable;
  • 2 times “ifup” looks OK but the speedtest command gave “urlopen error [Errno -3] Temporary failure in name resolution>”

Definitely looks like something incorrect is going on in the PHY. Looked up some documentation and found the Kernel docs saying something that caught my attention ( Diagnostic Concept for Investigating Twisted Pair Ethernet Variants at OSI Layer 1 — The Linux Kernel documentation )

About combined negotiation mode:

Some PHYs may not work reliably if the link partner switches from one forced mode to another. In this case, only a down/up cycle may help.

Could something like this be the origin of the problem? I checked in the logs, every time ethtool mentioned autonegotiation “on” in full duplex.

Thanks for your help.

22

The kernel docs talk about a situation where the master/slave role switches mid operation, AFAIU. In your case, this does not seem to happen, but weirdly the SBC does not get the same role on every negotiation, but ends up as master or as slave randomly, even that it is set to preferred slave.

I would need to check again with my Quartz64, but all currently attached devices are in slave mode, i.e. the router Ethernet ports always take the master role, which is AFAIK also common.

Interestingly exactly this more common result is causing issues in your case, hence your Quartz64 seems to have issues following the clock timing of the router. Instead it works better when its own oscillator is the timing reference for the connection. Generally, both should work equally well, as long as there is no hardware damage. Since you tested it with two routers, a damage at the router side is ruled out.

If you want to force the Quartz64 to be in master mode, you need to disable autonegotiation I guess:

sudo ethtool -s eth0 speed 1000 duplex full autoneg off master-slave forced-master

The correct value is forced-master rather than master.

23

Hi,

I tried to put it in forced-master mode with autonegotiation “off” but that didn’t work either:

netlink error: master/slave configuration not supported by device (offset 60)
netlink error: Operation not supported

This is supposed to work doesn’t it?

Then tried to get the interface back up but that didn’t work either. I first get the next lines:

rk_gmac_dwmac fe010000.ethernet eth0: Failed to reset the dma
rk_gmac_dwmac fe010000.ethernet eth0: stmmac_hw_setup: DMA engine initialization failed
rk_gmac_dwmac fe010000.ethernet eth0: __stmmac_open: hw setup failed
RTNETLINK answers: Connection timed out

Then I get the following error a few times

DHCPDISCOVER on eth0 to 255.255.255.255 port 67 interval ..
send_packet: Network is down
dhclient.c:2600: Failed to send 300 bytes long packet over eth0 interface.
receive_packet failed on eth0: Network is down

And at the end

No DHCPOFFERS received.
No working leases in persistent database - sleeping

Had to restart the server to restore connection.

You say “[…] as long as there is no hardware damage.”. How can I know it is a Hardware problem (or exclude it is a software one)?

Thanks,

Dietpi64

24

First of all, I just verified that my board connects in slave mode:

master-slave cfg: preferred slave
master-slave status: slave

I guess my router has preferred master, and this should be the most common case.

Switching to master mode worked well:

root@SOQuartz:~# sudo ethtool -s eth0 speed 1000 duplex full autoneg off master-slave forced-master
root@SOQuartz:~# ethtool eth0
Settings for eth0:
        Supported ports: [ TP    MII ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Advertised pause frame use: Symmetric Receive-only
        Advertised auto-negotiation: No
        Advertised FEC modes: Not reported
        Speed: 1000Mb/s
        Duplex: Full
        Auto-negotiation: off
        master-slave cfg: forced master
        master-slave status: master
        Port: Twisted Pair
        PHYAD: 0
        Transceiver: external
        MDI-X: Unknown
        Supports Wake-on: ug
        Wake-on: d
        Current message level: 0x0000003f (63)
                               drv probe link timer ifdown ifup
        Link detected: yes

This is the SOQuartz again, not the Model A, but it uses the Model A baseboard, i.e. identical Ethernet. To compare the kernel log:

[    2.508226] rk_gmac-dwmac fe010000.ethernet: IRQ sfty not found
[    2.508970] rk_gmac-dwmac fe010000.ethernet: clock input or output? (input).
[    2.509628] rk_gmac-dwmac fe010000.ethernet: TX delay(0x30).
[    2.510133] rk_gmac-dwmac fe010000.ethernet: RX delay(0x10).
[    2.510639] rk_gmac-dwmac fe010000.ethernet: integrated PHY? (no).
[    2.511206] rk_gmac-dwmac fe010000.ethernet: clock input from PHY
[    2.516760] rk_gmac-dwmac fe010000.ethernet: init for RGMII
[    2.518741] rk_gmac-dwmac fe010000.ethernet: User ID: 0x30, Synopsys ID: 0x51
[    2.519379] rk_gmac-dwmac fe010000.ethernet:         DWMAC4/5
[    2.519839] rk_gmac-dwmac fe010000.ethernet: DMA HW capability register supported
[    2.520501] rk_gmac-dwmac fe010000.ethernet: Active PHY interface: RGMII (1)
[    2.521142] rk_gmac-dwmac fe010000.ethernet: RX Checksum Offload Engine supported
[    2.521805] rk_gmac-dwmac fe010000.ethernet: TX Checksum insertion supported
[    2.522426] rk_gmac-dwmac fe010000.ethernet: Wake-Up On Lan supported
[    2.523052] rk_gmac-dwmac fe010000.ethernet: TSO supported
[    2.523541] rk_gmac-dwmac fe010000.ethernet: Enable RX Mitigation via HW Watchdog Timer
[    2.524249] rk_gmac-dwmac fe010000.ethernet: Enabled RFS Flow TC (entries=10)
[    2.524880] rk_gmac-dwmac fe010000.ethernet: TSO feature enabled
[    2.525429] rk_gmac-dwmac fe010000.ethernet: Using 32/32 bits DMA host/device width
[  139.847602] rk_gmac-dwmac fe010000.ethernet eth0: Register MEM_TYPE_PAGE_POOL RxQ-0
[  139.850073] rk_gmac-dwmac fe010000.ethernet eth0: PHY [stmmac-0:00] driver [YT8511 Gigabit Ethernet] (irq=POLL)
[  139.861115] rk_gmac-dwmac fe010000.ethernet eth0: No Safety Features support found
[  139.861610] rk_gmac-dwmac fe010000.ethernet eth0: IEEE 1588-2008 Advanced Timestamp supported
[  139.861950] rk_gmac-dwmac fe010000.ethernet eth0: registered PTP clock
[  139.861966] rk_gmac-dwmac fe010000.ethernet eth0: configuring for phy/rgmii link mode
[  143.946689] rk_gmac-dwmac fe010000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off
[  245.105693] rk_gmac-dwmac fe010000.ethernet eth0: Link is Down
[  249.194488] rk_gmac-dwmac fe010000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off

Most of my test devices use WiFi for network access, Ethernet has been connected later, an ip l s eth0 up to bring up the interface at 139s, the switch no slave mode at 245s, which seems to imply a reconfiguration of the interface. Obviously, no IP is assigned, no DHCP used on that interface, aside of SLAAC IPv6 LLA and GUA.

Maybe start similarly, with Ethernet deconfigured entirely at boot, just to rule out that the configuration/DHCP somehow cause changes to fail afterwards.

I don’t know for sure. But you say you tested it with two different routers, I do not believe that a cable damage can cause these symptoms with transfer speeds depending on master vs slave mode, and it works well here with the same SBC, kernel, and userland.

Ah, btw the link needs to be up to change operation mode. So if you did ifdown eth0 or so before attempting this, bring the base link power back up with ip l s eth0 up.

25

Hi,

I rebooted until I had a bad connection (1Mb/s) and without putting the interface down I was able to force it to be master. After that, the connection is magically getting speeds to 200Mb/s.

Glad to have a workaround! But still frustrating not knowing what the origin of the bug is.

@MichaIng you said to test with Ethernet entirely deconfigured at boot. Do you mean the deactivation option in dietpi-config or something else?

I don’t use wifi on my setup so that one is always deactivated.

26

Right. But it worked as well using the Ethernet adapter for Internet access some days ago. I was just too lazy to switch back and forth today. However, attempting this your end was mostly meant as attempt to solve switching from slave to master mode. But that works now. Good that it also works mid operation. That this might cause issues with some adapters is how I understood the kernel docs you linked.

I do still wonder why it uses master mode OOTB in some cases, despite preferred slave. With the other router, the performance was bad on every boot, wasn’t it? Maybe it is just coincidence that the new router has preferred slave as well, so that it is 50:50 chance now who is taking which role.

So to make the workaround persistent, you could add a line to the iface eth0 block in /etc/network/interfaces:

up ethtool -s eth0 speed 1000 duplex full autoneg off master-slave forced-master

So this is done automatically whenever the interface is configured. up, opposed to pre-up, runs after the interface has been brought up, to assure master-slave role switch works.

27

There was no real difference between both routers. Here are screenshots of the results:

Edgerouter X:

image
image1418×1340 215 KB

And the default router (Experia box V10A):

image
image2196×1337 333 KB

So both routers had approximately the same behavior:

  • 1/3 chances to be master and work well
  • 1/3 chances to be slave and to be too slow
  • 1/3 chances to encounter an error (mostly DHCPOFFER but sometimes just failure to resolve a domain name during the speedtest)

I’ll try with the ethernet shutdown, reboot and then activation of ethernet. See is that comes out consistently.

28

I wonder whether the issue is not particularly the slave mode, but at a lower level, something which affects the ability to autonegotiate and adapt to the clock signal of the other side properly. Or it sometimes chooses master mode in the first place, because it has issues noticing the timing of the router somehow. In any case some issue with the PHY where all this timing stuff happens.

This does not help, does it?

sudo ethtool --reset phy eth0

Do you see any difference in dmesg | grep 'rk_gmac-dwmac' between good (master) an bad (slave) boots, also compared to my output above?

I also recognize now that you do get bad speedtest download speeds as well in slave mode, while RX was fine when testing with iperf3 (Quartz64 as server)? EDIT: Most likely due to ACK packet loss in the other direction. So not plain RX is bad, but real downloads which do still include TX packets, and servers reacting to that.

29

Problem with the PHY seems plausible. I was no able to do the command you gave though (bad command line argument). Saw in the manpage that the order should be reversed:

sudo ethtool --reset eth0 phy

So I tried that and computer says no:

ETHTOOL_RESET 0x40
Cannot issue ETHTOOL_RESET: Operation not supported

Do I have to bring the link down to reset it?

The comparison between your dmesg output and both mines showed:

  • It doesn’t matter what the slave-status is, both give exactly the same output in dmesg
  • Compared to yours, only the last lines difference.

Where you had

rk_gmac-dwmac fe010000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off
rk_gmac-dwmac fe010000.ethernet eth0: Link is Down
rk_gmac-dwmac fe010000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off

I have

rk_gmac-dwmac fe010000.ethernet eth0: Link is Up - 1Gbps/Full - flow control rx/tx

And yes that’s right. When the connection is bad (so slave), speedtest gives bad speeds on both directions whereas the iperf test is only bad when the Quartz is the client.

30

Right, ethtool --reset eth0 phy is the correct argument order.

I did not test it either, just try it out.

Did you already test disabling flow control?

sudo ethtool -A eth0 rx off tx off

This is autonegotiated, so it would need to be applied along with autoneg off to remain effective, just like link speed, forced-master etc. And similarly, for testing, it might be best to change while the link is up, but network not further configured yet.

31

First I disabled flow control and removed the master-slave preference:

sudo ifdown eth0
sudo ip l s eth0 up
sudo ethtool -s eth0 speed 1000 duplex full autoneg off
sudo ethtool -A eth0 rx off tx off
sudo ifup eth0

A check with dmesg came up with the following:

[235022.258510] rk_gmac-dwmac fe010000.ethernet eth0: Link is Down
[235074.245295] rk_gmac-dwmac fe010000.ethernet eth0: Register MEM_TYPE_PAGE_POOL RxQ-0
[235074.247473] rk_gmac-dwmac fe010000.ethernet eth0: PHY [stmmac-0:00] driver [YT8511 Gigabit Ethernet] (irq=POLL)
[235074.263976] rk_gmac-dwmac fe010000.ethernet eth0: No Safety Features support found
[235074.264018] rk_gmac-dwmac fe010000.ethernet eth0: IEEE 1588-2008 Advanced Timestamp supported
[235074.264355] rk_gmac-dwmac fe010000.ethernet eth0: registered PTP clock
[235074.264366] rk_gmac-dwmac fe010000.ethernet eth0: configuring for phy/rgmii link mode
[235077.329568] rk_gmac-dwmac fe010000.ethernet eth0: Link is Up - 1Gbps/Full - flow control rx/tx
[235251.714945] rk_gmac-dwmac fe010000.ethernet eth0: Link is Down
[235254.769388] rk_gmac-dwmac fe010000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off

So flow control has been disabled. Unfortunately, I checked the speed and it was bad. So I tried the same commands this time adding the master-slave preference:

sudo ifdown eth0
sudo ip l s eth0 up
sudo ethtool -s eth0 speed 1000 duplex full autoneg off master-slave forced-master
sudo ethtool -A eth0 rx off tx off
sudo ifup eth0

A check with dmesg came up with the following:

[235856.906827] rk_gmac-dwmac fe010000.ethernet eth0: Link is Down
[235859.959841] rk_gmac-dwmac fe010000.ethernet eth0: Register MEM_TYPE_PAGE_POOL RxQ-0
[235859.962231] rk_gmac-dwmac fe010000.ethernet eth0: PHY [stmmac-0:00] driver [YT8511 Gigabit Ethernet] (irq=POLL)
[235859.971979] rk_gmac-dwmac fe010000.ethernet eth0: No Safety Features support found
[235859.972028] rk_gmac-dwmac fe010000.ethernet eth0: IEEE 1588-2008 Advanced Timestamp supported
[235859.972401] rk_gmac-dwmac fe010000.ethernet eth0: registered PTP clock
[235859.972415] rk_gmac-dwmac fe010000.ethernet eth0: configuring for phy/rgmii link mode
[235863.025529] rk_gmac-dwmac fe010000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off
[235877.938470] rk_gmac-dwmac fe010000.ethernet eth0: Link is Down
[235882.033482] rk_gmac-dwmac fe010000.ethernet eth0: Link is Up - 1Gbps/Full - flow control off

Same, control flow has been disabled. Only this time the speed is nice and fast (because of the master status I suppose).

So my guess would be that the speed isn’t linked to the flow control being activated or not.