I’ve got 3 systems running v7.4.2 of Dietpi (1x x86, 2x Pi4) and since updating to 7.4.2 I’m now getting really laggy CLI over SSH from a variety of different clients. OpenSSH is running on all 3. Both exhibit the same lagginess. A reboot resolves the problem for about a day but then it slowly returns over a 24hr period until the system becomes so laggy its unusable. Local CLI is fine, its only over SSH.
All devices are ethernet attached with a single connection. WiFi disabled.
I’ve switched sshd to Dropbear and the session is still laggy and unresponsive for many seconds sometimes. Reboot is the only thing that resolves it…for a few hours.
Edit: I’ve just used dietpi-services to bounce all the services and this seems to have a positive effect as it restores the responsiveness of an SSH session…but then the laggy SSH problem slowly returns.
I did some DuckDuckFu and I found a post saying that adding ‘UseDNS no’ to /etc/ssh/sshd_config helps. I tried it and was about to post that that fixed it. Alas it hasn’t but it has improved the lag a bit.
If I tell Dietpi-services to stop everything, the lag problem is resolved. I’m going to systematically start each service one by one to see if I can narrow things down.
I’ve done some more digging and this is a more low level issue than I thought. Its network packet loss. Running a ping alongside an SSH session, the SSH lag corresponds to the packet loss. Both my 2x Pi4 and 1 x86 box all connected to the same switch via ethernet. I have swapped out cables and the switch but this does not resolve the problem.
The root of the problem seems to be with Apps that are configured to use an NFS and/or SMB mount as their target directory or working directory. All works fine after a reboot but within hours, when these Apps start to use their network mounts, network connectivity starts to become intermittent and packet loss occurs. To be clear, this is not the Apps saturating the network link, the packet loss occurs when the Apps are idle and just polling the network mounts. Apps include nzbget, qbittorrent, PlexMediaCenter. If I stop these Apps using Dietpi-Services the packet loss is immediately gone. When I start them, the packet loss returns.
Eth0 is only about 3% utilised during the packet loss.
I’ve narrowed things down further:
On the x86 box qbittorrent is the culprit. If I stop the service the packet loss disappears. If it has no active jobs, there is no packet loss. As soon as you start a torrent with the default connection settings, the packet loss starts. I moved all Torrents activity off the NFS mount to a local disk and this did not make a difference. I found a guide on tuning the qBittorrent connections settings are now the packet loss is gone. Perhaps local system resources were impacted with the default Connection settings? .
On the Pi4, the culprit is PlexMediaServer. If I stop the service using Dietpi-Services, the packet loss disappears. If I restart it, the packet loss returns after about 20mins. PMS is configured to look at a locally mounted SMB share on a NAS for its content.
OK, I’ve got this sussed now. I don’t know how or why this conflict exists but PMS likes to broadcast its presence on the LAN it’s connected to on a regular basis. My Draytek Router has IGMP Snooping enabled. If I turn that off, all the packet loss problems are gone. Turn it back on, they return.
I don’t know exactly what PMS is doing without reviewing a packet trace but its interfering with IGMP. Thankfully I don’t need any IGMP so I’ve disabled it to resolve the issue.
Yes, agreed. Not a DietPi / Debian problem. DrayTek routers are very feature rich afford enterprise like devices but I accept they are niche devices for use in the home. Scanning through the official Plex Forums, I can see IGMP mentioned a lot. Hopefully this thread will help others if they were to hit the same issue.
I will mark one of your last statements as solution to make it clear for other where the challenge is coming from (even if not fully solved from our side)
Interesting indeed. I just started to read a bit deeper into broadcast and multicast traffic when learning that DLNA (e.g. ReadyMedia) uses broadcast traffic and the question how to route such through a VPN for remote DLNA communication.
IGMP is a multicast group management protocol while Plex likely uses some mDNS (multicast DNS, like Avahi, using UDP port 5353) to broadcast its hostname, but that shouldn’t interfere if an IGMP-capable router . A little search reveals that it also uses SSDP for local network discovery (also via multicast, via UDP port 1900, the same which DLNA uses, as UPnP, both are based on). And then probably Plex also has some video broadcasting features?
tcpdump could help to catch all packets, what is coming from where in which frequency. Probably there is some loop .
