SSH-Client problem (Long delay before password prompt/login) -> Problem found, but not solved

Hi all
I have a strange problem and can’t find a fix. Maybe someone has an idea which points me in the right direction:
I have some Pi ZeroWs which sends commands over ssh. For example a Pi ZeroW with RFID-Reader. If it reads a badge, it send “access granted” over ssh.
The problem is now that i have a delay from about 10s(!) until the password prompt came up. The good thing is that i have two ZeroW. One works fine and the other not! Means on one device the prompt came up in let’s say 2s and on the other device it needs 10s. The server is always the same.
With this fact you may think it should be easy to find the problem. But believe me, its not easy!
I set up a new ZeroW with actual dietpi v8.24.1 which has also a delay of 10s. I also compared “ssh -vvv” which looks similar. I compared the modules but can’t find any differences.
It hangs on this message (ssh -vvv):

debug1: kex: algorithm: sntrup761x25519-sha512@openssh.com
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: compression: none

During this step, the CPU-Load goes up to 100%. And i think this is the problem why it took so long! (specially on the pi zero)
For me it looks like there is something like “hardware acceleration” missing for sha or something like that. But the module “sha256_generic” and “libsha256” is loaded. On the working device, i can not see such high cpu load during login.

I hope someone can point me in the right direction, because 10s delay is horrible for sending commands! :wink:

Some additional information to test this if you have a pi zerow:
SSH into your pi zerow and after successful login connect to another server on your network:

ssh root@192.168.1.1

Look how long it took until the password prompt is shown.

Edit:
I of course compared the versions of ssh, and the version number is the same.
But:

Working: OpenSSH_9.2p1 Raspbian-2, OpenSSL 3.0.11 19 Sep 2023
Problem: OpenSSH_9.2p1 -2+deb12u1, OpenSSL 3.0.11 19 Sep 2023

So i copied the “ssh” from the working one to the non working and → YES!!! That was the problem!
So it looks like the “Raspbian-2” version has some “Hardware-Accelerration” or something like that in it which reduces cpu load and speed up the login process (and maybe more?).
But i can not understand why one device has this version and the other has not the same. Because both uses dietpi v8.24.1.
Is this a bug?

I found out that this happens after updating from dietpi v8.23.3 to 8.24.1. I ask me if there are other packages with the same problem (changed from “Raspbian-2” to “-2+deb12u1”). And why does this happen? Hast the source changed? Should i open a git ticket for this problem?

You are talking about the OpenSSH Client package? We don’t maintain these packages ourselves. On RPi ZeroW we use official Raspbian apt package source. Index of /raspbian/pool/main/o/openssh

You can check all packages sources as follow

for i in /etc/apt/sources.list{,.d/*.list}; do echo "$i:"; cat "$i"; done

As far as I can seem, OpenSSH client package has been updated on Oct 9th and is named openssh-client_9.2p1-2+deb12u1_armhf.deb

If there is an issue with this package, it might be needed to check with Raspbian guys.

Yes, correct. I am talking about the OpenSSH Client package. As you can see in my first post, the version is the same, but one is named “Raspbian-2” and the other is “-2+deb12u1”.
The “Raspbian-2” is working well, but the “-2+deb12u1” not. It produces high cpu load and a delay of about 10s to login to SSH-Server on pi zeroW. In my setup i use this to first “ask” the status, and then “set” the new status over SSH. So it needs to connect two times to the SSH-Server. With the “Raspbian-2” version this took about 4s (more or less), but with the “-2+deb12u1” it took about 20s! I think because the pi zeroW has only a single core cpu. On a pi4 it has no delay because it has enough cpu power. So this “bug” can only be spotted on pi zero.
Here is a video with working and non working version:

As you can see it took VERY long until the login is shown.

As stated above already, we don’t maintain this package. It’s the original one from Raspbian repository.

Yes i think you are right. As i can see the source has not changed, but the package has. I can not understand why the package has changed but has the same version number. Maybe raspian joined there source with the new bookworm from the orignal raspberry repository? Before it was a raspian package, and now it looks like a debian package.