SSH - Never get to prompt !

SSH - dropbear acting up again!

Putty cannot connect, “network error: Connection timed out”

Here’s the output of my console after a restart of dropbear


You can see tcpdump is showing traffic on port 22 to my PC: 192.168.10.143. And the status is ok.

I did the ss -tulpn again after and like trendy is mentioning, the Send-Q for port 22 is 1000.

According to Wikipedia: Send-Q The count of bytes not acknowledged by the remote host.

  • While this is going on, my PC is running a continuous ping to the PiZero with no loss.

So this is a Raspberry Pi Zero W (wifi), why would I have this issue after a while when it worked for a few days without and issue?

The last thing I did before getting there was to update/add some libraries to Node-Red. Which works fine.

Like I stated before, this Wi-Fi is solid with 4 people working remotely over it.

One hour later… I get a prompt, I can enter userid/pass… but never get the prompt. TCPDUMP shows that the PiZ receives the packet if I press enter in putty, but I get no prompt after entering the password.

Weird!

A status on the service says that the password auth succeeded for root.

Then Putty times out and abort.

If I restart dropbear, I get back to the issue with no ask for user/pass.

There is something wrong here.
In the third line of the tcpdump the putty client is sending a reset packet, although it should be sending an ack.
In the next lines the dropbear is sending synack to putty, but putty seems to be ignoring them and keeps resending syn.
It looks to me that putty is not receiving the packets from dropbear, hence the high send-q number.

probably good to user wireshark on client side to see what is going on there. Maybe this will complete the picture.

12h later, after a night’s sleep! All is ok, I can log in through dropbear and get a prompt.

Seems that every time I touch something on this Pi Zero, it takes 12h to stabilize and work!

Oh! Talking to fast here… I get a prompt but the prompt is frozen.

If in putty I press enter, I see on the console of the PiZero tcpdump an SEQ from putty and an ACK from dropbear. But nothing happens on the putty side. So I’m connected but I don’t get any response back in Putty!

Suddenly, the command I typed in putty, appeared like 2mins after. So I get something but goes through a wormhole to the alpha quadrant and back! :wink:

Looking at htop while blocked/slowed down, the CPU is 4% to 7% and I see 2 dropbear PIDs running. Both at 0% CPU.

Then for a few seconds, I get a responsive prompt I can do a couple of commands then lost connection.

not sure but sounds like duplicate IP address. Not all packages going to arrive on correct destination.

Joulinar Interesting point… so from my PC, I still have the continuous ping running. I went to the PiZero console and stopped the networking.service. Ping stopped and was not picked-up by sometime else.

My network is DHCP between .10.10 and 10.149 leaving 10.1 to 10.9 for special devices like the PiZero with a static IP.

And nothing is registered on 10.3 and ping does not answer… so not an issue with duplicate IP.


Update: Unsure why, but I had to go through dietpi-config and re-apply the network config page to re-enable the network. Only doing a start on the service did not work.

I checked with putty again after a restart of network, still no prompt for login. “software caused connection abort”

tcpdump shows ack/win lines going both ways, from the PiZero to PC and back.

Running a ping -s 65500 from the PiZ to my router with no issue

Thanks!

Check arp tables of PC, RPi, and router to verify that they all have the correct mac/ip bindings.
Disconnect everything else from the network; leave only router, RPi and PC connected and try again.

do you have another desktop that can be used to check SSH connection using Putty? Or use a mobile device with SSH app??

Here’s what I tried:

  1. Wireshark on the PC and tcpdump on the PI. I see lots of re-transmission and spurious packets… unsure why.
  • But it’s clear that there’s a communication between the PC and the PI
  1. Stopped Node-red to see if there’s negative interactions
  • Putty cannot connect
  1. ARP:
  • PC see the correct IP/MAC of PI
  • PI see the correct IP/MAC of PC
  • Router see the correct IP/MAC of the PI and PC
  • Wifi AP see the correct IP/MAC of the PI and PC
  1. SSH from another device
  • From a VM running DietPi : Same issue - connecting timeout
  • From another laptop on the same wifi : Same issue - connecting timeout
  • The same PC can SSH ok with Putty to remote servers over a DirectAccess or VPN tunnel fine, do this every day!
  1. Stopped Mosquitto
  • ss -tulpn now only show :

Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port
tcp LISTEN 0 1000 0.0.0.0:22 0.0.0.0:* users:((“dropbear”,pid=15050,fd=3))
tcp LISTEN 0 1000 [::]:22 [::]:* users:((“dropbear”,pid=15050,fd=4))

  • Putty cannot connect
  1. Started a SMB connection to a Windows share to capture outputs and it’s working fine.

  2. Stopped DropBear

  • ss -tulpn show not port listening
  • Putty cannot connect - This was to test if something was also listening on port 22

:sunglasses: Restarted DropBear

  • Cannot connect
  1. from the PiZero console, I tested the dropbear.service
  • Connected properly to localhost with dbclient
  • Connected properly to localhost with ssh

So dropbear seems to work properly!

  1. Changed dropbear port to 2222
  • Started Putty - asked to accept to new host key
  • Logged in with root / pass with no issue!
  • So something is messing up port 22 over the network! Unsure what!
  • ss -tulpn show port 2222 is listening

Wow! Clues anyone?

  1. Did you disconnect all other devices from the network, leaving the PC and the RPi only?
  2. Also can you explain a bit the infrastructure you have there? Are both devices connected over the router or are there other devices, like switches, access points?
  3. Do you have full control you and only you over these devices?
  4. Are you using anytype of firewall, antivirus, antimalware on the RPi?
  5. Have you tried to boot a clean installation of dietpi on a different SD card?

trendy

Just tried a separate Wi-Fi created with my phone. A laptop to PiZero on this “private” network, Putty port 22 works fine.

Reactivated the wifi on the Bell router, connected a laptop and the PiZ, ssh works.

So seems to be the AP TP-LIKN EAP245 with an issue

I’m wondering if being on the same SSID and the laptop connected with 5G and the Pi with 2.4G would be an issue?

Here’s a rough drawing of my setup. Redlines are ssh not working, green are ssh working.


Thanks

did you tried to disable 5GHz WiFi and continue running on 2.4GHz only? Just to see how it’s behaving?

It should not be an issue. But give it a try.
Is the Access Point EAP245 running on latest firmware? I would try to factore restore it and apply a bare minimum configuration to make sure it works.

Tested on a 2.4G only SSID. Connected both PiZero and laptop, same issue. So it’s not an issue of 2.4G vs 5G

Then I tested another laptop connected with a wired connection to connect to the PiZero on a 2.4G wifi. So not going through wifi on the Putty side did not change that Putty cannot connect.

So…

Laptop (EAP245 wifi 5G) > PiZero (EAP245 2.4G) SSH on port 22 = Not working
Laptop (EAP245 wifi 5G) > PiZero (EAP245 2.4G) SSH on port 2222 = working
Laptop (EAP245 wifi 2.G) > PiZero (EAP245 2.4G) SSH on port 22 = Not working
Laptop (wired) > PiZero (EAP245 2.4G) SSH on port 22 = Not working
Laptop (Phone Wifi) > PiZero (Phone Wifi ) SSH on port 22 = working
Laptop (Bell-router Wifi) > PiZero (Bell router) SSH on port 22 = working

So, the issue is around port 22 over my EAP245 Wifi.

I checked the EAP245 config, no SSH is configured/activated on it. Firmware is up2date.
I don’t have issues doing ssh (putty) to remote servers over VPN/DirectAccess with the Wifi on EAP245

So, I’m about to close this case and put the PiZero ssh port to 2222

Ok! So I went back to port 2222… did not work like before! Then changed to port 1234… did not work… WTF!

So I reboot the AP EAP245 and bang, worked!

So it’s really the EAP245 which is buggy!

Went back to port 22… no issue!

I will open a ticket to TP-LINK

What a complicated path to find an issue… never thought an AP could be buggy for a specific protocol like SSH, since I’m doing a lot of SSH to work servers.

Thanks for all the brainstorming help guys… really appreciative.

Indeed quite strange an AP could have such influence. But good we found it. Even if it was a long way :slight_smile: