SSH - Never get to prompt ! Topic is solved

Having issues with your DietPi installation or found a bug? Post it here.
SimonFili
Posts: 23
Joined: Sun Nov 22, 2020 4:43 pm

Re: SSH - Never get to prompt !

Post by SimonFili »

SSH - dropbear acting up again!

Putty cannot connect, "network error: Connection timed out"

Here's the output of my console after a restart of dropbear
IMG_20201129_185955.jpg
You can see tcpdump is showing traffic on port 22 to my PC: 192.168.10.143. And the status is ok.

I did the ss -tulpn again after and like @trendy is mentioning, the Send-Q for port 22 is 1000.

According to Wikipedia: Send-Q The count of bytes not acknowledged by the remote host.
- While this is going on, my PC is running a continuous ping to the PiZero with no loss.

So this is a Raspberry Pi Zero W (wifi), why would I have this issue after a while when it worked for a few days without and issue?

The last thing I did before getting there was to update/add some libraries to Node-Red. Which works fine.

Like I stated before, this Wi-Fi is solid with 4 people working remotely over it.
SimonFili
Posts: 23
Joined: Sun Nov 22, 2020 4:43 pm

Re: SSH - Never get to prompt !

Post by SimonFili »

One hour later... I get a prompt, I can enter userid/pass... but never get the prompt. TCPDUMP shows that the PiZ receives the packet if I press enter in putty, but I get no prompt after entering the password.

Weird!

A status on the service says that the password auth succeeded for root.

Then Putty times out and abort.

If I restart dropbear, I get back to the issue with no ask for user/pass.
User avatar
trendy
Posts: 281
Joined: Tue Feb 25, 2020 2:54 pm

Re: SSH - Never get to prompt !

Post by trendy »

There is something wrong here.
In the third line of the tcpdump the putty client is sending a reset packet, although it should be sending an ack.
In the next lines the dropbear is sending synack to putty, but putty seems to be ignoring them and keeps resending syn.
It looks to me that putty is not receiving the packets from dropbear, hence the high send-q number.
User avatar
Joulinar
Posts: 4113
Joined: Sat Nov 16, 2019 12:49 am

Re: SSH - Never get to prompt !

Post by Joulinar »

probably good to user wireshark on client side to see what is going on there. Maybe this will complete the picture.
Pls let us know if a solution is working. This could help others if they hit by similar situation. Your DietPi Team
SimonFili
Posts: 23
Joined: Sun Nov 22, 2020 4:43 pm

Re: SSH - Never get to prompt !

Post by SimonFili »

12h later, after a night's sleep! All is ok, I can log in through dropbear and get a prompt.

Seems that every time I touch something on this Pi Zero, it takes 12h to stabilize and work!

Oh! Talking to fast here... I get a prompt but the prompt is frozen.

If in putty I press enter, I see on the console of the PiZero tcpdump an SEQ from putty and an ACK from dropbear. But nothing happens on the putty side. So I'm connected but I don't get any response back in Putty!

Suddenly, the command I typed in putty, appeared like 2mins after. So I get something but goes through a wormhole to the alpha quadrant and back! ;-)

Looking at htop while blocked/slowed down, the CPU is 4% to 7% and I see 2 dropbear PIDs running. Both at 0% CPU.

Then for a few seconds, I get a responsive prompt I can do a couple of commands then lost connection.
User avatar
Joulinar
Posts: 4113
Joined: Sat Nov 16, 2019 12:49 am

Re: SSH - Never get to prompt !

Post by Joulinar »

not sure but sounds like duplicate IP address. Not all packages going to arrive on correct destination.
Pls let us know if a solution is working. This could help others if they hit by similar situation. Your DietPi Team
SimonFili
Posts: 23
Joined: Sun Nov 22, 2020 4:43 pm

Re: SSH - Never get to prompt !

Post by SimonFili »

@Joulinar Interesting point... so from my PC, I still have the continuous ping running. I went to the PiZero console and stopped the networking.service. Ping stopped and was not picked-up by sometime else.

My network is DHCP between .10.10 and 10.149 leaving 10.1 to 10.9 for special devices like the PiZero with a static IP.

And nothing is registered on 10.3 and ping does not answer... so not an issue with duplicate IP.


Update: Unsure why, but I had to go through dietpi-config and re-apply the network config page to re-enable the network. Only doing a start on the service did not work.

I checked with putty again after a restart of network, still no prompt for login. "software caused connection abort"

tcpdump shows ack/win lines going both ways, from the PiZero to PC and back.

Running a ping -s 65500 from the PiZ to my router with no issue

Thanks!
User avatar
trendy
Posts: 281
Joined: Tue Feb 25, 2020 2:54 pm

Re: SSH - Never get to prompt !

Post by trendy »

Check arp tables of PC, RPi, and router to verify that they all have the correct mac/ip bindings.
Disconnect everything else from the network; leave only router, RPi and PC connected and try again.
User avatar
Joulinar
Posts: 4113
Joined: Sat Nov 16, 2019 12:49 am

Re: SSH - Never get to prompt !

Post by Joulinar »

do you have another desktop that can be used to check SSH connection using Putty? Or use a mobile device with SSH app??
Pls let us know if a solution is working. This could help others if they hit by similar situation. Your DietPi Team
SimonFili
Posts: 23
Joined: Sun Nov 22, 2020 4:43 pm

Re: SSH - Never get to prompt !

Post by SimonFili »

Here's what I tried:
1) Wireshark on the PC and tcpdump on the PI. I see lots of re-transmission and spurious packets... unsure why.
- But it's clear that there's a communication between the PC and the PI

2) Stopped Node-red to see if there's negative interactions
- Putty cannot connect

3) ARP:
- PC see the correct IP/MAC of PI
- PI see the correct IP/MAC of PC
- Router see the correct IP/MAC of the PI and PC
- Wifi AP see the correct IP/MAC of the PI and PC

4) SSH from another device
- From a VM running DietPi : Same issue - connecting timeout
- From another laptop on the same wifi : Same issue - connecting timeout
- The same PC can SSH ok with Putty to remote servers over a DirectAccess or VPN tunnel fine, do this every day!

5) Stopped Mosquitto
- ss -tulpn now only show :
Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port
tcp LISTEN 0 1000 0.0.0.0:22 0.0.0.0:* users:(("dropbear",pid=15050,fd=3))
tcp LISTEN 0 1000 [::]:22 [::]:* users:(("dropbear",pid=15050,fd=4))
- Putty cannot connect

6) Started a SMB connection to a Windows share to capture outputs and it's working fine.

7) Stopped DropBear
- ss -tulpn show not port listening
- Putty cannot connect - This was to test if something was also listening on port 22

8) Restarted DropBear
- Cannot connect

9) from the PiZero console, I tested the dropbear.service
- Connected properly to localhost with dbclient
- Connected properly to localhost with ssh
> So dropbear seems to work properly!

10) Changed dropbear port to 2222
- Started Putty - asked to accept to new host key
- Logged in with root / pass with no issue!
- So something is messing up port 22 over the network! Unsure what!
- ss -tulpn show port 2222 is listening

Wow! Clues anyone?
Post Reply