SSH stopped working suddenly

Creating a bug report/issue

I have searched the existing open and closed issues

Required Information

  • DietPi version 8.25.1 | cat /boot/dietpi/.version
  • Distro version bookworm | echo $G_DISTRO_NAME $G_RASPBIAN
  • Kernel version 6.1.0-17-amd64 | uname -a
  • Architecture amd64 | dpkg --print-architecture
  • SBC model Native PC (x86_64) | echo $G_HW_MODEL_NAME or (EG: RPi3)
  • Power supply used 60w power brick on a Wyse Thin Client| (EG: 5V 1A RAVpower)
  • SD card used | (EG: SanDisk ultra)

Additional Information (if applicable)

  • Software title SSHD (Open SSH)| (EG: Nextcloud)
  • Was the software title installed freshly or updated/migrated? Not outside of normal APT updates
  • Can this issue be replicated on a fresh installation of DietPi? Have no way to try
    ← If you sent a “dietpi-bugreport”, please paste the ID here →
  • Bug report ID | echo $G_HW_UUID

Steps to reproduce

  1. … Everything has been working just fine, but I noticed this morning that I was unable to connect via SH as I normally do. I have the box set up on an alternative port and using key-authentication, but when I attempt to connect from anywhere (my normal workstation and also from another dietpi instance right next to it), I get no response. Oddly, what I get back from a linux box is different than from a Windows PC. From a Windows box nothing happens at all, but from another linux box I get"

dietpi@617NUC:~$ ssh -p 60022 dietpi@192.168.0.77
ssh: connect to host 192.168.0.77 port 60022: Invalid argument

  1. … Also, the server IS up, as it’s serving DNS just fine w/ PiHole/Unbound and also working fine as a VPN server using Wireguard/PiVPN, I just cannot get to it via SSH, nor does the PiHole admin page seem to work.

Expected behaviour

  • … Normal SSH connection

Actual behaviour

  • No connection.

Extra details

  • … I can connect via an attached kb/display, and networking seems to work just fine on it, as I can get out to the Internet via the dietpi-config test.

Did you tried to reboot?

Repeatedly, on both the server in question and on multiple different clients, on multiple different OSs.

I did mention that I was using an alternate SSH port and using public key authorization, right? Not my first rodeo.

In any case, still no idea why it’s not working, and the clue that something is maybe amiss with my network itself is the “Invalid Argument” error I get back, plus lack of an ICMP ping response, which works just fine on my other two DietPi systems.

Did a little more digging on the SSH client side, but still nothing useful.

617NUC is a DietPi instance running inside my internal network at 192.168.0.82
PiBackup is a DietPi instance running inside my internet network at 192.168.0.77

Both are essentially identical machines, and .77 has been running just fine for more than a year now before this issue came up.

It’s also not just an SSH issue, as Cloudflare secure tunnels also stopped working at the same time on it, while continuing to work just fine on .82.

Confusingly, .77 is still working just fine with PiHole, Unbound, and PiVPN

dietpi@617NUC:~$ ssh -p 60022 -v -v -v -B eth0 dietpi@192.168.0.77
OpenSSH_9.2p1 Debian-2+deb12u2, OpenSSL 3.0.11 19 Sep 2023
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug2: resolve_canonicalize: hostname 192.168.0.77 is address
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/dietpi/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/dietpi/.ssh/known_hosts2'
debug3: ssh_connect_direct: entering
debug1: Connecting to 192.168.0.77 [192.168.0.77] port 60022.
debug3: set_sock_tos: set socket 3 IP_TOS 0x10
debug1: ssh_create_socket: bound to 192.168.0.82
debug1: connect to address 192.168.0.77 port 60022: Invalid argument
ssh: connect to host 192.168.0.77 port 60022: Invalid argument
dietpi@617NUC:~$ 

dietpi@617NUC:~$ host pibackup
pibackup has address 192.168.0.77

dietpi@617NUC:~$ nslookup 192.168.0.77
77.0.168.192.in-addr.arpa	name = PiBackup.

Best is to get local access to check what is going on. Might be something wrong or data corruption. But without access, we don’t know.

Yep, I have stood at the console and looked and don’t see anything amiss…

Doing a search of the logs (journalctl -B | grep -i sshd, for example) doesn’t show an issue.

If nobody here has anything better to offer in terms of troubleshooting beyond telling me to reboot it (duh) and stand at the console, I guess I am on my own?

did you checked service status of SSH server?

systemctl status ssh

Usually you should see login attempts within system log journalctl

journalctl -u ssh

Yes, the service shows as up and running normally. That’s the very first thing I checked. I was doing the journalctl to look for further errors or related ones not explicitly tied to sshd.

I’ll have it send a bug report when I can, just need to write it down and transcribe the number.

And no, no evidence of logon attempts on the 77 box.

Hmm If there are no login attachments, not sure what we should troubleshoot on this as we don’t have any logs.

Can you try to login to your system on the specific port just es regular user without any key, just to see if it connects at all.

Furthermore you can check listen port

ss -tulpn | grep LISTEN

Sure thing… It’s the “Invalid argument” thing that throws me. Not sure what that means in this context, except that maybe something is wrong on the routing table on the server.

dietpi@617NUC:~$ sudo route
Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         Router          0.0.0.0         UG    0      0        0 eth0
10.106.204.0    0.0.0.0         255.255.255.0   U     0      0        0 wg0
172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
172.18.0.0      0.0.0.0         255.255.0.0     U     0      0        0 br-d471f9b57f53
172.19.0.0      0.0.0.0         255.255.0.0     U     0      0        0 br-9e24de427fb8
192.168.0.0     0.0.0.0         255.255.255.0   U     0      0        0 eth0
PiBackup        0.0.0.0         255.255.255.255 UH    0      0        0 *


dietpi@617NUC:~$ ssh -v -v -v -B eth0 dietpi@192.168.0.77
OpenSSH_9.2p1 Debian-2+deb12u2, OpenSSL 3.0.11 19 Sep 2023
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug2: resolve_canonicalize: hostname 192.168.0.77 is address
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/dietpi/.ssh/known_hosts'
debug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/dietpi/.ssh/known_hosts2'
debug3: ssh_connect_direct: entering
debug1: Connecting to 192.168.0.77 [192.168.0.77] port 22.
debug3: set_sock_tos: set socket 3 IP_TOS 0x10
debug1: ssh_create_socket: bound to 192.168.0.82
debug1: connect to address 192.168.0.77 port 22: Invalid argument
ssh: connect to host 192.168.0.77 port 22: Invalid argument
dietpi@617NUC:~$ sudo ss -tulpn | grep listen
dietpi@617NUC:~$

I think I figured it out, but not sure how it got that way.

Doing an IP route, the IP for the .77 shows up as “blackhole”, and removing that route from the .82 server (to the .77 one) made everything seem to work again.

root@617NUC:~# ip route
default via 192.168.0.1 dev eth0 onlink 
10.106.204.0/24 dev wg0 proto kernel scope link src 10.106.204.1 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 
172.18.0.0/16 dev br-d471f9b57f53 proto kernel scope link src 172.18.0.1 linkdown 
172.19.0.0/16 dev br-9e24de427fb8 proto kernel scope link src 172.19.0.1 
192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.82 
blackhole 192.168.0.77]

root@617NUC:~# ip route del blackhole 192.168.0.77
root@617NUC:~# ip route
default via 192.168.0.1 dev eth0 onlink 
10.106.204.0/24 dev wg0 proto kernel scope link src 10.106.204.1 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 
172.18.0.0/16 dev br-d471f9b57f53 proto kernel scope link src 172.18.0.1 linkdown 
172.19.0.0/16 dev br-9e24de427fb8 proto kernel scope link src 172.19.0.1 
192.168.0.0/24 dev eth0 proto kernel scope link src 192.168.0.82 
root@617NUC:~# ping 192.168.0.77
PING 192.168.0.77 (192.168.0.77) 56(84) bytes of data.
64 bytes from 192.168.0.77: icmp_seq=1 ttl=64 time=0.379 ms
64 bytes from 192.168.0.77: icmp_seq=2 ttl=64 time=0.392 ms
64 bytes from 192.168.0.77: icmp_seq=3 ttl=64 time=0.357 ms
^C
--- 192.168.0.77 ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2054ms
rtt min/avg/max/mdev = 0.357/0.376/0.392/0.014 ms
root@617NUC:~# [A