Why is SSH hard-disconnected on reboot?

Troubleshooting
1

Hello.
I experience a strange phenomen and hope there is an easy fix for that:

I run DietPi 10.3.3 with openssh on my ODROID C2.

ssh is running fine. But when entering ‘reboot’, I experience a very crazy behavior:
Every other Linux gently closes the ssh connection (so I get a “Connection closed by remote host”), while DietPi seems to immediately cut the connection. This forces me to manually close the terminal window and open a new one. Every time.
→ So this is extremely nasty.

I need openssh instead of dropbear because dropbear does not support keys as openssh does.

How can I get a shooth and gently reboot as all other Linux systems do?

2

you mean SSH keys to be able to login without password?

3

Yes.

The problem is that ssh simply drops the connection without saying “goodbye”. so my terminal stays in a frozen condition and I need to close it.

4

but you can use regular SSH keys on Dropbear without issue. I use SSH keys to login to all my systems.

5

I tried multiple times to run ‘ssh-copy-id’ to transfer ther related files.
The only OS where it doesn’t work is DietPi 10.3.x
gemini recommended me to install openssh instead, and it immediately worked as expected → So I use openssh now (and I also have a working history)

6

Best to my knowledge ssh-copy-id is not working with Dropbear, see comment Dropbear not using SSH keys - #11 by Trimble-tech

the key needs to be moved and configured manually as described in Dropbear not using SSH keys - #2 by Joulinar

7

Exactly. I usually use only openSUSE (Leap) and I like all its features. For me all Debian derivates are a challenge. But it is ok. After 1 hour of installing extensions it behaves mostly like a SuSE.

But I can’t get behind the secret of WHY DietPi so brutally cuts the connected ssh client when I enter reboot.

8

Maybe it’s a ssh client issue? I do not experience this behavior with PuTTY nor with connection via console from another machine.

9

Hi, Jappe.

No, definitely not.

I use SSH with many clients (most of them are running openWRT, but also some with other Debians. No problem. All of them close first all network connections, then all running services, then shutdown or reboot.
Not here with DietPi.
The crazy behavior is visible on ssh, but this makes me think about other services. I have NFS mounts, daemons running (and writing) files. If DietPi is there also “such gentle”, then good night.

10

we don’t do any magic on the OpenSSH installation. It’s just the plain Debian package we install. No configuration adjustments.

And usually the SSH service should wait for the network stack before stopping it’s service. Just checked on a demo system and the service contains the correct After=network.target setting within the ssh.service

root@DietPiZ2:~# systemctl cat ssh.service
# /usr/lib/systemd/system/ssh.service
[Unit]
Description=OpenBSD Secure Shell server
Documentation=man:sshd(8) man:sshd_config(5)
After=network.target nss-user-lookup.target auditd.service
ConditionPathExists=!/etc/ssh/sshd_not_to_be_run

[Service]
EnvironmentFile=-/etc/default/ssh
ExecStartPre=/usr/sbin/sshd -t
ExecStart=/usr/sbin/sshd -D $SSHD_OPTS
ExecReload=/usr/sbin/sshd -t
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure
RestartPreventExitStatus=255
Type=notify
RuntimeDirectory=sshd
RuntimeDirectoryMode=0755

[Install]
WantedBy=multi-user.target
Alias=sshd.service

Maybe @MichaIng has an idea.

@RaiHan maybe you can compare your OpenWRT service config with the one from OpenSSH Server. As Said, it should be the original one provided by the Debian package.

11

Hello.
I compared the systemctl service config you showed, it is identical.
I also have dropbear uninstalled.

So far my setup should behave like yours.

Seems there is a little race condition then? Hope @MichaIng is reading this and has an idea.

12

Check out systemctl cat dietpi-kill_ssh, which runs on shutdown/reboot to kill the SSH processes. This was actually added with the aim to solve the issue you describe, but obviously, the ssh CLI client behaves differently than e.g. PuTTY:

  • By default, the network connection of the server is cut on shutdown, before the SSH client/connection workers are terminated.
  • In this case, PuTTY does not show any disconnect, but the client connection hangs indefinitely. You need to close the window, and open a new one. Instead, if the connection process is terminated, PuTTY shows a disconnect immediately, so one can select “reconnect” immediately, e.g. when doing a reboot of the server system.
  • So the ssh client behaves the opposite way round? If the network connection is terminated e.g. on a reboot, it shows a message, and allows to quickly reconnect? And if the SSH connection process is terminated, you cannot reconnect quickly? But why do you need to close the terminal window, does CTRL+C not work?

Anyway, hence this should solve it:

sudo systemctl disable dietpi-kill_ssh

Maybe we can identify the SSH client somehow (though difficult, as far as I can think of), and let the script kill or leave the processes conditionally :thinking:.

Ah yeah, that old bug in ssh-copy-id, misbehaving when seeing a Dropbear server. I opened a PR to fix this years ago, but was referred to the author of the (upstream) script. However, that one never answered my emails: [ssh-copy-id] Do not treat Dropbear special by MichaIng · Pull Request #250 · openssh/openssh-portable · GitHub
Then finally, someone from OpenSSH contacted him again, so it was fixed with OpenSSH v9.4: Special case OpenWrt instead of Dropbear. · openssh/openssh-portable@bdcaf79 · GitHub

So since Debian Trixie and Ubuntu Noble, ssh-copy-id works with Dropbear as well.

13

Hi, MichaIng.
Thank you for answering and explaining.

I disabled dietpi-kill_ssh, rebooted the ODROID C2, reconnected by ssh.
So the dietpi-kill_ssh ist for sure disabled now and the system was started from scratch.

Sadly I need to tell you there is no difference in behavior.

Is it possible that the order of shutdown of processes is wrong?

Normally I would say that shutting down the network should be one of the very last things before the system shuts off itself. But what I see makes me think that something shuts off the LAN very early.

If I can be of any help to you, please let me know. I have also an RPi 1B running on DietPi 10.3.3, there I have the same situation, so I can do tests for you.