Why is SSH hard-disconnected on reboot?

Troubleshooting
1

Hello.
I experience a strange phenomen and hope there is an easy fix for that:

I run DietPi 10.3.3 with openssh on my ODROID C2.

ssh is running fine. But when entering ‘reboot’, I experience a very crazy behavior:
Every other Linux gently closes the ssh connection (so I get a “Connection closed by remote host”), while DietPi seems to immediately cut the connection. This forces me to manually close the terminal window and open a new one. Every time.
→ So this is extremely nasty.

I need openssh instead of dropbear because dropbear does not support keys as openssh does.

How can I get a shooth and gently reboot as all other Linux systems do?

2

you mean SSH keys to be able to login without password?

3

Yes.

The problem is that ssh simply drops the connection without saying “goodbye”. so my terminal stays in a frozen condition and I need to close it.

4

but you can use regular SSH keys on Dropbear without issue. I use SSH keys to login to all my systems.

5

I tried multiple times to run ‘ssh-copy-id’ to transfer ther related files.
The only OS where it doesn’t work is DietPi 10.3.x
gemini recommended me to install openssh instead, and it immediately worked as expected → So I use openssh now (and I also have a working history)

6

Best to my knowledge ssh-copy-id is not working with Dropbear, see comment Dropbear not using SSH keys - #11 by Trimble-tech

the key needs to be moved and configured manually as described in Dropbear not using SSH keys - #2 by Joulinar

7

Exactly. I usually use only openSUSE (Leap) and I like all its features. For me all Debian derivates are a challenge. But it is ok. After 1 hour of installing extensions it behaves mostly like a SuSE.

But I can’t get behind the secret of WHY DietPi so brutally cuts the connected ssh client when I enter reboot.

8

Maybe it’s a ssh client issue? I do not experience this behavior with PuTTY nor with connection via console from another machine.

9

Hi, Jappe.

No, definitely not.

I use SSH with many clients (most of them are running openWRT, but also some with other Debians. No problem. All of them close first all network connections, then all running services, then shutdown or reboot.
Not here with DietPi.
The crazy behavior is visible on ssh, but this makes me think about other services. I have NFS mounts, daemons running (and writing) files. If DietPi is there also “such gentle”, then good night.

10

we don’t do any magic on the OpenSSH installation. It’s just the plain Debian package we install. No configuration adjustments.

And usually the SSH service should wait for the network stack before stopping it’s service. Just checked on a demo system and the service contains the correct After=network.target setting within the ssh.service

root@DietPiZ2:~# systemctl cat ssh.service
# /usr/lib/systemd/system/ssh.service
[Unit]
Description=OpenBSD Secure Shell server
Documentation=man:sshd(8) man:sshd_config(5)
After=network.target nss-user-lookup.target auditd.service
ConditionPathExists=!/etc/ssh/sshd_not_to_be_run

[Service]
EnvironmentFile=-/etc/default/ssh
ExecStartPre=/usr/sbin/sshd -t
ExecStart=/usr/sbin/sshd -D $SSHD_OPTS
ExecReload=/usr/sbin/sshd -t
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure
RestartPreventExitStatus=255
Type=notify
RuntimeDirectory=sshd
RuntimeDirectoryMode=0755

[Install]
WantedBy=multi-user.target
Alias=sshd.service

Maybe @MichaIng has an idea.

@RaiHan maybe you can compare your OpenWRT service config with the one from OpenSSH Server. As Said, it should be the original one provided by the Debian package.

11

Hello.
I compared the systemctl service config you showed, it is identical.
I also have dropbear uninstalled.

So far my setup should behave like yours.

Seems there is a little race condition then? Hope @MichaIng is reading this and has an idea.

12

Check out systemctl cat dietpi-kill_ssh, which runs on shutdown/reboot to kill the SSH processes. This was actually added with the aim to solve the issue you describe, but obviously, the ssh CLI client behaves differently than e.g. PuTTY:

  • By default, the network connection of the server is cut on shutdown, before the SSH client/connection workers are terminated.
  • In this case, PuTTY does not show any disconnect, but the client connection hangs indefinitely. You need to close the window, and open a new one. Instead, if the connection process is terminated, PuTTY shows a disconnect immediately, so one can select “reconnect” immediately, e.g. when doing a reboot of the server system.
  • So the ssh client behaves the opposite way round? If the network connection is terminated e.g. on a reboot, it shows a message, and allows to quickly reconnect? And if the SSH connection process is terminated, you cannot reconnect quickly? But why do you need to close the terminal window, does CTRL+C not work?

Anyway, hence this should solve it:

sudo systemctl disable dietpi-kill_ssh

Maybe we can identify the SSH client somehow (though difficult, as far as I can think of), and let the script kill or leave the processes conditionally :thinking:.

Ah yeah, that old bug in ssh-copy-id, misbehaving when seeing a Dropbear server. I opened a PR to fix this years ago, but was referred to the author of the (upstream) script. However, that one never answered my emails: [ssh-copy-id] Do not treat Dropbear special by MichaIng · Pull Request #250 · openssh/openssh-portable · GitHub
Then finally, someone from OpenSSH contacted him again, so it was fixed with OpenSSH v9.4: Special case OpenWrt instead of Dropbear. · openssh/openssh-portable@bdcaf79 · GitHub

So since Debian Trixie and Ubuntu Noble, ssh-copy-id works with Dropbear as well.

13

Hi, MichaIng.
Thank you for answering and explaining.

I disabled dietpi-kill_ssh, rebooted the ODROID C2, reconnected by ssh.
So the dietpi-kill_ssh ist for sure disabled now and the system was started from scratch.

Sadly I need to tell you there is no difference in behavior.

Is it possible that the order of shutdown of processes is wrong?

Normally I would say that shutting down the network should be one of the very last things before the system shuts off itself. But what I see makes me think that something shuts off the LAN very early.

If I can be of any help to you, please let me know. I have also an RPi 1B running on DietPi 10.3.3, there I have the same situation, so I can do tests for you.

14

Generally, the shutdown schedule is working as expected:

  • The parent SSH server service is stopped first, hence no new connections can be established.
  • The existing connection workers are however not stopped. This is an intended feature, so systemctl restart ssh does not abort the connection, and it is possible to maintain the system, upgrade the SSH server package, and even switch from one SSH server to another (purge OpenSSH, install Dropbear, vice versa) while connected via SSH, and the connection remains intact, instead of aborting the operation in an unfinished state. This is achieved by KillMode=process in both: dropbear.service as well as ssh.service, as you can check on all Linux distros, and upstream systemd units.
  • The network connection is terminated late, but it is explicitly terminated as part of networking.service at some point of the “regular” shutdown schedule. The SSH connection workers are not scheduled anywhere to be terminated, hence systemd just sends a kill signal to those after the regular schedule finished, like it does to all remaining processes that are still up at this point.

It seems logind is terminating SSH client connections as well on shutdown, via session cgroup. I am not sure why dietpi-kill_ssh is not working for you, as it should have the same effect. Maybe in edge cases, it needs to be explicitly ordered After=networking.service (or better After=network.target), to assure it is stopped before network is teared down (the ordering directives are reversed on shutdown). Can you test this:

G_CONFIG_INJECT 'After=' 'After=network.target' /etc/systemd/system/dietpi-kill_ssh.service '\[Unit\]'
systemctl enable dietpi-kill_ssh

And to define more precisely when it runs, after the SSH server services (parent processes) stopped:

G_CONFIG_INJECT 'Before=' 'Before=dropbear.service ssh.service' /etc/systemd/system/dietpi-kill_ssh.service '\[Unit\]'
systemctl daemon-reload

When enabling persistent journal logs, one can check the shutdown schedule. This is how it looks like on two of my systems:

journalctl -u dropbear -u ssh -u dietpi-kill_ssh -u network.target -u network-online.target -u networking

One with Dropbear:

May 02 17:01:13 host systemd[1]: Starting dietpi-kill_ssh.service - DietPi-Kill_SSH on shutdown...
May 02 17:01:13 host dropbear[558]: [558] May 02 17:01:13 Early exit: Terminated by signal
May 02 17:01:13 host systemd[1]: Stopping dropbear.service - Dropbear...
May 02 17:01:13 host dropbear[930]: [930] May 02 17:01:13 Exit (root) from <XXXX>: Terminated by signal
May 02 17:01:13 host dropbear[930]: [930] May 02 17:01:13 syslogin_perform_logout: logout(pts/0) returned an error: No such file or directory
May 02 17:01:13 host systemd[1]: dropbear.service: Deactivated successfully.
May 02 17:01:13 host systemd[1]: dropbear.service: Unit process 935 (screen) remains running after unit stopped.
May 02 17:01:13 host systemd[1]: dropbear.service: Unit process 936 (bash) remains running after unit stopped.
May 02 17:01:13 host systemd[1]: dropbear.service: Unit process 1120 (systemctl) remains running after unit stopped.
May 02 17:01:13 host systemd[1]: dropbear.service: Unit process 1121 (systemd-tty-ask) remains running after unit stopped.
May 02 17:01:13 host systemd[1]: Stopped dropbear.service - Dropbear.
May 02 17:01:13 host systemd[1]: dropbear.service: Consumed 1.280s CPU time over 44.561s wall clock time, 37.5M memory peak.
May 02 17:01:14 host systemd[1]: Stopped target network-online.target - Network is Online.
May 02 17:01:15 host systemd[1]: dietpi-kill_ssh.service: Deactivated successfully.
May 02 17:01:15 host systemd[1]: Finished dietpi-kill_ssh.service - DietPi-Kill_SSH on shutdown.
May 02 17:01:25 host systemd[1]: Stopped target network.target - Network.
May 02 17:01:25 host systemd[1]: Stopping networking.service - Raise network interfaces...
May 02 17:01:25 host systemd[1]: networking.service: Deactivated successfully.
May 02 17:01:25 host systemd[1]: Stopped networking.service - Raise network interfaces.

One with OpenSSH:

May 02 12:56:49 host systemd[1]: Starting dietpi-kill_ssh.service - DietPi-Kill_SSH on shutdown...
May 02 12:56:49 host systemd[1]: Stopping ssh.service - OpenBSD Secure Shell server...
May 02 12:56:49 host systemd[1]: ssh.service: Deactivated successfully.
May 02 12:56:49 host systemd[1]: ssh.service: Unit process 2968812 (screen) remains running after unit stopped.
May 02 12:56:49 host systemd[1]: ssh.service: Unit process 3992509 (gpg-agent) remains running after unit stopped.
May 02 12:56:49 host systemd[1]: ssh.service: Unit process 2009186 (bash) remains running after unit stopped.
May 02 12:56:49 host systemd[1]: ssh.service: Unit process 3489706 (sshd-session) remains running after unit stopped.
May 02 12:56:49 host systemd[1]: ssh.service: Unit process 3489712 (sshd-session) remains running after unit stopped.
May 02 12:56:49 host systemd[1]: ssh.service: Unit process 3489716 (screen) remains running after unit stopped.
May 02 12:56:49 host systemd[1]: ssh.service: Unit process 3501386 (systemctl) remains running after unit stopped.
May 02 12:56:49 host systemd[1]: ssh.service: Unit process 3501387 (systemd-tty-ask) remains running after unit stopped.
May 02 12:56:49 host systemd[1]: Stopped ssh.service - OpenBSD Secure Shell server.
May 02 12:56:49 host systemd[1]: ssh.service: Consumed 32min 42.363s CPU time over 1w 4d 7h 56min 43.474s wall clock time, 14.9G memory peak.
May 02 12:56:51 host systemd[1]: dietpi-kill_ssh.service: Deactivated successfully.
May 02 12:56:51 host systemd[1]: Finished dietpi-kill_ssh.service - DietPi-Kill_SSH on shutdown.
May 02 12:56:52 host systemd[1]: Stopped target network-online.target - Network is Online.
May 02 12:56:52 host systemd[1]: Stopped target network.target - Network.
May 02 12:56:52 host systemd[1]: Stopping networking.service - Raise network interfaces...
May 02 12:56:52 host systemd[1]: networking.service: Deactivated successfully.
May 02 12:56:52 host systemd[1]: Stopped networking.service - Raise network interfaces.

You can see the logs form the SSH service stop, leaving the processes from the client connection active: the client connection processes, GNU screen (which I use), the shell, the systemctl command (reboot is a symlink to systemctl). But then our kill service terminates the SSH connection processes before network is teared down. It would be however cleaner if our kill service runs after the SSH server services stopped, so it kills only the remaining child connection processes.

However, if in your case networking.service stops before our kill service runs, our service cannot have any effect anymore.

If ordering is not the issue, try to enable logind instead:

systemctl disable dietpi-kill_ssh
systemctl unmask systemd-logind
apt install dbus
systemctl start dbus systemd-logind

Would be also interesting the compare the shutdown logs in that case, whether the SSH client connection processes are terminated explicitly as well (by logind/PAM/whatever). Or maybe there is even another signal involved that causes a more graceful SSH connection stop. If so, we could use that signal in our service as well.