Apt Update & APT Upgrade Broke my DietPi(s)

Troubleshooting
1

Hi,

I have 4 Raspberry Pi running DietPi. Today I ran the command

sudo apt update && sudo apt full-upgrade -y

as I usually do to update. After the update I proceeded to restart the RPis just to not been able to SSH into any oft them. Any of the services running stoped working. I had 2 RPis running Pi-hole, 1 RPi running WireGuard and 1 running Homebridge. I then went and flash all of them with a DietPi in order to start from scratch. I can SSH into them. Run the initially setup. After it’s done and I reboot (I’m guessing it updated everything during the install), I can’t SSH into it again. I tried this with two different RPis and same result. At the time this happened I was (am) running Dietpi 8.0.2 64-bit.

Any suggestions as to what could be breaking? It seems to happened in the latest APT update/upgrade.

Side note (I’m fairly new to terminal/linux.

2

it’s a kernel issue on RPi https://github.com/MichaIng/DietPi/issues/5187

reported at RPi side already https://github.com/raspberrypi/firmware/issues/1686

3

Thanks

4

I am having the same issue. My pihole box is now non-bootable. Luckily I do have a backup of the pihole configuration so I can restore it.

One thing to note though is that this also affects new installations since Dietpi will automatically run apt upgrade during installation. Anyone trying to install it now is getting a dead system (unless they disconnect their pi from the internet).

5

this also affects new installations since Dietpi will automatically run apt upgrade during installation.

Correct, but disconnection the network is not an option as you are not able to complete the initial setup this way.

6

Ok for time being, kernel package has been removed and is not available anymore. At least no failing systems over night :slight_smile:

7

That’s great news. Thank you!

8

Does this mean if I do a clean install now, it will work for now?

9

Theoretically yes it should work. At least I did an apt update and kernel package was not available anymore. Just give it a try.

Btw: looks like removing one entry on configuration file fix it as well https://github.com/MichaIng/DietPi/issues/5187#issuecomment-1017085024

10

It looks like apt update & upgrade is done automatically on my Pi, luckily it did not grab that faulty kernel (probably due to the timezone I was in). Thinking back, I think it was PiVPN Wireguard that configured the system to check for updates every day, is there a way to undo this change? Or at least change it to weekly?

11

t looks like apt update & upgrade is done automatically on my Pi, luckily it did not grab that faulty kernel (probably due to the timezone I was in).

yes luckey RPi guys removed the kernel package quickly after our reports.

I think it was PiVPN Wireguard that configured the system to check for updates every day

yes a bad behaviour of PiVPN. But if I’m not mistaken we suppress this in meantime.

In your case there should be unattended-upgrades tool running. I will link the Debian Wiki. There you have a section about timers who should manage the execution.

https://wiki.debian.org/UnattendedUpgrades

BTW: DietPi as well offer an automatic update feature since a while. To be sure you can check following

# Daily check for APT package updates: 0=disable | 1=check only | 2=check and upgrade automatically
cat /boot/dietpi.txt | grep CONFIG_CHECK_APT_UPDATES
12

Thank you, it looks like I don’t have unattended-upgrades on my system, dpkg-query -l | grep unattended lists nothing

Dietpi-Cron daily is set for 3am and looks to be responsible for the apt upgrades in /root/logfile_storage/apt/history.log:

Start-Date: 2022-01-19  03:00:45
Commandline: apt-get -qq upgrade
Upgrade: libraspberrypi0:arm64 (1:2+git20211125~155417+14b90ff-2, 1:2+git20211125~155417+14b90ff-3), libraspberrypi-bin:arm64 (1:2+git20211125~155417+14b90ff-2, 1:2+git20211125~155417+14b90ff-3)
End-Date: 2022-01-19  03:00:46

Output of cat /boot/dietpi.txt | grep CONFIG_CHECK_APT_UPDATES returns 2, so I will change this to 1 in dietpi.txt

13

in meantime issue has been identified by RPi developer. Unfortunately it was a rar combination of a setting we do on DietPi and how RPi kernel was handling a specific situation. A kernel fix is on the way and I already tested it on my demo system

root@DietPi4:~# uname -a
Linux DietPi4 5.10.92-v8+ #1514 SMP PREEMPT Mon Jan 17 17:39:38 GMT 2022 aarch64 GNU/Linux
root@DietPi4:~#
14

new kernel + bootloader has been released by RPi devs

raspberrypi-bootloader/stable 1:1.20220120-1 armhf [upgradable from: 1:1.20211118-3]
raspberrypi-kernel/stable 1:1.20220120-1 armhf [upgradable from: 1:1.20211118-3]