Portainer keeps restarting after update to latest docker version

daf · 5 February 2023 21:56

Creating a bug report/issue

After using “dietpi upgrade” my system updated to the latest version of docker. After this update portainer was no longer running. I tried removing and reistalling the portainer package using “dietpi-software reinstall 185” but then I ran into an error that the path to apparmor could not be found. So I installed apparmor; now portainer installs but it keeps on restarting itself.

Any clues how tyo solve this? I’m not sure how to get the docker error logs…

Daf

Required Information

DietPi version | cat /boot/dietpi/.version
G_DIETPI_VERSION_CORE=8
G_DIETPI_VERSION_SUB=13
G_DIETPI_VERSION_RC=2
G_GITBRANCH=‘master’
G_GITOWNER=‘MichaIng’
G_LIVE_PATCH_STATUS[0]=‘not applied’
G_LIVE_PATCH_STATUS[1]=‘not applicable’
Kernel version | uname -a
Linux DietPi 4.9.312-arm64 #1 SMP PREEMPT Tue Nov 22 21:32:44 CET 2022 aarch64 GNU/Linux
SBC model | echo $G_HW_MODEL_NAME or (EG: RPi3)
Odroid C4/HC4 (aarch64)
Power supply used | (EG: 5V 1A RAVpower)
SD card used | (EG: SanDisk ultra)

Additional Information (if applicable)

Software title | (EG: Nextcloud)
Was the software title installed freshly or updated/migrated?
Can this issue be replicated on a fresh installation of DietPi?
← If you sent a “dietpi-bugreport”, please paste the ID here →
Bug report ID | echo $G_HW_UUID

Steps to reproduce

…
…

Expected behaviour

…

Actual behaviour

…

Extra details

…

Joulinar · 5 February 2023 22:00

Yes this is know issue with lates docker version. Docker guys out it into their release note as known issue Docker Engine 23.0 release notes | Docker Docs

This will show logs from Docker engine.

journalctl -u docker.service

daf · 5 February 2023 22:13

Thank you Joulinar,

This is the main error I’m getting for this container:

Feb 05 22:57:25 DietPi ff9a154da795[11833]: 2023/02/05 09:57PM INF /portainer/portainer/api/cmd/portainer/main.go:530 > encryption key file not present | filename=portainer
Feb 05 22:57:25 DietPi ff9a154da795[11833]: 2023/02/05 09:57PM INF /portainer/portainer/api/cmd/portainer/main.go:549 > proceeding without encryption key |
Feb 05 22:57:25 DietPi ff9a154da795[11833]: 2023/02/05 09:57PM INF /portainer/portainer/api/database/boltdb/db.go:124 > loading PortainerDB | filename=portainer.db
Feb 05 22:57:25 DietPi ff9a154da795[11833]: 2023/02/05 21:57:25 server: Reverse tunnelling enabled
Feb 05 22:57:25 DietPi ff9a154da795[11833]: 2023/02/05 21:57:25 server: Fingerprint 0f:b3:69:0a:5a:4b:f6:7e:da:bb:40:83:3c:2a:c6:5f
Feb 05 22:57:25 DietPi ff9a154da795[11833]: 2023/02/05 21:57:25 server: Listening on 0.0.0.0:8000...
Feb 05 22:57:25 DietPi ff9a154da795[11833]: 2023/02/05 09:57PM FTL /portainer/portainer/api/cmd/portainer/main.go:704 > failed starting tunnel server | error="listen tcp 0.0.0.0:8000: socket: permission denied"

Joulinar · 5 February 2023 22:53

How did you setup Docker / Portainer? Did you used dietpi-software? Because we don’t user port 8000 on Portainer. Do you run Docker as unprivileged user?

MichaIng · 5 February 2023 22:55

listen tcp 0.0.0.0:8000: socket: permission denied

But there is nothing else running on that socket, is it?

ss -tlpn

In your case, as you use an old Odroid image with Linux 4.9, it may be a kernel feature Docker now uses, which is not provided by Linux 4.9 yet. Can yous how:

grep -q 'systemd.unified_cgroup_hierarchy' /boot/boot.ini

daf · 5 February 2023 23:09

I used dietpi-software

There is nothing on socket 8000

grep -q 'systemd.unified_cgroup_hierarchy' /boot/boot.ini

Gives me nothing.

Reading here shows it might be related with apparmor (that I installed)… Is there a way to roll back to the previous version of docker? Or might there be another way out.

Thanks already for helping out

Joulinar · 5 February 2023 23:24

This is strange, because our Portainer installation will use port 9002 only and not 8000.

Can you try to reinstall

dietpi-software reinstall 185

WarHawk · 6 February 2023 03:59

Stop and uninstall the old portainer, then re-add the new portainer

How to Update Portainer in 2023 - WunderTech

It’s a full refresh up to the latest release as well
Blow it out (setting will be retained) then re-install

Joulinar · 6 February 2023 09:27

simply run dietpi-software reinstall 185 if Portainer has been installed from our software catalogue. No need to follow any guide at this point.

naddel91 · 7 February 2023 21:39

it gives me this when trying to update:

as always: joulinar did the trick already:

Joulinar · 8 February 2023 08:50

Just to avoid a misunderstanding. The issue with apparmor is not an issue of DietPi. It’s a challenge of latest Docker release v23.

daf · 8 February 2023 10:45

Dear people, thanks already for the answers.

I tried uninstalling portainer and reinstalling with apparmor disabled using
systemctl disable --now apparmor
but I still get the fatal error
FTL github.com/portainer/portainer/api/cmd/portainer/main.go:716 > failed start ing tunnel server | error="listen tcp 0.0.0.0:8000: socket: permission denied"

I think when I first set my system up I did not use the portainer package/script from dietpi so it might thjat it’s looking at older config files? Is there a way to compl;eteley remove docker/portainer and its settings and install it anew?

Daf

Joulinar · 8 February 2023 10:57

can share following

docker container ls -all
docker image ls
docker volume ls

daf · 8 February 2023 11:04

docker container ls -all
CONTAINER ID   IMAGE                    COMMAND        CREATED         STATUS                          PORTS     NAMES
5f1aca2884ab   portainer/portainer-ce   "/portainer"   9 minutes ago   Restarting (1) 44 seconds ago             portainer

docker image ls
REPOSITORY                               TAG       IMAGE ID       CREATED         SIZE
portainer/portainer-ce                   latest    0315eb3100ec   38 hours ago    267MB
slskd/slskd                              latest    281b1eca68fa   9 days ago      238MB
slskd/slskd                              <none>    b2ed1a90ad40   4 weeks ago     238MB
slskd/slskd                              <none>    8ca2b1bb7e87   6 weeks ago     238MB
slskd/slskd                              <none>    3ef6e8dc9ef8   8 weeks ago     238MB
slskd/slskd                              <none>    8fd6c872f08e   2 months ago    233MB
slskd/slskd                              <none>    1905468a1e94   2 months ago    233MB
slskd/slskd                              <none>    b399b04e88db   3 months ago    233MB
slskd/slskd                              <none>    0fee92b00d2e   3 months ago    232MB
slskd/slskd                              <none>    655d4ac00c6c   4 months ago    232MB
slskd/slskd                              <none>    0ecce3aed08d   4 months ago    233MB
slskd/slskd                              <none>    67a70b476051   5 months ago    232MB
slskd/slskd                              <none>    ab3e40bcc204   5 months ago    232MB
realies/soulseek                         latest    5a9483739566   5 months ago    471MB
portainer/portainer-ee                   latest    10db4019e5b5   6 months ago    297MB
portainer/portainer-ce                   2.13.1    ba1c6a880a3c   9 months ago    265MB
homeassistant/home-assistant             latest    f321193ab8b0   14 months ago   1.16GB
cr.portainer.io/portainer/portainer-ce   2.9.3     725d6dd70b20   14 months ago   237MB
domoticz/domoticz                        latest    afd5cb0f11e4   15 months ago   269MB
portainer/portainer-ce                   <none>    dfbe0287e051   17 months ago   171MB
airsonicadvanced/airsonic-advanced       latest    38636017a800   17 months ago   850MB
linuxserver/qbittorrent                  latest    a16fda57bbe2   17 months ago   398MB
portainer/portainer-ce                   <none>    a4223b939682   18 months ago   175MB
epoupon/lms                              latest    8397665947a0   20 months ago   34.8MB
danielguerra/soulseek                    latest    70ce73a40a98   21 months ago   520MB
portainer/portainer                      latest    ce24b0500fe9   23 months ago   68.4MB
lsiobase/java.arm64                      latest    b68b81e6f3b6   3 years ago     437MB

docker volume ls
DRIVER    VOLUME NAME
local     7f92902bc5079dd8439970e15dc86e8190ab4bc98ee556140589a5a7cb3bdf1b
local     ee5fb641b4e8331d2ff48be032b726b3d6a155b094a0f9e8df6cec4e67da4b51
local     portainer_data

Joulinar · 8 February 2023 11:21

I would recommend dropping current container:

docker rm --force portainer

Did you already install portainer from our catalogue? Let’s check actual status

dietpi-software list | grep Portainer

If it returns 0 do following

dietpi-software install 185

If it returns 2 do following

dietpi-software reinstall 185

daf · 8 February 2023 11:50

dietpi-software list | grep Portainer
ID 185 | =2 | Portainer: Simplifies container management in Docker (standalone host) | +Docker | https://dietpi.com/docs/software/programming/#portainer

After reinstalling I still get the error:

Feb 08 12:44:14 DietPi be35b5dea920[3402]: [11:44:14 INF] Share cache successfully restored from backup
Feb 08 12:44:14 DietPi be35b5dea920[3402]: [11:44:14 INF] Share cache loaded from disk successfully. Sharing 18623 directories and 17
1385 files
Feb 08 12:44:27 DietPi ca68740d19bf[3402]: 2023/02/08 11:44AM INF github.com/portainer/portainer/api/cmd/port
ainer/main.go:534 > encryption key file not present | filename=portainer
Feb 08 12:44:27 DietPi ca68740d19bf[3402]: 2023/02/08 11:44AM INF github.com/portainer/portainer/api/cmd/port
ainer/main.go:553 > proceeding without encryption key |
Feb 08 12:44:27 DietPi ca68740d19bf[3402]: 2023/02/08 11:44AM INF github.com/portainer/portainer/api/database
/boltdb/db.go:124 > loading PortainerDB | filename=portainer.db
Feb 08 12:44:28 DietPi ca68740d19bf[3402]: 2023/02/08 11:44AM FTL github.com/portainer/portainer/api/cmd/port
ainer/main.go:716 > failed starting tunnel server | error="listen tcp 0.0.0.0:8000: socket: permission denied"
Feb 08 12:44:28 DietPi ca68740d19bf[3402]: 2023/02/08 11:44:28 server: Reverse tunnelling enabled
Feb 08 12:44:28 DietPi ca68740d19bf[3402]: 2023/02/08 11:44:28 server: Fingerprint 0f:b3:69:0a:5a:4b:f6:7e:da:bb:40:83:3c:2a:c6:5f
Feb 08 12:44:28 DietPi ca68740d19bf[3402]: 2023/02/08 11:44:28 server: Listening on 0.0.0.0:8000...
Feb 08 12:45:30 DietPi ca68740d19bf[3402]: 2023/02/08 11:45AM INF github.com/portainer/portainer/api/cmd/port
ainer/main.go:534 > encryption key file not present | filename=portainer
Feb 08 12:45:30 DietPi ca68740d19bf[3402]: 2023/02/08 11:45AM INF github.com/portainer/portainer/api/cmd/port
ainer/main.go:553 > proceeding without encryption key |
Feb 08 12:45:30 DietPi ca68740d19bf[3402]: 2023/02/08 11:45AM INF github.com/portainer/portainer/api/database
/boltdb/db.go:124 > loading PortainerDB | filename=portainer.db
Feb 08 12:45:30 DietPi ca68740d19bf[3402]: 2023/02/08 11:45:30 server: Reverse tunnelling enabled
Feb 08 12:45:30 DietPi ca68740d19bf[3402]: 2023/02/08 11:45:30 server: Fingerprint 0f:b3:69:0a:5a:4b:f6:7e:da:bb:40:83:3c:2a:c6:5f
Feb 08 12:45:30 DietPi ca68740d19bf[3402]: 2023/02/08 11:45:30 server: Listening on 0.0.0.0:8000...
Feb 08 12:45:30 DietPi ca68740d19bf[3402]: 2023/02/08 11:45AM FTL github.com/portainer/portainer/api/cmd/port
ainer/main.go:716 > failed starting tunnel server | error="listen tcp 0.0.0.0:8000: socket: permission denied"

Could it be it’s loading data from an older DB (because of apparmor?

Would it work deleting the portainer data?

Daf

Joulinar · 8 February 2023 11:55

This has nothing to do with apparmor. If you don’t mind, you could remove Portainer completely (including all Poratiner data)

dietpi-software uninstall 185

Reboot your system afterwards and check Docker logs again.

daf · 8 February 2023 11:59

hmm removing gives me an error that it is using a volume that I don’t know

  │  - Command: docker volume rm portainer_data                                                                          │ 
                      │  - Exit code: 1                                                                                                      │ 
                      │  - DietPi version: v8.13.2 (MichaIng/master) | HW_MODEL: 16 | HW_ARCH: 3 | DISTRO: 5                                 │ 
                      │  - Image creator: DietPi Core Team                                                                                   │ 
                      │  - Pre-image: Meveric                                                                                                │ 
                      │  - Error log:                                                                                                        │ 
                      │ Error response from daemon: remove portainer_data: volume is in use -                                                │ 
                      │ [c6915cc05cf72928a85df77c39c5b0e3dd55a82aaeb0a594f1f961bc0c4dfa96]

df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            1.9G     0  1.9G   0% /dev
tmpfs           372M   23M  349M   7% /run
/dev/mmcblk1p2   29G  2.0G   28G   7% /
tmpfs           1.9G     0  1.9G   0% /dev/shm
tmpfs           5.0M  8.0K  5.0M   1% /run/lock
tmpfs           1.9G     0  1.9G   0% /sys/fs/cgroup
tmpfs           1.9G     0  1.9G   0% /tmp
tmpfs            50M   12K   50M   1% /var/log
/dev/mmcblk1p1  256M   77M  180M  30% /boot
/dev/sda1       7.3T  4.6T  2.3T  67% /mnt/da3b2016-9263-4ae2-99d0-7b08e5b9278d
tmpfs           372M     0  372M   0% /run/user/1001
overlay         7.3T  4.6T  2.3T  67% /mnt/da3b2016-9263-4ae2-99d0-7b08e5b9278d/dietpi_userdata/docker-data/overlay2/2477224125f0b3549d9c3b6bbaf5fa5595507c85d801332f435ffc7c37458f0e/merged

Joulinar · 8 February 2023 12:20

can you check this again.

daf · 8 February 2023 14:26

docker container ls -all
CONTAINER ID   IMAGE                COMMAND     CREATED      STATUS                     PORTS     NAMES
be35b5dea920   slskd/slskd:latest   "./slskd"   9 days ago   Exited (137) 2 hours ago             slskd

docker image ls
REPOSITORY                               TAG       IMAGE ID       CREATED         SIZE
slskd/slskd                              latest    281b1eca68fa   10 days ago     238MB
slskd/slskd                              <none>    b2ed1a90ad40   4 weeks ago     238MB
slskd/slskd                              <none>    8ca2b1bb7e87   6 weeks ago     238MB
slskd/slskd                              <none>    3ef6e8dc9ef8   8 weeks ago     238MB
slskd/slskd                              <none>    8fd6c872f08e   2 months ago    233MB
slskd/slskd                              <none>    1905468a1e94   2 months ago    233MB
slskd/slskd                              <none>    b399b04e88db   3 months ago    233MB
slskd/slskd                              <none>    0fee92b00d2e   3 months ago    232MB
slskd/slskd                              <none>    655d4ac00c6c   4 months ago    232MB
slskd/slskd                              <none>    0ecce3aed08d   4 months ago    233MB
slskd/slskd                              <none>    67a70b476051   5 months ago    232MB
slskd/slskd                              <none>    ab3e40bcc204   5 months ago    232MB
realies/soulseek                         latest    5a9483739566   5 months ago    471MB
portainer/portainer-ee                   latest    10db4019e5b5   6 months ago    297MB
homeassistant/home-assistant             latest    f321193ab8b0   14 months ago   1.16GB
cr.portainer.io/portainer/portainer-ce   2.9.3     725d6dd70b20   14 months ago   237MB
domoticz/domoticz                        latest    afd5cb0f11e4   15 months ago   269MB
portainer/portainer-ce                   <none>    dfbe0287e051   17 months ago   171MB
airsonicadvanced/airsonic-advanced       latest    38636017a800   17 months ago   850MB
linuxserver/qbittorrent                  latest    a16fda57bbe2   17 months ago   398MB
portainer/portainer-ce                   <none>    a4223b939682   18 months ago   175MB
epoupon/lms                              latest    8397665947a0   20 months ago   34.8MB
danielguerra/soulseek                    latest    70ce73a40a98   22 months ago   520MB
portainer/portainer                      latest    ce24b0500fe9   23 months ago   68.4MB
lsiobase/java.arm64                      latest    b68b81e6f3b6   3 years ago     437MB

docker volume ls
DRIVER    VOLUME NAME
local     7f92902bc5079dd8439970e15dc86e8190ab4bc98ee556140589a5a7cb3bdf1b
local     ee5fb641b4e8331d2ff48be032b726b3d6a155b094a0f9e8df6cec4e67da4b51
local     portainer_data