Cannot start docker containers

Having issues with your DietPi installation or found a bug? Post it here.
Post Reply
ivenhov
Posts: 6
Joined: Thu Oct 14, 2021 4:45 pm

Cannot start docker containers

Post by ivenhov »

Hi all

I'm on NanoPC-TV with dietpi migrated to bullseye.
I'm trying to install docker and start a hello-world container.
Upon installation via dietpi-software I get error

Code: Select all

Selecting previously unselected package docker-ce-cli.
(Reading database ... 46985 files and directories currently installed.)
Preparing to unpack .../docker-ce-cli_5%3a20.10.9~3-0~debian-bullseye_arm64.deb ...
Unpacking docker-ce-cli (5:20.10.9~3-0~debian-bullseye) ...
Selecting previously unselected package docker-ce.
Preparing to unpack .../docker-ce_5%3a20.10.9~3-0~debian-bullseye_arm64.deb ...
Unpacking docker-ce (5:20.10.9~3-0~debian-bullseye) ...
Setting up docker-ce-cli (5:20.10.9~3-0~debian-bullseye) ...
Setting up docker-ce (5:20.10.9~3-0~debian-bullseye) ...
Job for docker.service failed because the control process exited with error code.
See "systemctl status docker.service" and "journalctl -xe" for details.
invoke-rc.d: initscript docker, action "restart" failed.
● docker.service - Docker Application Container Engine
     Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
     Active: activating (auto-restart) (Result: exit-code) since Thu 2021-10-14 14:59:58 BST; 33ms ago
TriggeredBy: ● docker.socket
       Docs: https://docs.docker.com
    Process: 40474 ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock (code=exited, status=1/FAILURE)
   Main PID: 40474 (code=exited, status=1/FAILURE)

Oct 14 14:59:58 DietPi systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Oct 14 14:59:58 DietPi systemd[1]: docker.service: Failed with result 'exit-code'.
Oct 14 14:59:58 DietPi systemd[1]: Failed to start Docker Application Container Engine.
dpkg: error processing package docker-ce (--configure):
 installed docker-ce package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 docker-ce
E: Sub-process /usr/bin/dpkg returned an error code (1)
I can however start service later using

Code: Select all

service docker start

root@DietPi:~# service docker status
● docker.service - Docker Application Container Engine
     Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2021-10-14 15:28:20 BST; 22min ago
TriggeredBy: ● docker.socket
       Docs: https://docs.docker.com
   Main PID: 45030 (dockerd)
      Tasks: 11
     Memory: 26.0M
     CGroup: /system.slice/docker.service
             └─45030 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

Oct 14 15:28:20 DietPi dockerd[45030]: time="2021-10-14T15:28:20.452413178+01:00" level=info msg="Loading containers: done."
Oct 14 15:28:20 DietPi dockerd[45030]: time="2021-10-14T15:28:20.510148970+01:00" level=info msg="Docker daemon" commit=79ea9d3 graphdriver(s)=overlay2 version=20.10.9
Oct 14 15:28:20 DietPi dockerd[45030]: time="2021-10-14T15:28:20.510731428+01:00" level=info msg="Daemon has completed initialization"
Oct 14 15:28:20 DietPi systemd[1]: Started Docker Application Container Engine.
Oct 14 15:28:20 DietPi dockerd[45030]: time="2021-10-14T15:28:20.652845678+01:00" level=info msg="API listen on /run/docker.sock"
Oct 14 15:39:00 DietPi dockerd[45030]: time="2021-10-14T15:39:00.191039275+01:00" level=warning msg="Your kernel does not support cgroup namespaces.  Cgroup namespace setting discarded."
Oct 14 15:39:00 DietPi dockerd[45030]: time="2021-10-14T15:39:00.392383608+01:00" level=error msg="stream copy error: reading from a closed fifo"
Oct 14 15:39:00 DietPi dockerd[45030]: time="2021-10-14T15:39:00.392588233+01:00" level=error msg="stream copy error: reading from a closed fifo"
Oct 14 15:39:00 DietPi dockerd[45030]: time="2021-10-14T15:39:00.521419150+01:00" level=error msg="41b3e2ac426a574a9ec1c915056e1eac9e1a31ab566febef23331ee99d2636ed cleanup: failed to delete container from containerd: no>
Oct 14 15:39:00 DietPi dockerd[45030]: time="2021-10-14T15:39:00.521791025+01:00" level=error msg="Handler for POST /v1.41/containers/41b3e2ac426a574a9ec1c915056e1eac9e1a31ab566febef23331ee99d2636ed/start returned error>
root@DietPi:~#
When I try to start a container I get following error

Code: Select all

root@DietPi:~# docker -D run --name hello-world hello-world
DEBU[0000] [hijack] End of stdout
docker: Error response from daemon: OCI runtime create failed: cgroup namespaces aren't enabled in the kernel: unknown.
ERRO[0000] error waiting for container: context canceled
How to fix this?

System

Code: Select all

root@DietPi:~# uname -a
Linux DietPi 4.4.126 #7 SMP Thu Jul 19 22:51:16 EST 2018 aarch64 GNU/Linux
root@DietPi:~# cat /etc/*release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
root@DietPi:~#
Some info

Code: Select all

root@DietPi:~# docker -v
Docker version 20.10.9, build c2ea9bc

root@DietPi:~# cat /proc/cgroups
#subsys_name    hierarchy       num_cgroups     enabled
cpuset  0       111     1
cpu     0       111     1
cpuacct 0       111     1
blkio   0       111     1
memory  0       111     1
devices 0       111     1
freezer 0       111     1
net_cls 0       111     1
perf_event      0       111     1
net_prio        0       111     1
hugetlb 0       111     1
pids    0       111     1
debug   0       111     1
root@DietPi:~# mount | grep cgroup
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime)
User avatar
Joulinar
Posts: 5996
Joined: Sat Nov 16, 2019 12:49 am

Re: Cannot start docker containers

Post by Joulinar »

basically we had a similar report on Ordroid device running similar old kernel. https://github.com/MichaIng/DietPi/issues/4705

Solution on this case was to set systemd.unified_cgroup_hierarchy=0 to the in boot.ini. Maybe you could check if this is helping you as well.
Pls let us know if a solution is working. This could help others if they hit by similar situation. Your DietPi Team
ivenhov
Posts: 6
Joined: Thu Oct 14, 2021 4:45 pm

Re: Cannot start docker containers

Post by ivenhov »

Thanks for this.
It looks like the right direction!

I cannot however locate boot.ini on my system

Code: Select all

root@DietPi:~# ls -la /DietPi/
total 32
drwxr-xr-x  3 root root  4096 Oct 14 17:54 .
drwxr-xr-x 23 root root  4096 Oct 14 00:24 ..
drwxr-xr-x  4 root root  4096 Oct 14 17:42 dietpi
-rwxr-xr-x  1 root root  6129 Jun 10  2018 dietpi-README.md
-rwxr-xr-x  1 root root 11182 Oct 14 17:34 dietpi.txt
root@DietPi:~# ls -la /boot/
total 32
drwxr-xr-x  3 root root  4096 Oct 14 17:54 .
drwxr-xr-x 23 root root  4096 Oct 14 00:24 ..
drwxr-xr-x  4 root root  4096 Oct 14 17:42 dietpi
-rwxr-xr-x  1 root root  6129 Jun 10  2018 dietpi-README.md
-rwxr-xr-x  1 root root 11182 Oct 14 17:34 dietpi.txt
root@DietPi:~#
I used find command but the file does not exist.
I've created it under /DierPi/boot.ini with a single line

systemd.unified_cgroup_hierarchy=0

but that did not solve the problem, likely because I'm modifying wrong file.
Any further help would be appreciated.

Regards
Daniel
User avatar
MichaIng
Site Admin
Posts: 3357
Joined: Sat Nov 18, 2017 6:21 pm

Re: Cannot start docker containers

Post by MichaIng »

Strange thing, where is the kernel, actually? Can you show:

Code: Select all

ls -l /
And also to identify the model (as I cannot find a "NanoPC TV", only a NanoPC T4?):

Code: Select all

cat /proc/cpuinfo
cat /proc/device-tree/model
Where did you get the image from originally?
ivenhov
Posts: 6
Joined: Thu Oct 14, 2021 4:45 pm

Re: Cannot start docker containers

Post by ivenhov »

Hi
Sorry about spelling mistake, that is of course NanoPC-T4
https://www.friendlyarm.com/index.php?r ... uct_id=225

I guess kernel is in boot.img ?

Code: Select all

root@DietPi:~# find / -name boot.img
/lib/firmware/RTL8192E/boot.img
/mnt/dietpi-backup/data/lib/firmware/RTL8192E/boot.img


root@DietPi:~# ls -l /
total 76
drwxr-xr-x   2 root root  4096 Oct  7 01:23 bin
drwxr-xr-x   3 root root  4096 Oct 14 17:54 boot
drwxr-xr-x  16 root root  3540 Oct 14 17:53 dev
lrwxrwxrwx   1 root root     5 Jun 12  2020 DietPi -> /boot
drwxr-xr-x  99 root root  4096 Oct 14 15:28 etc
drwxr-xr-x   3 root root  4096 Jul 24  2018 home
drwxr-xr-x  17 root root  4096 Oct 14 15:26 lib
drwx------   2 root root 16384 Jul 24  2018 lost+found
drwxr-xr-x   5 root root  4096 Oct  2  2018 media
drwxr-xr-x   7 root root  4096 Nov  2  2018 mnt
drwxr-xr-x   5 root root  4096 Jul 19 22:44 opt
drwxr-xr-x   3 root root  4096 Oct 14 00:24 path
dr-xr-xr-x 198 root root     0 Jan  1  1970 proc
drwxr-xr-x  10 root root  4096 Oct 14 17:53 root
drwxr-xr-x  25 root root   760 Oct 14 17:53 run
drwxr-xr-x   2 root root  4096 Oct 14 15:26 sbin
drwxr-xr-x   2 root root  4096 Jul 24  2018 srv
dr-xr-xr-x  14 root root     0 Oct 14 20:26 sys
drwxr-xr-x   4 root root  4096 Jul 24  2018 system
drwxrwxrwt  12 root root   340 Oct 14 17:54 tmp
drwxr-xr-x  11 root root  4096 Sep 29  2018 usr
drwxr-xr-x  12 root root  4096 Dec 28  2018 var

Code: Select all

root@DietPi:~# cat /proc/cpuinfo
processor       : 0
BogoMIPS        : 48.00
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
CPU implementer : 0x41
CPU architecture: 8
CPU variant     : 0x0
CPU part        : 0xd03
CPU revision    : 4

processor       : 1
BogoMIPS        : 48.00
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
CPU implementer : 0x41
CPU architecture: 8
CPU variant     : 0x0
CPU part        : 0xd03
CPU revision    : 4

processor       : 2
BogoMIPS        : 48.00
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
CPU implementer : 0x41
CPU architecture: 8
CPU variant     : 0x0
CPU part        : 0xd03
CPU revision    : 4

processor       : 3
BogoMIPS        : 48.00
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
CPU implementer : 0x41
CPU architecture: 8
CPU variant     : 0x0
CPU part        : 0xd03
CPU revision    : 4

processor       : 4
BogoMIPS        : 48.00
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
CPU implementer : 0x41
CPU architecture: 8
CPU variant     : 0x0
CPU part        : 0xd08
CPU revision    : 2

processor       : 5
BogoMIPS        : 48.00
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
CPU implementer : 0x41
CPU architecture: 8
CPU variant     : 0x0
CPU part        : 0xd08
CPU revision    : 2

Serial          : d06b6174c57cd489
root@DietPi:~# cat /proc/device-tree/model
FriendlyElec NanoPC-T4root@DietPi:~#
I played a bit with images from FriendlyElec long time ago,
but I quickly switched to DietPi once I discovered it.

Daniel
User avatar
MichaIng
Site Admin
Posts: 3357
Joined: Sat Nov 18, 2017 6:21 pm

Re: Cannot start docker containers

Post by MichaIng »

That boot.img seems like a firmware, not a linux image, which should be called something with "linux" in its same. And most importantly it should be stored somewhere in the boot partition. In theory a bootloader can load it from everywhere, but there isn't some bootloader configuration or an initramfs, device tree or something like that, so I wonder how this an even boot.

I remember we had a GPT image for NanoPC T4 once, installed via some Android tools based installer image. Can you check whether there is an EFI partition?

Code: Select all

fdisk -l "$(lsblk -npo PKNAME "$G_ROOTFS_DEV")"
ivenhov
Posts: 6
Joined: Thu Oct 14, 2021 4:45 pm

Re: Cannot start docker containers

Post by ivenhov »

Hi

Here's the output

Code: Select all

root@DietPi:~# fdisk -l "$(lsblk -npo PKNAME "$G_ROOTFS_DEV")"
Disk /dev/mmcblk1: 14.56 GiB, 15634268160 bytes, 30535680 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Looking for linux file

Code: Select all

root@DietPi:~# find / -iname *linux* -size +10M -exec ls -lh {} \;
-rwxr-xr-x 1 root root 21M Jan 10  2021 /usr/bin/aarch64-linux-gnu-lto-dump-10
Daniel
ivenhov
Posts: 6
Joined: Thu Oct 14, 2021 4:45 pm

Re: Cannot start docker containers

Post by ivenhov »

My NanoPC-T4 is using NVMe if that changes anything.

Thanks
Daniel
User avatar
MichaIng
Site Admin
Posts: 3357
Joined: Sat Nov 18, 2017 6:21 pm

Re: Cannot start docker containers

Post by MichaIng »

The root filesystem is not located on an NVMe drive but on an SD card or eMMC /dev/mmcblk1. You should see the same when running df and the root filesystem with e.g. /dev/mmcblk1p1 as source.

Strange is that the root drive itself seems to not contain any partition table. This looks the same on my external drives where I wrote the filesystem directory onto the raw drive without creating a partition table first. While this works for a root filesystem mount and external mounts, this cannot be used to boot a system from, so obviously you boot from a different drive.

Probably the system is booted (bootloader/EFI) from the NVMe while it then loads the root filesystem from the SD/eMMC drive? That would explain why we do not find any kernel. Can you show the following:

Code: Select all

df
fdisk -l
cat /etc/fstab
I didn't mean a file with "linux" in its name, but a package ;):

Code: Select all

dpkg -l | grep 'linux'
ivenhov
Posts: 6
Joined: Thu Oct 14, 2021 4:45 pm

Re: Cannot start docker containers

Post by ivenhov »

Output included
Thanks for your help with this so far

Code: Select all

root@DietPi:~# ls -la /dev/ | grep mmc
brw-rw----  1 root disk    179,   0 Oct 14 17:53 mmcblk1
brw-rw----  1 root disk    179,  32 Oct 14 17:53 mmcblk1boot0
brw-rw----  1 root disk    179,  64 Oct 14 17:53 mmcblk1boot1
brw-rw----  1 root disk    179,   1 Oct 14 17:53 mmcblk1p1
brw-rw----  1 root disk    179,   2 Oct 14 17:53 mmcblk1p2
brw-rw----  1 root disk    179,   3 Oct 14 17:53 mmcblk1p3
brw-rw----  1 root disk    179,   4 Oct 14 17:53 mmcblk1p4
brw-rw----  1 root disk    179,   5 Oct 14 17:53 mmcblk1p5
brw-rw----  1 root disk    179,   6 Oct 14 17:53 mmcblk1p6
brw-rw----  1 root disk    179,  96 Oct 14 17:53 mmcblk1rpmb
root@DietPi:~#

root@DietPi:~# df
Filesystem     1K-blocks      Used Available Use% Mounted on
/dev/root       14631928   9864784   4149012  71% /
devtmpfs         1974228         0   1974228   0% /dev
tmpfs            1975132         8   1975124   1% /dev/shm
tmpfs             790056     84984    705072  11% /run
tmpfs               5120         4      5116   1% /run/lock
tmpfs            1984512      1844   1982668   1% /tmp
tmpfs              51200      4340     46860   9% /var/log
/dev/sda1      961303548 895590668  16858372  99% /media/Toshiba1TB
root@DietPi:~# fdisk -l
Disk /dev/ram0: 4 MiB, 4194304 bytes, 8192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/mmcblk1: 14.56 GiB, 15634268160 bytes, 30535680 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mmcblk1boot1: 4 MiB, 4194304 bytes, 8192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mmcblk1boot0: 4 MiB, 4194304 bytes, 8192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/sda: 931.51 GiB, 1000204883968 bytes, 1953525164 sectors
Disk model: External USB 3.0
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x33437ca8

Device     Boot Start        End    Sectors   Size Id Type
/dev/sda1          63 1953525163 1953525101 931.5G 83 Linux
root@DietPi:~# cat /etc/fstab
# You can use "dietpi-drive_manager" to setup mounts.
# NB: It overwrites and re-creates physical drive mount entries on use.
#----------------------------------------------------------------
# NETWORK
#----------------------------------------------------------------


#----------------------------------------------------------------
# TMPFS
#----------------------------------------------------------------
tmpfs /tmp tmpfs defaults,size=1938M,noatime,nodev,nosuid,mode=1777 0 0
tmpfs /var/log tmpfs defaults,size=50m,noatime,nodev,nosuid,mode=1777 0 0

#----------------------------------------------------------------
# MISC: ecryptfs, vboxsf (VirtualBox shared folder), gluster, bind mounts
#----------------------------------------------------------------


#----------------------------------------------------------------
# SWAP SPACE
#----------------------------------------------------------------


#----------------------------------------------------------------
# PHYSICAL DRIVES
#----------------------------------------------------------------
UUID=2899cd3f-0af0-4395-934f-e72272e315e2 / ext4 noatime,lazytime,rw 0 1
UUID=e828d2ab-ce94-4235-8e0f-7590ceb3450b /media/Toshiba1TB ext4 noatime,lazytime,rw,nofail,noauto,x-systemd.automount
root@DietPi:~#
Post Reply