Cannot start docker containers

Hi all

I’m on NanoPC-TV with dietpi migrated to bullseye.
I’m trying to install docker and start a hello-world container.
Upon installation via dietpi-software I get error

Selecting previously unselected package docker-ce-cli.
(Reading database ... 46985 files and directories currently installed.)
Preparing to unpack .../docker-ce-cli_5%3a20.10.9~3-0~debian-bullseye_arm64.deb ...
Unpacking docker-ce-cli (5:20.10.9~3-0~debian-bullseye) ...
Selecting previously unselected package docker-ce.
Preparing to unpack .../docker-ce_5%3a20.10.9~3-0~debian-bullseye_arm64.deb ...
Unpacking docker-ce (5:20.10.9~3-0~debian-bullseye) ...
Setting up docker-ce-cli (5:20.10.9~3-0~debian-bullseye) ...
Setting up docker-ce (5:20.10.9~3-0~debian-bullseye) ...
Job for docker.service failed because the control process exited with error code.
See "systemctl status docker.service" and "journalctl -xe" for details.
invoke-rc.d: initscript docker, action "restart" failed.
● docker.service - Docker Application Container Engine
     Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
     Active: activating (auto-restart) (Result: exit-code) since Thu 2021-10-14 14:59:58 BST; 33ms ago
TriggeredBy: ● docker.socket
       Docs: https://docs.docker.com
    Process: 40474 ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock (code=exited, status=1/FAILURE)
   Main PID: 40474 (code=exited, status=1/FAILURE)

Oct 14 14:59:58 DietPi systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Oct 14 14:59:58 DietPi systemd[1]: docker.service: Failed with result 'exit-code'.
Oct 14 14:59:58 DietPi systemd[1]: Failed to start Docker Application Container Engine.
dpkg: error processing package docker-ce (--configure):
 installed docker-ce package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 docker-ce
E: Sub-process /usr/bin/dpkg returned an error code (1)

I can however start service later using

service docker start

root@DietPi:~# service docker status
● docker.service - Docker Application Container Engine
     Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2021-10-14 15:28:20 BST; 22min ago
TriggeredBy: ● docker.socket
       Docs: https://docs.docker.com
   Main PID: 45030 (dockerd)
      Tasks: 11
     Memory: 26.0M
     CGroup: /system.slice/docker.service
             └─45030 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

Oct 14 15:28:20 DietPi dockerd[45030]: time="2021-10-14T15:28:20.452413178+01:00" level=info msg="Loading containers: done."
Oct 14 15:28:20 DietPi dockerd[45030]: time="2021-10-14T15:28:20.510148970+01:00" level=info msg="Docker daemon" commit=79ea9d3 graphdriver(s)=overlay2 version=20.10.9
Oct 14 15:28:20 DietPi dockerd[45030]: time="2021-10-14T15:28:20.510731428+01:00" level=info msg="Daemon has completed initialization"
Oct 14 15:28:20 DietPi systemd[1]: Started Docker Application Container Engine.
Oct 14 15:28:20 DietPi dockerd[45030]: time="2021-10-14T15:28:20.652845678+01:00" level=info msg="API listen on /run/docker.sock"
Oct 14 15:39:00 DietPi dockerd[45030]: time="2021-10-14T15:39:00.191039275+01:00" level=warning msg="Your kernel does not support cgroup namespaces.  Cgroup namespace setting discarded."
Oct 14 15:39:00 DietPi dockerd[45030]: time="2021-10-14T15:39:00.392383608+01:00" level=error msg="stream copy error: reading from a closed fifo"
Oct 14 15:39:00 DietPi dockerd[45030]: time="2021-10-14T15:39:00.392588233+01:00" level=error msg="stream copy error: reading from a closed fifo"
Oct 14 15:39:00 DietPi dockerd[45030]: time="2021-10-14T15:39:00.521419150+01:00" level=error msg="41b3e2ac426a574a9ec1c915056e1eac9e1a31ab566febef23331ee99d2636ed cleanup: failed to delete container from containerd: no>
Oct 14 15:39:00 DietPi dockerd[45030]: time="2021-10-14T15:39:00.521791025+01:00" level=error msg="Handler for POST /v1.41/containers/41b3e2ac426a574a9ec1c915056e1eac9e1a31ab566febef23331ee99d2636ed/start returned error>
root@DietPi:~#

When I try to start a container I get following error

root@DietPi:~# docker -D run --name hello-world hello-world
DEBU[0000] [hijack] End of stdout
docker: Error response from daemon: OCI runtime create failed: cgroup namespaces aren't enabled in the kernel: unknown.
ERRO[0000] error waiting for container: context canceled

How to fix this?

System

root@DietPi:~# uname -a
Linux DietPi 4.4.126 #7 SMP Thu Jul 19 22:51:16 EST 2018 aarch64 GNU/Linux
root@DietPi:~# cat /etc/*release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
root@DietPi:~#

Some info

root@DietPi:~# docker -v
Docker version 20.10.9, build c2ea9bc

root@DietPi:~# cat /proc/cgroups
#subsys_name    hierarchy       num_cgroups     enabled
cpuset  0       111     1
cpu     0       111     1
cpuacct 0       111     1
blkio   0       111     1
memory  0       111     1
devices 0       111     1
freezer 0       111     1
net_cls 0       111     1
perf_event      0       111     1
net_prio        0       111     1
hugetlb 0       111     1
pids    0       111     1
debug   0       111     1
root@DietPi:~# mount | grep cgroup
cgroup2 on /sys/fs/cgroup type cgroup2 (rw,nosuid,nodev,noexec,relatime)

basically we had a similar report on Ordroid device running similar old kernel. https://github.com/MichaIng/DietPi/issues/4705

Solution on this case was to set systemd.unified_cgroup_hierarchy=0 to the in boot.ini. Maybe you could check if this is helping you as well.

Thanks for this.
It looks like the right direction!

I cannot however locate boot.ini on my system

root@DietPi:~# ls -la /DietPi/
total 32
drwxr-xr-x  3 root root  4096 Oct 14 17:54 .
drwxr-xr-x 23 root root  4096 Oct 14 00:24 ..
drwxr-xr-x  4 root root  4096 Oct 14 17:42 dietpi
-rwxr-xr-x  1 root root  6129 Jun 10  2018 dietpi-README.md
-rwxr-xr-x  1 root root 11182 Oct 14 17:34 dietpi.txt
root@DietPi:~# ls -la /boot/
total 32
drwxr-xr-x  3 root root  4096 Oct 14 17:54 .
drwxr-xr-x 23 root root  4096 Oct 14 00:24 ..
drwxr-xr-x  4 root root  4096 Oct 14 17:42 dietpi
-rwxr-xr-x  1 root root  6129 Jun 10  2018 dietpi-README.md
-rwxr-xr-x  1 root root 11182 Oct 14 17:34 dietpi.txt
root@DietPi:~#

I used find command but the file does not exist.
I’ve created it under /DierPi/boot.ini with a single line

systemd.unified_cgroup_hierarchy=0

but that did not solve the problem, likely because I’m modifying wrong file.
Any further help would be appreciated.

Regards
Daniel

Strange thing, where is the kernel, actually? Can you show:

ls -l /

And also to identify the model (as I cannot find a “NanoPC TV”, only a NanoPC T4?):

cat /proc/cpuinfo
cat /proc/device-tree/model

Where did you get the image from originally?

Hi
Sorry about spelling mistake, that is of course NanoPC-T4
https://www.friendlyarm.com/index.php?route=product/product&product_id=225

I guess kernel is in boot.img ?

root@DietPi:~# find / -name boot.img
/lib/firmware/RTL8192E/boot.img
/mnt/dietpi-backup/data/lib/firmware/RTL8192E/boot.img


root@DietPi:~# ls -l /
total 76
drwxr-xr-x   2 root root  4096 Oct  7 01:23 bin
drwxr-xr-x   3 root root  4096 Oct 14 17:54 boot
drwxr-xr-x  16 root root  3540 Oct 14 17:53 dev
lrwxrwxrwx   1 root root     5 Jun 12  2020 DietPi -> /boot
drwxr-xr-x  99 root root  4096 Oct 14 15:28 etc
drwxr-xr-x   3 root root  4096 Jul 24  2018 home
drwxr-xr-x  17 root root  4096 Oct 14 15:26 lib
drwx------   2 root root 16384 Jul 24  2018 lost+found
drwxr-xr-x   5 root root  4096 Oct  2  2018 media
drwxr-xr-x   7 root root  4096 Nov  2  2018 mnt
drwxr-xr-x   5 root root  4096 Jul 19 22:44 opt
drwxr-xr-x   3 root root  4096 Oct 14 00:24 path
dr-xr-xr-x 198 root root     0 Jan  1  1970 proc
drwxr-xr-x  10 root root  4096 Oct 14 17:53 root
drwxr-xr-x  25 root root   760 Oct 14 17:53 run
drwxr-xr-x   2 root root  4096 Oct 14 15:26 sbin
drwxr-xr-x   2 root root  4096 Jul 24  2018 srv
dr-xr-xr-x  14 root root     0 Oct 14 20:26 sys
drwxr-xr-x   4 root root  4096 Jul 24  2018 system
drwxrwxrwt  12 root root   340 Oct 14 17:54 tmp
drwxr-xr-x  11 root root  4096 Sep 29  2018 usr
drwxr-xr-x  12 root root  4096 Dec 28  2018 var



root@DietPi:~# cat /proc/cpuinfo
processor       : 0
BogoMIPS        : 48.00
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
CPU implementer : 0x41
CPU architecture: 8
CPU variant     : 0x0
CPU part        : 0xd03
CPU revision    : 4

processor       : 1
BogoMIPS        : 48.00
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
CPU implementer : 0x41
CPU architecture: 8
CPU variant     : 0x0
CPU part        : 0xd03
CPU revision    : 4

processor       : 2
BogoMIPS        : 48.00
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
CPU implementer : 0x41
CPU architecture: 8
CPU variant     : 0x0
CPU part        : 0xd03
CPU revision    : 4

processor       : 3
BogoMIPS        : 48.00
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
CPU implementer : 0x41
CPU architecture: 8
CPU variant     : 0x0
CPU part        : 0xd03
CPU revision    : 4

processor       : 4
BogoMIPS        : 48.00
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
CPU implementer : 0x41
CPU architecture: 8
CPU variant     : 0x0
CPU part        : 0xd08
CPU revision    : 2

processor       : 5
BogoMIPS        : 48.00
Features        : fp asimd evtstrm aes pmull sha1 sha2 crc32
CPU implementer : 0x41
CPU architecture: 8
CPU variant     : 0x0
CPU part        : 0xd08
CPU revision    : 2

Serial          : d06b6174c57cd489
root@DietPi:~# cat /proc/device-tree/model
FriendlyElec NanoPC-T4root@DietPi:~#

I played a bit with images from FriendlyElec long time ago,
but I quickly switched to DietPi once I discovered it.

Daniel

That boot.img seems like a firmware, not a linux image, which should be called something with “linux” in its same. And most importantly it should be stored somewhere in the boot partition. In theory a bootloader can load it from everywhere, but there isn’t some bootloader configuration or an initramfs, device tree or something like that, so I wonder how this an even boot.

I remember we had a GPT image for NanoPC T4 once, installed via some Android tools based installer image. Can you check whether there is an EFI partition?

fdisk -l "$(lsblk -npo PKNAME "$G_ROOTFS_DEV")"

Hi

Here’s the output

root@DietPi:~# fdisk -l "$(lsblk -npo PKNAME "$G_ROOTFS_DEV")"
Disk /dev/mmcblk1: 14.56 GiB, 15634268160 bytes, 30535680 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

Looking for linux file

root@DietPi:~# find / -iname *linux* -size +10M -exec ls -lh {} \;
-rwxr-xr-x 1 root root 21M Jan 10  2021 /usr/bin/aarch64-linux-gnu-lto-dump-10

Daniel

My NanoPC-T4 is using NVMe if that changes anything.

Thanks
Daniel

The root filesystem is not located on an NVMe drive but on an SD card or eMMC /dev/mmcblk1. You should see the same when running df and the root filesystem with e.g. /dev/mmcblk1p1 as source.

Strange is that the root drive itself seems to not contain any partition table. This looks the same on my external drives where I wrote the filesystem directory onto the raw drive without creating a partition table first. While this works for a root filesystem mount and external mounts, this cannot be used to boot a system from, so obviously you boot from a different drive.

Probably the system is booted (bootloader/EFI) from the NVMe while it then loads the root filesystem from the SD/eMMC drive? That would explain why we do not find any kernel. Can you show the following:

df
fdisk -l
cat /etc/fstab

I didn’t mean a file with “linux” in its name, but a package :wink::

dpkg -l | grep 'linux'

Output included
Thanks for your help with this so far

root@DietPi:~# ls -la /dev/ | grep mmc
brw-rw----  1 root disk    179,   0 Oct 14 17:53 mmcblk1
brw-rw----  1 root disk    179,  32 Oct 14 17:53 mmcblk1boot0
brw-rw----  1 root disk    179,  64 Oct 14 17:53 mmcblk1boot1
brw-rw----  1 root disk    179,   1 Oct 14 17:53 mmcblk1p1
brw-rw----  1 root disk    179,   2 Oct 14 17:53 mmcblk1p2
brw-rw----  1 root disk    179,   3 Oct 14 17:53 mmcblk1p3
brw-rw----  1 root disk    179,   4 Oct 14 17:53 mmcblk1p4
brw-rw----  1 root disk    179,   5 Oct 14 17:53 mmcblk1p5
brw-rw----  1 root disk    179,   6 Oct 14 17:53 mmcblk1p6
brw-rw----  1 root disk    179,  96 Oct 14 17:53 mmcblk1rpmb
root@DietPi:~#

root@DietPi:~# df
Filesystem     1K-blocks      Used Available Use% Mounted on
/dev/root       14631928   9864784   4149012  71% /
devtmpfs         1974228         0   1974228   0% /dev
tmpfs            1975132         8   1975124   1% /dev/shm
tmpfs             790056     84984    705072  11% /run
tmpfs               5120         4      5116   1% /run/lock
tmpfs            1984512      1844   1982668   1% /tmp
tmpfs              51200      4340     46860   9% /var/log
/dev/sda1      961303548 895590668  16858372  99% /media/Toshiba1TB
root@DietPi:~# fdisk -l
Disk /dev/ram0: 4 MiB, 4194304 bytes, 8192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes


Disk /dev/mmcblk1: 14.56 GiB, 15634268160 bytes, 30535680 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mmcblk1boot1: 4 MiB, 4194304 bytes, 8192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/mmcblk1boot0: 4 MiB, 4194304 bytes, 8192 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes


Disk /dev/sda: 931.51 GiB, 1000204883968 bytes, 1953525164 sectors
Disk model: External USB 3.0
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x33437ca8

Device     Boot Start        End    Sectors   Size Id Type
/dev/sda1          63 1953525163 1953525101 931.5G 83 Linux
root@DietPi:~# cat /etc/fstab
# You can use "dietpi-drive_manager" to setup mounts.
# NB: It overwrites and re-creates physical drive mount entries on use.
#----------------------------------------------------------------
# NETWORK
#----------------------------------------------------------------


#----------------------------------------------------------------
# TMPFS
#----------------------------------------------------------------
tmpfs /tmp tmpfs defaults,size=1938M,noatime,nodev,nosuid,mode=1777 0 0
tmpfs /var/log tmpfs defaults,size=50m,noatime,nodev,nosuid,mode=1777 0 0

#----------------------------------------------------------------
# MISC: ecryptfs, vboxsf (VirtualBox shared folder), gluster, bind mounts
#----------------------------------------------------------------


#----------------------------------------------------------------
# SWAP SPACE
#----------------------------------------------------------------


#----------------------------------------------------------------
# PHYSICAL DRIVES
#----------------------------------------------------------------
UUID=2899cd3f-0af0-4395-934f-e72272e315e2 / ext4 noatime,lazytime,rw 0 1
UUID=e828d2ab-ce94-4235-8e0f-7590ceb3450b /media/Toshiba1TB ext4 noatime,lazytime,rw,nofail,noauto,x-systemd.automount
root@DietPi:~#

And dpkg -l | grep ‘linux’?

So I’m pretty sure your system is based on the old GPT installer image, which had this 6 partitions set. But to be true I don’t know much about it, this was basically before my time, our new image has a single partition to be flashed to the target drive directly. But let’s see what we can find out.

The above output would be good to know whether there is a kernel package installed at all, so a chance to upgrade it. And then we need to find the partition which does contains the boot configuration. I’m confused that fdisk -l doesn’t show any partitions, not even a partition table :thinking:. Can you show the output of:

lsblk -po NAME,SIZE,TYPE,FSTYPE,MOUNTPOINTS

Last resort is mounting all of them and checking the content:

mkdir /mnt/tmp
for i in /dev/mmcblk1?*
do
mount "$i" /mnt/tmp
ls -l /mnt/tmp
umount /mnt/tmp
sleep 1
done