Page 1 of 1

Dockerd not launching

Posted: Sun Oct 18, 2020 11:57 am
by Rilic
Something happened to docker while I was asleep. I woke up to it not being able to read or write to the volumes, so I rebooted the server to find that the dietpi-service won't start.

Here's the error log if I try and run it manually using dockerd:

Code: Select all

dietpi@Ouroboros:/mnt/dietpi_userdata/docker$ sudo dockerd
INFO[2020-10-18T10:56:07.841865186+01:00] Starting up                                  
INFO[2020-10-18T10:56:07.845164779+01:00] parsed scheme: "unix"                         module=grpc
INFO[2020-10-18T10:56:07.845527816+01:00] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2020-10-18T10:56:07.845761020+01:00] ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock 0  <nil>}] <nil>}  module=grpc
INFO[2020-10-18T10:56:07.845967927+01:00] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2020-10-18T10:56:07.848953297+01:00] parsed scheme: "unix"                         module=grpc
INFO[2020-10-18T10:56:07.849027205+01:00] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2020-10-18T10:56:07.849080797+01:00] ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock 0  <nil>}] <nil>}  module=grpc
INFO[2020-10-18T10:56:07.849111020+01:00] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2020-10-18T10:56:10.047054240+01:00] [graphdriver] using prior storage driver: overlay2 
failed to start daemon: lstat /mnt/da5b9436-32b8-4bb0-890b-b8c842fa424c/dietpi_userdata/docker-data/image/overlay2/layerdb/sha256/c354f176fdd0c64e161958ecd579f0eb0b430d8b69d0c223c182e715964c55cd: bad message
What's going on there? Is there anything I can do to fix it without backing up volumes and reinstalling?

EDIT: Upon further inspection, it seems the directory it's trying to find does not exist or has some kind of error:

Code: Select all

root@Ouroboros:~# cd /mnt/da5b9436-32b8-4bb0-890b-b8c842fa424c/dietpi_userdata/docker-data/image/overlay2/layerdb/sha256/c354f176fdd0c64e161958ecd579f0eb0b430d8b69d0c223c182e715964c55cd
bash: cd: /mnt/da5b9436-32b8-4bb0-890b-b8c842fa424c/dietpi_userdata/docker-data/image/overlay2/layerdb/sha256/c354f176fdd0c64e161958ecd579f0eb0b430d8b69d0c223c182e715964c55cd: Bad message
root@Ouroboros:~# 
Is this file corruption...?

Re: Dockerd not launching

Posted: Sun Oct 18, 2020 1:37 pm
by Joulinar
Hi,

yes indeed looks like issues with your external HDD. You could try to cd into the directories one by one, to see where it stuck. As wll you could check the disk.

Code: Select all

fsck -a /dev/sdXX
sdXX would need to be replaced with your device name.

Re: Dockerd not launching

Posted: Sun Oct 18, 2020 4:42 pm
by Rilic
I took the drive out and plugged it into my main Linux machine. I ran the command you specified and this was the output:

Code: Select all

❯ sudo fsck -a /dev/sdd1
fsck from util-linux 2.36
/dev/sdd1 contains a file system with errors, check forced.
/dev/sdd1: Inodes that were part of a corrupted orphan linked list found.  

/dev/sdd1: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
        (i.e., without -a or -p options)
Do I do it? Or will that do something bad?

Re: Dockerd not launching

Posted: Sun Oct 18, 2020 10:56 pm
by Joulinar
well you could clone your disk before doing fsck /dev/sdd1 -y . The -y flag answers yes to all the prompts it would otherwise have given you. You don't have to use it but it is recommended.

Re: Dockerd not launching

Posted: Mon Oct 19, 2020 1:52 am
by Rilic
Doing that fixed the issue and now Docker can run again! But how did the corruption happen in the first place? I looked at smartctl and while the drive didn't support the SMART standard it seemed to pass basic health checks. Should I run a proper test?

EDIT: Nevermind, trying to interface with Docker is bringing up an issue. I'm trying to use my docker-compose file, but it comes up with this error:

Code: Select all

dietpi@Ouroboros:/mnt/dietpi_userdata/docker$ sudo docker-compose up -d
ERROR: layer does not exist
What could be causing that?

Re: Dockerd not launching

Posted: Mon Oct 19, 2020 3:02 am
by WarHawk
Might need to delay docker service from starting until after the OS mounts the drive/partition that the docker images are located on...

I had this problem and solved it by delaying the systemd configuration file to wait about 15-30 seconds while the rest of the OS stabilizes and mounts everything

The corruption might have happened when it tried to start docker before the OS mounted all the required share locations

Re: Dockerd not launching

Posted: Mon Oct 19, 2020 9:26 am
by Joulinar
could be that file system was corrected but here might be files still being corrupted. Probably you would need to recreated them or reload the affected image

Re: Dockerd not launching

Posted: Wed Oct 21, 2020 10:11 am
by Rilic
I fixed it! I just needed to redownload an image and delete a few conflicting files so the install could complete. Everything's back up and running now.

Is there a utility I can use in dietpi to check the health of a disk? I'm worried that it might be on its way out and that maybe that deal on a 2.5" Toshiba hdd was a bad idea :lol:

Re: Dockerd not launching

Posted: Wed Oct 21, 2020 12:00 pm
by Joulinar
ok good that it is solved for now. You can have a look to drive manger. There you have a Check & Repair option. But it's a manuell task. But I need to say, I never used it myself :roll: