Dockerd not launching Topic is solved

Having issues with your DietPi installation, or, found a bug? Post it here.
Post Reply
User avatar
Rilic
Posts: 46
Joined: Thu May 07, 2020 4:14 pm

Dockerd not launching

Post by Rilic »

Something happened to docker while I was asleep. I woke up to it not being able to read or write to the volumes, so I rebooted the server to find that the dietpi-service won't start.

Here's the error log if I try and run it manually using dockerd:

Code: Select all

dietpi@Ouroboros:/mnt/dietpi_userdata/docker$ sudo dockerd
INFO[2020-10-18T10:56:07.841865186+01:00] Starting up                                  
INFO[2020-10-18T10:56:07.845164779+01:00] parsed scheme: "unix"                         module=grpc
INFO[2020-10-18T10:56:07.845527816+01:00] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2020-10-18T10:56:07.845761020+01:00] ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock 0  <nil>}] <nil>}  module=grpc
INFO[2020-10-18T10:56:07.845967927+01:00] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2020-10-18T10:56:07.848953297+01:00] parsed scheme: "unix"                         module=grpc
INFO[2020-10-18T10:56:07.849027205+01:00] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2020-10-18T10:56:07.849080797+01:00] ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock 0  <nil>}] <nil>}  module=grpc
INFO[2020-10-18T10:56:07.849111020+01:00] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2020-10-18T10:56:10.047054240+01:00] [graphdriver] using prior storage driver: overlay2 
failed to start daemon: lstat /mnt/da5b9436-32b8-4bb0-890b-b8c842fa424c/dietpi_userdata/docker-data/image/overlay2/layerdb/sha256/c354f176fdd0c64e161958ecd579f0eb0b430d8b69d0c223c182e715964c55cd: bad message
What's going on there? Is there anything I can do to fix it without backing up volumes and reinstalling?

EDIT: Upon further inspection, it seems the directory it's trying to find does not exist or has some kind of error:

Code: Select all

root@Ouroboros:~# cd /mnt/da5b9436-32b8-4bb0-890b-b8c842fa424c/dietpi_userdata/docker-data/image/overlay2/layerdb/sha256/c354f176fdd0c64e161958ecd579f0eb0b430d8b69d0c223c182e715964c55cd
bash: cd: /mnt/da5b9436-32b8-4bb0-890b-b8c842fa424c/dietpi_userdata/docker-data/image/overlay2/layerdb/sha256/c354f176fdd0c64e161958ecd579f0eb0b430d8b69d0c223c182e715964c55cd: Bad message
root@Ouroboros:~# 
Is this file corruption...?
User avatar
Joulinar
Posts: 2638
Joined: Sat Nov 16, 2019 12:49 am

Re: Dockerd not launching

Post by Joulinar »

Hi,

yes indeed looks like issues with your external HDD. You could try to cd into the directories one by one, to see where it stuck. As wll you could check the disk.

Code: Select all

fsck -a /dev/sdXX
sdXX would need to be replaced with your device name.
Pls let us know if a solution is working. This could help others if they hit by similar situation. Your DietPi Team
User avatar
Rilic
Posts: 46
Joined: Thu May 07, 2020 4:14 pm

Re: Dockerd not launching

Post by Rilic »

I took the drive out and plugged it into my main Linux machine. I ran the command you specified and this was the output:

Code: Select all

❯ sudo fsck -a /dev/sdd1
fsck from util-linux 2.36
/dev/sdd1 contains a file system with errors, check forced.
/dev/sdd1: Inodes that were part of a corrupted orphan linked list found.  

/dev/sdd1: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
        (i.e., without -a or -p options)
Do I do it? Or will that do something bad?
User avatar
Joulinar
Posts: 2638
Joined: Sat Nov 16, 2019 12:49 am

Re: Dockerd not launching

Post by Joulinar »

well you could clone your disk before doing fsck /dev/sdd1 -y . The -y flag answers yes to all the prompts it would otherwise have given you. You don't have to use it but it is recommended.
Pls let us know if a solution is working. This could help others if they hit by similar situation. Your DietPi Team
User avatar
Rilic
Posts: 46
Joined: Thu May 07, 2020 4:14 pm

Re: Dockerd not launching

Post by Rilic »

Doing that fixed the issue and now Docker can run again! But how did the corruption happen in the first place? I looked at smartctl and while the drive didn't support the SMART standard it seemed to pass basic health checks. Should I run a proper test?

EDIT: Nevermind, trying to interface with Docker is bringing up an issue. I'm trying to use my docker-compose file, but it comes up with this error:

Code: Select all

dietpi@Ouroboros:/mnt/dietpi_userdata/docker$ sudo docker-compose up -d
ERROR: layer does not exist
What could be causing that?
User avatar
WarHawk
Posts: 635
Joined: Thu Jul 20, 2017 8:55 am

Re: Dockerd not launching

Post by WarHawk »

Might need to delay docker service from starting until after the OS mounts the drive/partition that the docker images are located on...

I had this problem and solved it by delaying the systemd configuration file to wait about 15-30 seconds while the rest of the OS stabilizes and mounts everything

The corruption might have happened when it tried to start docker before the OS mounted all the required share locations
User avatar
Joulinar
Posts: 2638
Joined: Sat Nov 16, 2019 12:49 am

Re: Dockerd not launching

Post by Joulinar »

could be that file system was corrected but here might be files still being corrupted. Probably you would need to recreated them or reload the affected image
Pls let us know if a solution is working. This could help others if they hit by similar situation. Your DietPi Team
User avatar
Rilic
Posts: 46
Joined: Thu May 07, 2020 4:14 pm

Re: Dockerd not launching

Post by Rilic »

I fixed it! I just needed to redownload an image and delete a few conflicting files so the install could complete. Everything's back up and running now.

Is there a utility I can use in dietpi to check the health of a disk? I'm worried that it might be on its way out and that maybe that deal on a 2.5" Toshiba hdd was a bad idea :lol:
User avatar
Joulinar
Posts: 2638
Joined: Sat Nov 16, 2019 12:49 am

Re: Dockerd not launching

Post by Joulinar »

ok good that it is solved for now. You can have a look to drive manger. There you have a Check & Repair option. But it's a manuell task. But I need to say, I never used it myself :roll:
Pls let us know if a solution is working. This could help others if they hit by similar situation. Your DietPi Team
Post Reply