Something happened to docker while I was asleep. I woke up to it not being able to read or write to the volumes, so I rebooted the server to find that the dietpi-service won’t start.
Here’s the error log if I try and run it manually using dockerd:
dietpi@Ouroboros:/mnt/dietpi_userdata/docker$ sudo dockerd
INFO[2020-10-18T10:56:07.841865186+01:00] Starting up
INFO[2020-10-18T10:56:07.845164779+01:00] parsed scheme: "unix" module=grpc
INFO[2020-10-18T10:56:07.845527816+01:00] scheme "unix" not registered, fallback to default scheme module=grpc
INFO[2020-10-18T10:56:07.845761020+01:00] ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock 0 <nil>}] <nil>} module=grpc
INFO[2020-10-18T10:56:07.845967927+01:00] ClientConn switching balancer to "pick_first" module=grpc
INFO[2020-10-18T10:56:07.848953297+01:00] parsed scheme: "unix" module=grpc
INFO[2020-10-18T10:56:07.849027205+01:00] scheme "unix" not registered, fallback to default scheme module=grpc
INFO[2020-10-18T10:56:07.849080797+01:00] ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock 0 <nil>}] <nil>} module=grpc
INFO[2020-10-18T10:56:07.849111020+01:00] ClientConn switching balancer to "pick_first" module=grpc
INFO[2020-10-18T10:56:10.047054240+01:00] [graphdriver] using prior storage driver: overlay2
failed to start daemon: lstat /mnt/da5b9436-32b8-4bb0-890b-b8c842fa424c/dietpi_userdata/docker-data/image/overlay2/layerdb/sha256/c354f176fdd0c64e161958ecd579f0eb0b430d8b69d0c223c182e715964c55cd: bad message
What’s going on there? Is there anything I can do to fix it without backing up volumes and reinstalling?
EDIT: Upon further inspection, it seems the directory it’s trying to find does not exist or has some kind of error:
root@Ouroboros:~# cd /mnt/da5b9436-32b8-4bb0-890b-b8c842fa424c/dietpi_userdata/docker-data/image/overlay2/layerdb/sha256/c354f176fdd0c64e161958ecd579f0eb0b430d8b69d0c223c182e715964c55cd
bash: cd: /mnt/da5b9436-32b8-4bb0-890b-b8c842fa424c/dietpi_userdata/docker-data/image/overlay2/layerdb/sha256/c354f176fdd0c64e161958ecd579f0eb0b430d8b69d0c223c182e715964c55cd: Bad message
root@Ouroboros:~#
yes indeed looks like issues with your external HDD. You could try to cd into the directories one by one, to see where it stuck. As wll you could check the disk.
fsck -a /dev/sdXX
sdXX would need to be replaced with your device name.
I took the drive out and plugged it into my main Linux machine. I ran the command you specified and this was the output:
❯ sudo fsck -a /dev/sdd1
fsck from util-linux 2.36
/dev/sdd1 contains a file system with errors, check forced.
/dev/sdd1: Inodes that were part of a corrupted orphan linked list found.
/dev/sdd1: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
(i.e., without -a or -p options)
well you could clone your disk before doing fsck /dev/sdd1 -y . The -y flag answers yes to all the prompts it would otherwise have given you. You don’t have to use it but it is recommended.
Doing that fixed the issue and now Docker can run again! But how did the corruption happen in the first place? I looked at smartctl and while the drive didn’t support the SMART standard it seemed to pass basic health checks. Should I run a proper test?
EDIT: Nevermind, trying to interface with Docker is bringing up an issue. I’m trying to use my docker-compose file, but it comes up with this error:
dietpi@Ouroboros:/mnt/dietpi_userdata/docker$ sudo docker-compose up -d
ERROR: layer does not exist
Might need to delay docker service from starting until after the OS mounts the drive/partition that the docker images are located on…
I had this problem and solved it by delaying the systemd configuration file to wait about 15-30 seconds while the rest of the OS stabilizes and mounts everything
The corruption might have happened when it tried to start docker before the OS mounted all the required share locations
could be that file system was corrected but here might be files still being corrupted. Probably you would need to recreated them or reload the affected image
I fixed it! I just needed to redownload an image and delete a few conflicting files so the install could complete. Everything’s back up and running now.
Is there a utility I can use in dietpi to check the health of a disk? I’m worried that it might be on its way out and that maybe that deal on a 2.5" Toshiba hdd was a bad idea
ok good that it is solved for now. You can have a look to drive manger. There you have a Check & Repair option. But it’s a manuell task. But I need to say, I never used it myself
