Services don't start

Hey everyone, I’m having a very weird issue.

I have a PI 4 set up with Sonarr, Deluge and Jellyfin, but out of nowhere none of these services work.
I thought the IP had changed, but I was able to log in via ssh and restart dietpi-services with no problem (everything says “ok” there).
Do you have any ideas?

To start looking into it, can you share dietpi-services status

Here you go

[  OK  ] DietPi-Services | nmbd                 active (running) since Tue 2021-03-09 12:40:51 GMT; 23min ago
[  OK  ] DietPi-Services | smbd                 active (running) since Tue 2021-03-09 12:40:51 GMT; 23min ago
[FAILED] DietPi-Services | ● deluged.service - Deluge Daemon (DietPi)
   Loaded: loaded (/etc/systemd/system/deluged.service; disabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Tue 2021-03-09 12:41:00 GMT; 23min ago
     Docs: man:deluged
  Process: 1541 ExecStart=/usr/bin/deluged -d -l /var/log/deluged/daemon.log -L warning (code=exited, status=1/FAILURE)
 Main PID: 1541 (code=exited, status=1/FAILURE)

Mar 09 12:41:00 DietPi systemd[1]: deluged.service: Service RestartSec=100ms expired, scheduling restart.
Mar 09 12:41:00 DietPi systemd[1]: deluged.service: Scheduled restart job, restart counter is at 5.
Mar 09 12:41:00 DietPi systemd[1]: Stopped Deluge Daemon (DietPi).
Mar 09 12:41:00 DietPi systemd[1]: deluged.service: Start request repeated too quickly.
Mar 09 12:41:00 DietPi systemd[1]: deluged.service: Failed with result 'exit-code'.
Mar 09 12:41:00 DietPi systemd[1]: Failed to start Deluge Daemon (DietPi).
[  OK  ] DietPi-Services | jellyfin             active (running) since Tue 2021-03-09 13:04:11 GMT; 2s ago
[  OK  ] DietPi-Services | jackett              activating (auto-restart) since Tue 2021-03-09 13:04:13 GMT; 1s ago
[  OK  ] DietPi-Services | sonarr               active (running) since Tue 2021-03-09 12:40:51 GMT; 23min ago
[  OK  ] DietPi-Services | radarr               active (running) since Tue 2021-03-09 12:40:51 GMT; 23min ago
[  OK  ] DietPi-Services | deluge-web           active (running) since Tue 2021-03-09 12:40:51 GMT; 23min ago
[  OK  ] DietPi-Services | cron                 active (running) since Tue 2021-03-09 12:40:52 GMT; 23min ago
[  OK  ] DietPi-Services | dropbear             active (running) since Tue 2021-03-09 12:39:21 GMT; 24min ago
[  OK  ] DietPi-Services | dietpi-ramlog        active (exited) since Tue 2021-03-09 12:39:01 GMT; 25min ago
[  OK  ] DietPi-Services | dietpi-preboot       active (exited) since Tue 2021-03-09 12:39:01 GMT; 25min ago
[  OK  ] DietPi-Services | dietpi-boot          active (exited) since Tue 2021-03-09 12:39:21 GMT; 24min ago
[  OK  ] DietPi-Services | dietpi-postboot      active (exited) since Tue 2021-03-09 12:39:21 GMT; 24min ago
[ INFO ] DietPi-Services | dietpi-wifi-monitor  inactive (dead)

ok let’s check deluged

systemctl restart deluged
cat /var/log/deluged/daemon.log

Ok, this starts making a bit of sense.

I have 2 HDs conneced to the pi and they do have space left. But I had a problem recently where one of the drives was completly wiped and I never knew why it happend. Could these problems be related?

Here’s the log:

[WARNING ] 14:48:46 preferencesmanager:528 GeoIP Unavailable
[ERROR   ] 14:48:46 main:248 [Errno 28] No space left on device
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/deluge/main.py", line 241, in start_daemon
    Daemon(options, args)
  File "/usr/lib/python2.7/dist-packages/deluge/core/daemon.py", line 176, in __init__
    _file.write("%s;%s\n" % (os.getpid(), port))
IOError: [Errno 28] No space left on device

No space left on device is quite a good reason for your system not starting correctly. Can you share df -hT

Here you go. It makes sense.

I believe this occured because of a secondary problem: when I go to the drive manager, the sdb shows as “no filesystem”. It was working perfectly 5 hours ago. It’s the same drive that was erased earlier. Probably deluge tried to download something to the internal memory since it couldn’t find the external HD.

I can reformat it using the drive manager, but how can I clean /dev/root?

Filesystem     Type      Size  Used Avail Use% Mounted on
/dev/root      ext4       34G   33G     0 100% /
devtmpfs       devtmpfs  3.7G     0  3.7G   0% /dev
tmpfs          tmpfs     3.7G   20K  3.7G   1% /dev/shm
tmpfs          tmpfs     3.7G  170M  3.6G   5% /run
tmpfs          tmpfs     5.0M     0  5.0M   0% /run/lock
tmpfs          tmpfs     3.7G     0  3.7G   0% /sys/fs/cgroup
tmpfs          tmpfs     3.9G     0  3.9G   0% /tmp
tmpfs          tmpfs      50M  2.2M   48M   5% /var/log
/dev/mmcblk0p1 vfat      253M   32M  221M  13% /boot

yes your rootFS is completely full. That’s why you have issues. what was located on that external HDD?

let’s check for biggest files. Following will give the top 20

du -a / | sort -n -r | head -n 20

As well, let’s have a look to your devices

lsblk -o name,fstype,label,size,ro,type,mountpoint,partuuid,uuid

Here it is. /disks is the mount location of my external hard drives.
edit: the other under /disks are just files inside this folder.

du: cannot access '/proc/14104/task/14104/fd/3': No such file or directory
du: cannot access '/proc/14104/task/14104/fdinfo/3': No such file or directory
du: cannot access '/proc/14104/fd/4': No such file or directory
du: cannot access '/proc/14104/fdinfo/4': No such file or directory
987468250       /
984880012       /disks



NAME        FSTYPE LABEL   SIZE RO TYPE MOUNTPOINT    PARTUUID                             UUID
sda                        3.7T  0 disk                                                    
├─sda1      ext4           200M  0 part               af39fadd-4f71-4fbe-ae82-551205a03e7e 284ce4db-751c-47bf-a01b-2a6c2348f3d4
├─sda2      ext4   TV      1.8T  0 part /disks/TV     28821df4-0dde-4d1a-89ee-1008df2b103c 1443803a-7890-4f73-9b66-87d206f4237b
└─sda3      ext4   Movies  1.8T  0 part /disks/Movies 274406d9-e205-4007-a76e-d6d75fe333d6 55d1999c-a676-4194-8d0f-955bdd83a36c
sdb                        3.7T  0 disk                                                    
mmcblk0                   59.5G  0 disk                                                    
├─mmcblk0p1 vfat   boot    256M  0 part /boot         ad09722e-01                          54E3-79CE
└─mmcblk0p2 ext4   rootfs 59.2G  0 part /             ad09722e-02                          c6dd3b94-a789-4d57-9080-1472f721804b

some interesting points,

your /disks - sda2 and sda3 are not showing up on the command df -hT :thinking:
While it seems to be mounted

Next point would be your SD card. Looks like you have a 60GB on rootFS

mmcblk0p2 ext4   rootfs 59.2G  0 part /

while the mounted file system shows 34 GB only :thinking:

Filesystem     Type      Size  Used Avail Use% Mounted on
/dev/root      ext4       34G   33G     0 100% /

Maybe it was not extended fully.


Anyway, let’s check what it blocking your rootFS and exclude sda2 and sda3

du -a / --exclude /disks/TV --exclude /disks/Movies | sort -n -r | head -n 20

Another option is to disconnect your HDD (sda) - if possible - and try to search again for large files.

Excluding /disks, this is all it shows. And yes, you’re right, I have a 64GB card as internal storage.

du: cannot access '/proc/1203/task/1203/fd/3': No such file or directory
du: cannot access '/proc/1203/task/1203/fdinfo/3': No such file or directory
du: cannot access '/proc/1203/fd/4': No such file or directory
du: cannot access '/proc/1203/fdinfo/4': No such file or directory
2389578 /
1133120 /usr
875492  /usr/lib
541584  /mnt
541544  /mnt/dietpi_userdata
498948  /mnt/dietpi_userdata/jellyfin
454104  /mnt/dietpi_userdata/jellyfin/metadata
376820  /opt
367408  /usr/lib/mono
351356  /mnt/dietpi_userdata/jellyfin/metadata/People
232604  /opt/radarr
220016  /lib
197288  /usr/share
163568  /usr/lib/aarch64-linux-gnu
137400  /usr/lib/jellyfin
137392  /usr/lib/jellyfin/bin
123544  /opt/jackett
106496  /usr/lib/mono/aot-cache
106492  /usr/lib/mono/aot-cache/arm64
100112  /usr/lib/python2.7

did you exclude the entire /disks folder or just the 2 directories /disks/TV + /disks/Movies? Because it is only showing 2.3GB now. This doesn’t fit to the 33GB used on rootFS. Interesting system you have :thinking:

The entire /disks. When I was excluding both subfolders, it only showed the “cannot access” lines, for some reason.

it’s just a guess but probably most of your data are located somewhere inside /disks/ but not in /disks/TV + /disks/Movies as this is another HDD.

For testing are you able to unmount /disks/TV + /disks/Movies ?? And run du -a / | sort -n -r | head -n 20 again

Btw: are you running this as user root ?

This is so weird. I tried unmounting the drives on drive manager but they are not mounted.
Just to make sure, I physically unplugged both HDs, rebooted the Pi and ran du -a / | sort -n -r | head -n 20 again.
Here’s what I got:

du: cannot access '/proc/797/task/797/fd/3': No such file or directory
du: cannot access '/proc/797/task/797/fdinfo/3': No such file or directory
du: cannot access '/proc/797/fd/4': No such file or directory
du: cannot access '/proc/797/fdinfo/4': No such file or directory
du: cannot access '/proc/973': No such file or directory
33634578        /
31245248        /disks
28521252        /disks/Software

And then subfolders of /disks/Software and /disks/Docs, that are partitions of the other HD and that it’s unplugged.
Also, yes I’m on root.

Quick update: I went to check /etc/fstab and the auto-mounts I had set up for the drives were completely wiped out.

we are getting closer :wink:

maybe /etc/fstab has been updated by drive manager while the disks are not mounted. This could lead to a clean up. But can be fixed once you connect the HDD

ok as your 2nd HDD vanished some time ago, all data are written to rootFS now, which hold 28GB on /disks/Software atm. You would need to move this data to the 2nd HDD or perform housekeeping to free up space

We are VERY close, but I’m still a bit lost. Here’s my fstab setup:

UUID=1443803a-7890-4f73-9b66-87d206f4237b /disks/TV ext4 noatime,lazytime,rw,nofail,noauto,x-systemd.automount
UUID=55d1999c-a676-4194-8d0f-955bdd83a36c /disks/Movies ext4 noatime,lazytime,rw,nofail,noauto,x-systemd.automount
UUID=199f3024-72ad-4fd8-9d23-933e9e3165e6 /disks/Docs ext4 noatime,lazytime,rw,nofail,noauto,x-systemd.automount
UUID=39ac04ac-3fd7-472e-a87e-f4269aeea48a /disks/Software ext4 noatime,lazytime,rw,nofail,noauto,x-systemd.automount
UUID=8693c0da-fac5-4289-ab38-846234b2d266 /disks/DietPi ext4 noatime,lazytime,rw,nofail,noauto,x-systemd.automount

The /disks/DietPi is where my data was supposed to be. When the fstab was erased, the data had nowhere to go, as you said.

I updated /etc/fstab again, connected the HDDs and restarted the Pi. All partitions showed up (confirmed by samba, I can see all files in all partitions in all HDDs like nothing ever happened), except for /disks/DietPi. Still, It was detected, so I went to Drive Manager and tried to move rootFS to the partition. It was a success, but when I restarted the Pi the services still won’t work and disks/DietPi was commented out of /etc/fstab. I’m not sure what to do next.

can you share lsblk -o name,fstype,label,size,ro,type,mountpoint,partuuid,uuid

It says -bash: lsblk: command not found. Should I install something?