No space left /dev/root

Troubleshooting
1

Hi,

since yesterday I’ve a problem with me RaspberryPi4 and the DietPi Installation.
The filesystem of /dev/root shows 100% fullness and I can’t figure out why neither how I can fix it or maybe I#m only to dumb :cry: .
Not all necessary services could be start.
Can anybody help me with this? It would be great if I can avoid a new installation and get an understanding for the problem and how it occurred :wink: .

Here are some information about my system:

  • DietP: v8.2.2
  • SD-Card: Western Digital WD Purple SC QD101 32GB
  • Device model : RPi 4 Model B (aarch64)
  • dietpi_userdata are switch to the extrenal HDD
  • system is only used for a nextcloud installation (version 23.0.3)

installed software:

Lighttpd: Extremely lightweight webserver                                                                
MariaDB: Persistent cached file-per-table database server                                                
PHP: Hypertext Preprocessor for dynamic web content                                                      
Redis: Volatile in-memory non-SQL database server                                                        
Certbot: Obtain and renew Let's Encrypt SSL certs for HTTPS                                              
DietPi-RAMlog: Makes /var/log a RAM disk, preserves file structure on reboot                             
Dropbear: Lightweight SSH server                                                                         
Nextcloud: File sync, sharing and collaboration platform                                                 
Neovim: heavily refactored vim fork

Filesystem Infos:

root@DietPi:/# df -hT
Dateisystem    Typ      Größe Benutzt Verf. Verw% Eingehängt auf
/dev/root      ext4       30G     30G     0  100% /
devtmpfs       devtmpfs  3,9G       0  3,9G    0% /dev
tmpfs          tmpfs     3,9G       0  3,9G    0% /dev/shm
tmpfs          tmpfs     1,6G     17M  1,6G    2% /run
tmpfs          tmpfs     5,0M       0  5,0M    0% /run/lock
tmpfs          tmpfs     3,9G       0  3,9G    0% /tmp
tmpfs          tmpfs      50M     24K   50M    1% /var/log
/dev/mmcblk0p1 vfat      127M     35M   92M   28% /boot
/dev/sda       ext4      1,8T    115G  1,6T    7% /mnt/monkey_hdd

SD-Card infos:

root@DietPi:/# cat /var/tmp/dietpi/logs/fs_partition_resize.log
Removed /etc/systemd/system/local-fs.target.wants/dietpi-fs_partition_resize.service.
Disk /dev/mmcblk0: 29.72 GiB, 31914983424 bytes, 62333952 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x0140a7f0

Old situation:

Device         Boot  Start     End Sectors   Size Id Type
/dev/mmcblk0p1 *      8192  270335  262144   128M  c W95 FAT32 (LBA)
/dev/mmcblk0p2      270336 1751375 1481040 723.2M 83 Linux

/dev/mmcblk0p2:
New situation:
Disklabel type: dos
Disk identifier: 0x0140a7f0

Device         Boot  Start      End  Sectors  Size Id Type
/dev/mmcblk0p1 *      8192   270335   262144  128M  c W95 FAT32 (LBA)
/dev/mmcblk0p2      270336 62333951 62063616 29.6G 83 Linux

The partition table has been altered.
resize2fs 1.46.2 (28-Feb-2021)
Filesystem at /dev/mmcblk0p2 is mounted on /; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 4
The filesystem on /dev/mmcblk0p2 is now 7757952 (4k) blocks long.

I tried the following command to get some more informations about the big files.
I’ve exclude the /mnt directory because I#ve mounted there me externel HDD.

root@DietPi:/# du -a / --exclude /mnt | sort -n -r | head -n 30
du: Zugriff auf '/proc/2948/task/2948/fd/3' nicht möglich: Datei oder Verzeichnis nicht gefunden
du: Zugriff auf '/proc/2948/task/2948/fdinfo/3' nicht möglich: Datei oder Verzeichnis nicht gefunden
du: Zugriff auf '/proc/2948/fd/4' nicht möglich: Datei oder Verzeichnis nicht gefunden
du: Zugriff auf '/proc/2948/fdinfo/4' nicht möglich: Datei oder Verzeichnis nicht gefunden
1782439	/
1024932	/usr
700340	/var
667976	/var/www
667848	/var/www/nextcloud
587440	/usr/lib
461836	/var/www/nextcloud/apps
232532	/usr/lib/firmware
212920	/usr/share
175628	/usr/bin
165076	/usr/lib/aarch64-linux-gnu
106108	/var/www/nextcloud/core
86080	/var/www/nextcloud/3rdparty
84172	/usr/lib/modules
84168	/usr/lib/modules/5.10.103-v8+
81708	/usr/lib/modules/5.10.103-v8+/kernel
71284	/usr/share/locale
47784	/var/www/nextcloud/apps/calendar
42956	/usr/lib/modules/5.10.103-v8+/kernel/drivers
42128	/var/www/nextcloud/apps/calendar/js
41964	/usr/share/doc
41484	/usr/sbin
35816	/var/www/nextcloud/apps/settings
35576	/var/www/nextcloud/apps/cookbook
34959	/boot
33876	/var/www/nextcloud/core/doc
33060	/var/www/nextcloud/apps/maps
29208	/var/www/nextcloud/3rdparty/aws
29204	/var/www/nextcloud/3rdparty/aws/aws-sdk-php
29180	/var/www/nextcloud/3rdparty/aws/aws-sdk-php/src




root@DietPi:/# G_TREESIZE /
du: Zugriff auf '/proc/2956/task/2956/fd/3' nicht möglich: Datei oder Verzeichnis nicht gefunden
du: Zugriff auf '/proc/2956/task/2956/fdinfo/3' nicht möglich: Datei oder Verzeichnis nicht gefunden
du: Zugriff auf '/proc/2956/fd/4' nicht möglich: Datei oder Verzeichnis nicht gefunden
du: Zugriff auf '/proc/2956/fdinfo/4' nicht möglich: Datei oder Verzeichnis nicht gefunden
143,2 GiB /
141,5 GiB /mnt
1000,9 MiB /usr
683,9 MiB /var
34,1 MiB /boot
16,5 MiB /run
3,6 MiB /etc
1,5 MiB /root
20,0 KiB /home
16,0 KiB /lost+found
4,0 KiB /srv
4,0 KiB /opt
4,0 KiB /media
0,0 bytes /tmp
0,0 bytes /sys
0,0 bytes /proc
0,0 bytes /dev
2

Try to search inside/mnt explicitly and exclude the exact mount point of your HDD

As well did you tried to reboot?

3

Thx for the fast answer and the hint to only exclude the external HDD.
In the morning my brain don’t realized that I’ve under the /mnt to mountpoints for the backup on the NAS :roll_eyes: . And it seems that my automatic backup process stopped working propable so that the backups stored locally and not directly on the NAS.

I’ve moved the local backup to the HDD and now only 6% of /dev/root is used :smiley:.
But after reboot I get the following error message:

-bash: /boot/dietpi/func/dietpi-globals: No such file or directory
[FAILED] DietPi-Login | Failed to load DietPi-Globals. Skipping DietPi login scripts...

I checked the filesystem and saw that there are some mountpoints are missing.

root@DietPi:~# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/root        30G  1.7G   27G   6% /
devtmpfs        3.9G     0  3.9G   0% /dev
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           1.6G  8.5M  1.6G   1% /run
tmpfs           5.0M     0  5.0M   0% /run/lock

I restored the /etc/fstab out of the last backup. Now the Pi starts “normaly”.

But there is still one problem left. The lighttpd service won’t start any more

Output systemctl status lighttpd:

root@DietPi:/etc/lighttpd# systemctl status lighttpd
● lighttpd.service - Lighttpd Daemon
     Loaded: loaded (/lib/systemd/system/lighttpd.service; disabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Sun 2022-03-27 17:29:27 CEST; 5min ago
    Process: 3130 ExecStartPre=/usr/sbin/lighttpd -tt -f /etc/lighttpd/lighttpd.conf (code=exited, status=255/EXCEPTION)
        CPU: 443ms

Mär 27 17:29:27 DietPi systemd[1]: lighttpd.service: Scheduled restart job, restart counter is at 5.
Mär 27 17:29:27 DietPi systemd[1]: Stopped Lighttpd Daemon.
Mär 27 17:29:27 DietPi systemd[1]: lighttpd.service: Start request repeated too quickly.
Mär 27 17:29:27 DietPi systemd[1]: lighttpd.service: Failed with result 'exit-code'.
Mär 27 17:29:27 DietPi systemd[1]: Failed to start Lighttpd Daemon.

Output journalctl -xe:

Mär 27 17:37:04 DietPi systemd[1]: lighttpd.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ The unit lighttpd.service has entered the 'failed' state with result 'exit-code'.
Mär 27 17:37:04 DietPi systemd[1]: Failed to start Lighttpd Daemon.
░░ Subject: A start job for unit lighttpd.service has failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░
░░ A start job for unit lighttpd.service has finished with a failure.
░░
░░ The job identifier is 3012 and the job result is failed.
4

After I delete the following files the lighttpd was starting again:

  • 50-dietpi-https.conf
  • 98-dietpi-https_redirect.conf
5

This is the HTTPS configuration. Probably certificate files got corrupted as well.

6

You can re-create the HTTPS configurations by re-applying dietpi-letsencrypt. Either these files were corrupted, or a module of Lighttpd used in those.

7

Hi there - I’m actually having the same issue with /dev/root filling up - now I cannot update Dietpi as there isn’t enough room left. Can you describe out how you moved the backups to a different location? My backups are configured to be stored on an external USB drive - which (I think) means it’s not the backups. Yes - I have already rebooted. Any other instructions to figure out what is filling up the space?

root@pi-usenet:/# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/root              29G   28G   87M 100% /
devtmpfs              3.9G     0  3.9G   0% /dev
tmpfs                 3.9G  8.0K  3.9G   1% /dev/shm
tmpfs                 1.6G  8.6M  1.6G   1% /run
tmpfs                 5.0M     0  5.0M   0% /run/lock
tmpfs                 3.9G     0  3.9G   0% /tmp
tmpfs                  50M  612K   50M   2% /var/log
/dev/mmcblk0p1        253M   35M  218M  14% /boot
/dev/sda2             4.6T  3.3G  4.6T   1% /mnt/seagate

Thanks in advance.

8

what does sudo du -a /dev/root | sort -n -r | head -n 10
show?

human readable sizes du -hsx -a /dev/root | sort -rh | head -10
Find the Largest Top 10 Files and Directories On a Linux (tutorialspoint.com)

Once identified, could a symlink to a directory on the /dev/sda2 device directory fix that issue…and/or change where the offending application stores it’s log/error/data files

9 
du: cannot access '/dev/root': No such file or directory

The only thing I can figure is I have /mnt/xxxx, /mnt/yyyy and /mnt/zzzz that points to an external NAS. Would I need to umount those and re-mount them to recover space?

Cheers.

10

Simply your root file system is full. Let’s check it

du -a / --exclude /mnt/seagate | sort -n -r | head -n 20
11

All large files are in /mnt/TV folder - which doesn’t make sense as that is pointing to a separate NAS.

12

You need to exclude all external shares and device. Means add an additional --exclude condition like this

du -a / --exclude /mnt/seagate --exclude /mnt/TV | sort -n -r | head -n 20
13

Results

root@pi-usenet:~# du -a / --exclude /mnt/seagate --exclude /mnt/TV --exclude /mnt/Movies --exclude /mnt/Music | sort -n -r | head -n 20
du: cannot access '/proc/574/task/56578': No such file or directory
du: cannot access '/proc/574/task/56622': No such file or directory
du: cannot access '/proc/56637/task/56637/fd/3': No such file or directory
du: cannot access '/proc/56637/task/56637/fdinfo/3': No such file or directory
du: cannot access '/proc/56637/fd/4': No such file or directory
du: cannot access '/proc/56637/fdinfo/4': No such file or directory
3326027	/
1628472	/usr
1140092	/usr/lib
919708	/mnt
919684	/mnt/dietpi_userdata
473708	/usr/lib/aarch64-linux-gnu
371352	/usr/lib/mono
352944	/mnt/dietpi_userdata/downloads
351208	/mnt/dietpi_userdata/downloads/sabnzbd_nzb_backup
289656	/lib
279236	/opt
260824	/mnt/dietpi_userdata/radarr
245552	/usr/share
244720	/mnt/dietpi_userdata/sonarr
233424	/mnt/dietpi_userdata/radarr/MediaCover
223340	/lib/firmware
157484	/opt/radarr
149116	/mnt/dietpi_userdata/sonarr/MediaCover
121748	/opt/bazarr
117704	/opt/bazarr/libs
14

I recovered ~360 MB using dietpi-cleaner, but I still can’t figure out what is using up so much space.

15

Are you sure /mnt/TV is an external device? Let’s check it

lsblk -o name,fstype,label,size,ro,type,mountpoint,partuuid,uuid
16

Well, I just updated my dietpi_userdata folder to my external USB (5 TB) - and the used space dropped from 29 GB to 1.2 GB. Now I’ll have to figure out which of my apps was hogging the /dietpi_userdata folder.

17

Oops, no, it dropped from 500 BM to 1.2 GB left available. Still something there…

root@pi-usenet:/# lsblk -o name,fstype,label,size,ro,type,mountpoint,partuuid,uuid
NAME        FSTYPE LABEL   SIZE RO TYPE MOUNTPOINT   PARTUUID                             UUID
sda                        4.5T  0 disk
├─sda1      vfat   EFI     200M  0 part              81af391e-21b4-4209-b08f-50596cb73d75 67E3-17ED
└─sda2      ext4           4.5T  0 part /mnt/seagate fe20d7b3-df21-4b8c-8146-725513ad6fa8 fd516b8e-fab1-4569-9f35-2073f69a1cc4
mmcblk0                   29.7G  0 disk
├─mmcblk0p1 vfat   boot    256M  0 part /boot        8f4dbd00-01                          4ADC-240F
└─mmcblk0p2 ext4   rootfs 29.5G  0 part /            8f4dbd00-02                          89cef7ba-d45a-47e7-8d42-aff18afcec80
18

mount

//192.168.0.5/Movies on /mnt/Movies type cifs
//192.168.0.5/Music on /mnt/Music
//192.168.0.5/TV on /mnt/TV type cifs
19

can you share the entire output of following again

df -h
20 
root@pi-usenet:/# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/root              29G   27G  1.3G  96% /
devtmpfs              3.9G     0  3.9G   0% /dev
tmpfs                 3.9G  8.0K  3.9G   1% /dev/shm
tmpfs                 1.6G   36M  1.6G   3% /run
tmpfs                 5.0M     0  5.0M   0% /run/lock
tmpfs                 3.9G  4.0K  3.9G   1% /tmp
tmpfs                  50M  2.0M   49M   4% /var/log
/dev/mmcblk0p1        253M   35M  218M  14% /boot
/dev/sda2             4.6T  4.5G  4.6T   1% /mnt/seagate
//192.168.0.5/Movies   35T   13T   23T  35% /mnt/Movies
//192.168.0.5/Music    35T   13T   23T  35% /mnt/Music
//192.168.0.5/TV       35T   13T   23T  35% /mnt/TV

/dev/mmcblk0p1 is my 32 GB SD card, running /boot and /root.
/dev/sda2 is my directly attached 5TB USB drive.
//192.168.0.5 is my NAS.