No space left on sd-card

Hello, today I have strange situation. I cannot login to my Emby and cannot do anything in my Nextcloud. I see also redis-server stuck on restart services. I try issue from github about rootfs https://github.com/MichaIng/DietPi/issues/3511 but donā€™t help.

root@DietPi:/# systemctl status redis-server.service 
ā— redis-server.service - Advanced key-value store
   Loaded: loaded (/lib/systemd/system/redis-server.service; disabled; vendor preset: enabled)
   Active: active (running) since Thu 2020-12-10 19:02:26 CET; 29min ago
     Docs: http://redis.io/documentation,
           man:redis-server(1)
  Process: 567 ExecStart=/usr/bin/redis-server /etc/redis/redis.conf (code=exited, status=0/SUCCESS)
 Main PID: 568 (redis-server)
    Tasks: 4 (limit: 4442)
   CGroup: /system.slice/redis-server.service
           ā””ā”€568 /usr/bin/redis-server 127.0.0.1:6379

Dec 10 19:31:15 DietPi redis[3474]: Write error saving DB on disk: No space left on device
Dec 10 19:31:15 DietPi redis[568]: Background saving error

How itā€™s possible I donā€™t have space?

root@DietPi:/# df -a
Filesystem      1K-blocks       Used  Available Use% Mounted on
/dev/root        30393948   29134376          0 100% /
devtmpfs           947680          0     947680   0% /dev
sysfs                   0          0          0    - /sys
proc                    0          0          0    - /proc
securityfs              0          0          0    - /sys/kernel/security
tmpfs              981472          0     981472   0% /dev/shm
devpts                  0          0          0    - /dev/pts
tmpfs              981472       8544     972928   1% /run
tmpfs                5120          0       5120   0% /run/lock
tmpfs              981472          0     981472   0% /sys/fs/cgroup
cgroup2                 0          0          0    - /sys/fs/cgroup/unified
cgroup                  0          0          0    - /sys/fs/cgroup/systemd
none                    0          0          0    - /sys/fs/bpf
cgroup                  0          0          0    - /sys/fs/cgroup/freezer
cgroup                  0          0          0    - /sys/fs/cgroup/cpu,cpuacct
cgroup                  0          0          0    - /sys/fs/cgroup/net_cls,net_prio
cgroup                  0          0          0    - /sys/fs/cgroup/cpuset
cgroup                  0          0          0    - /sys/fs/cgroup/blkio
cgroup                  0          0          0    - /sys/fs/cgroup/perf_event
cgroup                  0          0          0    - /sys/fs/cgroup/devices
cgroup                  0          0          0    - /sys/fs/cgroup/pids
systemd-1               0          0          0    - /proc/sys/fs/binfmt_misc
systemd-1               -          -          -    - /mnt/Toshiba
sunrpc                  0          0          0    - /run/rpc_pipefs
mqueue                  0          0          0    - /dev/mqueue
debugfs                 0          0          0    - /sys/kernel/debug
configfs                0          0          0    - /sys/kernel/config
tmpfs             1048576          4    1048572   1% /tmp
tmpfs               51200          4      51196   1% /var/log
/dev/mmcblk0p1     258095      56606     201490  22% /boot
/dev/sda       2883221128 1705043156 1031648260  63% /mnt/Toshiba
nfsd                    0          0          0    - /proc/fs/nfsd

Regards,
Przemek

You need to find out which files are blocking your RootFS as itā€™s completely full.

du -a / | sort -n -r | head -n 20

Hi, I have run dietpi-cleanrs and also stuck on stopping dietpi-services redis-server. In the same time I run Your command:

root@DietPi:/# du -a / | sort -n -r | head -n 20
du: cannot access '/proc/10046/task/10046/fd/3': No such file or directory
du: cannot access '/proc/10046/task/10046/fdinfo/3': No such file or directory
du: cannot access '/proc/10046/fd/4': No such file or directory
du: cannot access '/proc/10046/fdinfo/4': No such file or directory
1732575043	/
1705629276	/mnt
1703333204	/mnt/Toshiba
1700857544	/mnt/Toshiba/dietpi_userdata
916151384	/mnt/Toshiba/dietpi_userdata/Video
602783060	/mnt/Toshiba/dietpi_userdata/Zapas
475674520	/mnt/Toshiba/dietpi_userdata/Video/Filmy
440476860	/mnt/Toshiba/dietpi_userdata/Video/Seriale
346885684	/mnt/Toshiba/dietpi_userdata/Zapas/Przemek-dell
147057092	/mnt/Toshiba/dietpi_userdata/Zapas/Elwirka-laptop
92962012	/mnt/Toshiba/dietpi_userdata/Pictures
92962008	/mnt/Toshiba/dietpi_userdata/Pictures/Zdjęcia
85089372	/mnt/Toshiba/dietpi_userdata/Zapas/Lusia-Laptop
47298688	/mnt/Toshiba/dietpi_userdata/nextcloud_data
39541012	/mnt/Toshiba/dietpi_userdata/nextcloud_data/admin
39335764	/mnt/Toshiba/dietpi_userdata/Music
36234420	/mnt/Toshiba/dietpi_userdata/nextcloud_data/admin/files
29280388	/mnt/Toshiba/dietpi_userdata/Pictures/Zdjęcia/Obrazy
24274344	/var
22384544	/mnt/Toshiba/dietpi_userdata/Zapas/Ciocia

Regards.

can you detach your HDD for a moment?

I detach HDD and reboot. Now my Dietpi wonā€™t start. :confused:

Regards.

the idea was more to stop all services, just remove HDD and check again for large files

OK, now I start Rpi with HDD and then:

root@DietPi:/# du -a / | sort -n -r | head -n 20
du: cannot access '/proc/712/task/1860': No such file or directory
du: cannot access '/proc/1818/task/1818/fd/3': No such file or directory
du: cannot access '/proc/1818/task/1818/fdinfo/3': No such file or directory
du: cannot access '/proc/1818/fd/4': No such file or directory
du: cannot access '/proc/1818/fdinfo/4': No such file or directory
1732590715	/
1705644860	/mnt
1703348788	/mnt/Toshiba
1700873128	/mnt/Toshiba/dietpi_userdata
916151384	/mnt/Toshiba/dietpi_userdata/Video
602783060	/mnt/Toshiba/dietpi_userdata/Zapas
475674520	/mnt/Toshiba/dietpi_userdata/Video/Filmy
440476860	/mnt/Toshiba/dietpi_userdata/Video/Seriale
346885684	/mnt/Toshiba/dietpi_userdata/Zapas/Przemek-dell
147057092	/mnt/Toshiba/dietpi_userdata/Zapas/Elwirka-laptop
92962012	/mnt/Toshiba/dietpi_userdata/Pictures
92962008	/mnt/Toshiba/dietpi_userdata/Pictures/Zdjęcia
85089372	/mnt/Toshiba/dietpi_userdata/Zapas/Lusia-Laptop
47301960	/mnt/Toshiba/dietpi_userdata/nextcloud_data
39541024	/mnt/Toshiba/dietpi_userdata/nextcloud_data/admin
39335764	/mnt/Toshiba/dietpi_userdata/Music
36234428	/mnt/Toshiba/dietpi_userdata/nextcloud_data/admin/files
29280388	/mnt/Toshiba/dietpi_userdata/Pictures/Zdjęcia/Obrazy
24274336	/var
22384544	/mnt/Toshiba/dietpi_userdata/Zapas/Ciocia

after that I detach my HDD:

root@DietPi:/# du -a / | sort -n -r | head -n 20
du: cannot read directory '/mnt/Toshiba': Input/output error
du: cannot access '/proc/1909/task/1909/fd/3': No such file or directory
du: cannot access '/proc/1909/task/1909/fdinfo/3': No such file or directory
du: cannot access '/proc/1909/fd/4': No such file or directory
du: cannot access '/proc/1909/fdinfo/4': No such file or directory
29206815	/
24274348	/var
22229820	/var/cache
22224200	/var/cache/lighttpd
22215756	/var/cache/lighttpd/uploads
2296076	/mnt
2296044	/mnt/dietpi-backup
2296036	/mnt/dietpi-backup/data
1401568	/home
1401564	/home/dietpi
1195136	/var/lib
1164416	/var/lib/emby
1074124	/home/dietpi/emby
788912	/mnt/dietpi-backup/data/var
744784	/mnt/dietpi-backup/data/var/www
744708	/mnt/dietpi-backup/data/var/www/nextcloud
713136	/var/www
712480	/var/www/nextcloud
644452	/var/lib/emby/metadata
644440	/var/lib/emby/metadata/library

Regards.

majority of space is used on /var/cache/lighttpd/uploads

22215756	/var/cache/lighttpd/uploads

Can you have a look which files are in

Thereā€™re lot of files like this:

lighttpd-upload-007uSV	lighttpd-upload-APUL95	lighttpd-upload-KcGCqg	lighttpd-upload-V1ZUah	lighttpd-upload-fVeH6I	lighttpd-upload-pjlzM6
lighttpd-upload-00H0Ru	lighttpd-upload-APiDoz	lighttpd-upload-KcHQpX	lighttpd-upload-V1cBWd	lighttpd-upload-fVpXHq	lighttpd-upload-pjwPOL
lighttpd-upload-00IOZZ	lighttpd-upload-AQV9Vl	lighttpd-upload-KcOdSD	lighttpd-upload-V1m6FC	lighttpd-upload-fVu5Kd	lighttpd-upload-pkDLWx
lighttpd-upload-00Lpnx	lighttpd-upload-AQWsQO	lighttpd-upload-KcXQoL	lighttpd-upload-V1tSjc	lighttpd-upload-fVv3GE	lighttpd-upload-pkHyoi
lighttpd-upload-00P9SY	lighttpd-upload-AQlr8k	lighttpd-upload-KcqR6q	lighttpd-upload-V1tWNq	lighttpd-upload-fWeeIv	lighttpd-upload-pkf8Ze
lighttpd-upload-00SxeD	lighttpd-upload-AQptDx	lighttpd-upload-Kd1pQM	lighttpd-upload-V1yeZq	lighttpd-upload-fWpVla	lighttpd-upload-pklQb7
lighttpd-upload-00W3Vf	lighttpd-upload-ARPDMT	lighttpd-upload-KdJMhb	lighttpd-upload-V22O5q	lighttpd-upload-fWppVw	lighttpd-upload-pldqkR
lighttpd-upload-00dhS3	lighttpd-upload-ARjvTU	lighttpd-upload-KdjYHP

To much to post it.

can you check time stamp on these files? maybe some leftover from the past

Do You mean like this?

-rw------- 1 www-data www-data 1055736 Dec 10 13:42 lighttpd-upload-zxI5lq
-rw------- 1 www-data www-data 1051638 Dec 10 14:23 lighttpd-upload-zxVAjA
-rw------- 1 www-data www-data 1055736 Dec 10 13:46 lighttpd-upload-zxZ3LE
-rw------- 1 www-data www-data 1055736 Dec 10 14:05 lighttpd-upload-zxzGch
-rw------- 1 www-data www-data 1055736 Dec 10 14:14 lighttpd-upload-zy1FAX
-rw------- 1 www-data www-data 1055736 Dec 10 13:51 lighttpd-upload-zyU4VL
-rw------- 1 www-data www-data 1055736 Dec 10 13:46 lighttpd-upload-zylw4W
-rw------- 1 www-data www-data 1055736 Dec 10 13:47 lighttpd-upload-zyvupF
-rw------- 1 www-data www-data 1055736 Dec 10 14:08 lighttpd-upload-zzOfK7
-rw------- 1 www-data www-data 1055736 Dec 10 14:03 lighttpd-upload-zza1Il
-rw------- 1 www-data www-data 1055736 Dec 10 13:53 lighttpd-upload-zzlB8Y
-rw------- 1 www-data www-data 1055736 Dec 10 13:49 lighttpd-upload-zzmyrQ

I think itā€™s my foult because I play with my Nextcloud webdav protocol. I use it rclone webgui and try to munt it on my archlinux laptop in work.

Do You think move /rootfs to my HDD /mnt/Toshiba where I have /dietpi_userdata will help?

moving RootFS will format the target device. So not a good idea :wink: https://dietpi.com/forum/t/services-failing-after-swapping-backup-drives/3869/6

you could try to remove these cache files to gain space on your SD card.

Hi, I remove those files and for now everything is OK but Iā€™m afraid that it will be full after some time again. I must check in work that webdav, rclone and rsync connections I try yesterday.

Regards.

usually cache files should be deleted once upload was done. Maybe you need to watch it closely for the next time

Hi Joulinar today when I look at sd-card space itā€™s 96% used from 29GB. That /var/cache/lighttpd/uploads/ folder is also full. I delete every file again :confused:
I have notice when I reinstall nextcloud that thereā€™s old lighttpd installation present maybe this is problem?

Regards,
Przemek

what do you mean by an old lighttpd instance? lighttpd can be installed only once.

Maybe it would make sense to check on a NextCloud specialised board regarding the cache issue and why files did not get deleted automatically.

I mean when I reinstall nextcloud at the end I get information that thereā€™s old lighttpd configuration on system and will be saved as lighttpdā€¦old or something like that.

OK, I will post on some nextcloud forum.
Thanks for Your time and help.
Regards.
Przemek