No space left on sd-card

przemko · December 10, 2020, 6:36pm

Hello, today I have strange situation. I cannot login to my Emby and cannot do anything in my Nextcloud. I see also redis-server stuck on restart services. I try issue from github about rootfs https://github.com/MichaIng/DietPi/issues/3511 but don’t help.

root@DietPi:/# systemctl status redis-server.service 
● redis-server.service - Advanced key-value store
   Loaded: loaded (/lib/systemd/system/redis-server.service; disabled; vendor preset: enabled)
   Active: active (running) since Thu 2020-12-10 19:02:26 CET; 29min ago
     Docs: http://redis.io/documentation,
           man:redis-server(1)
  Process: 567 ExecStart=/usr/bin/redis-server /etc/redis/redis.conf (code=exited, status=0/SUCCESS)
 Main PID: 568 (redis-server)
    Tasks: 4 (limit: 4442)
   CGroup: /system.slice/redis-server.service
           └─568 /usr/bin/redis-server 127.0.0.1:6379

Dec 10 19:31:15 DietPi redis[3474]: Write error saving DB on disk: No space left on device
Dec 10 19:31:15 DietPi redis[568]: Background saving error

How it’s possible I don’t have space?

root@DietPi:/# df -a
Filesystem      1K-blocks       Used  Available Use% Mounted on
/dev/root        30393948   29134376          0 100% /
devtmpfs           947680          0     947680   0% /dev
sysfs                   0          0          0    - /sys
proc                    0          0          0    - /proc
securityfs              0          0          0    - /sys/kernel/security
tmpfs              981472          0     981472   0% /dev/shm
devpts                  0          0          0    - /dev/pts
tmpfs              981472       8544     972928   1% /run
tmpfs                5120          0       5120   0% /run/lock
tmpfs              981472          0     981472   0% /sys/fs/cgroup
cgroup2                 0          0          0    - /sys/fs/cgroup/unified
cgroup                  0          0          0    - /sys/fs/cgroup/systemd
none                    0          0          0    - /sys/fs/bpf
cgroup                  0          0          0    - /sys/fs/cgroup/freezer
cgroup                  0          0          0    - /sys/fs/cgroup/cpu,cpuacct
cgroup                  0          0          0    - /sys/fs/cgroup/net_cls,net_prio
cgroup                  0          0          0    - /sys/fs/cgroup/cpuset
cgroup                  0          0          0    - /sys/fs/cgroup/blkio
cgroup                  0          0          0    - /sys/fs/cgroup/perf_event
cgroup                  0          0          0    - /sys/fs/cgroup/devices
cgroup                  0          0          0    - /sys/fs/cgroup/pids
systemd-1               0          0          0    - /proc/sys/fs/binfmt_misc
systemd-1               -          -          -    - /mnt/Toshiba
sunrpc                  0          0          0    - /run/rpc_pipefs
mqueue                  0          0          0    - /dev/mqueue
debugfs                 0          0          0    - /sys/kernel/debug
configfs                0          0          0    - /sys/kernel/config
tmpfs             1048576          4    1048572   1% /tmp
tmpfs               51200          4      51196   1% /var/log
/dev/mmcblk0p1     258095      56606     201490  22% /boot
/dev/sda       2883221128 1705043156 1031648260  63% /mnt/Toshiba
nfsd                    0          0          0    - /proc/fs/nfsd

Regards,
Przemek

Joulinar · December 10, 2020, 7:36pm

You need to find out which files are blocking your RootFS as it’s completely full.

du -a / | sort -n -r | head -n 20

przemko · December 10, 2020, 7:44pm

Hi, I have run dietpi-cleanrs and also stuck on stopping dietpi-services redis-server. In the same time I run Your command:

root@DietPi:/# du -a / | sort -n -r | head -n 20
du: cannot access '/proc/10046/task/10046/fd/3': No such file or directory
du: cannot access '/proc/10046/task/10046/fdinfo/3': No such file or directory
du: cannot access '/proc/10046/fd/4': No such file or directory
du: cannot access '/proc/10046/fdinfo/4': No such file or directory
1732575043	/
1705629276	/mnt
1703333204	/mnt/Toshiba
1700857544	/mnt/Toshiba/dietpi_userdata
916151384	/mnt/Toshiba/dietpi_userdata/Video
602783060	/mnt/Toshiba/dietpi_userdata/Zapas
475674520	/mnt/Toshiba/dietpi_userdata/Video/Filmy
440476860	/mnt/Toshiba/dietpi_userdata/Video/Seriale
346885684	/mnt/Toshiba/dietpi_userdata/Zapas/Przemek-dell
147057092	/mnt/Toshiba/dietpi_userdata/Zapas/Elwirka-laptop
92962012	/mnt/Toshiba/dietpi_userdata/Pictures
92962008	/mnt/Toshiba/dietpi_userdata/Pictures/Zdjęcia
85089372	/mnt/Toshiba/dietpi_userdata/Zapas/Lusia-Laptop
47298688	/mnt/Toshiba/dietpi_userdata/nextcloud_data
39541012	/mnt/Toshiba/dietpi_userdata/nextcloud_data/admin
39335764	/mnt/Toshiba/dietpi_userdata/Music
36234420	/mnt/Toshiba/dietpi_userdata/nextcloud_data/admin/files
29280388	/mnt/Toshiba/dietpi_userdata/Pictures/Zdjęcia/Obrazy
24274344	/var
22384544	/mnt/Toshiba/dietpi_userdata/Zapas/Ciocia

Regards.

Joulinar · December 10, 2020, 7:46pm

can you detach your HDD for a moment?

przemko · December 10, 2020, 8:12pm

I detach HDD and reboot. Now my Dietpi won’t start.

Regards.

Joulinar · December 10, 2020, 8:14pm

the idea was more to stop all services, just remove HDD and check again for large files

przemko · December 10, 2020, 8:25pm

OK, now I start Rpi with HDD and then:

root@DietPi:/# du -a / | sort -n -r | head -n 20
du: cannot access '/proc/712/task/1860': No such file or directory
du: cannot access '/proc/1818/task/1818/fd/3': No such file or directory
du: cannot access '/proc/1818/task/1818/fdinfo/3': No such file or directory
du: cannot access '/proc/1818/fd/4': No such file or directory
du: cannot access '/proc/1818/fdinfo/4': No such file or directory
1732590715	/
1705644860	/mnt
1703348788	/mnt/Toshiba
1700873128	/mnt/Toshiba/dietpi_userdata
916151384	/mnt/Toshiba/dietpi_userdata/Video
602783060	/mnt/Toshiba/dietpi_userdata/Zapas
475674520	/mnt/Toshiba/dietpi_userdata/Video/Filmy
440476860	/mnt/Toshiba/dietpi_userdata/Video/Seriale
346885684	/mnt/Toshiba/dietpi_userdata/Zapas/Przemek-dell
147057092	/mnt/Toshiba/dietpi_userdata/Zapas/Elwirka-laptop
92962012	/mnt/Toshiba/dietpi_userdata/Pictures
92962008	/mnt/Toshiba/dietpi_userdata/Pictures/Zdjęcia
85089372	/mnt/Toshiba/dietpi_userdata/Zapas/Lusia-Laptop
47301960	/mnt/Toshiba/dietpi_userdata/nextcloud_data
39541024	/mnt/Toshiba/dietpi_userdata/nextcloud_data/admin
39335764	/mnt/Toshiba/dietpi_userdata/Music
36234428	/mnt/Toshiba/dietpi_userdata/nextcloud_data/admin/files
29280388	/mnt/Toshiba/dietpi_userdata/Pictures/Zdjęcia/Obrazy
24274336	/var
22384544	/mnt/Toshiba/dietpi_userdata/Zapas/Ciocia

after that I detach my HDD:

root@DietPi:/# du -a / | sort -n -r | head -n 20
du: cannot read directory '/mnt/Toshiba': Input/output error
du: cannot access '/proc/1909/task/1909/fd/3': No such file or directory
du: cannot access '/proc/1909/task/1909/fdinfo/3': No such file or directory
du: cannot access '/proc/1909/fd/4': No such file or directory
du: cannot access '/proc/1909/fdinfo/4': No such file or directory
29206815	/
24274348	/var
22229820	/var/cache
22224200	/var/cache/lighttpd
22215756	/var/cache/lighttpd/uploads
2296076	/mnt
2296044	/mnt/dietpi-backup
2296036	/mnt/dietpi-backup/data
1401568	/home
1401564	/home/dietpi
1195136	/var/lib
1164416	/var/lib/emby
1074124	/home/dietpi/emby
788912	/mnt/dietpi-backup/data/var
744784	/mnt/dietpi-backup/data/var/www
744708	/mnt/dietpi-backup/data/var/www/nextcloud
713136	/var/www
712480	/var/www/nextcloud
644452	/var/lib/emby/metadata
644440	/var/lib/emby/metadata/library

Regards.

Joulinar · December 10, 2020, 8:32pm

majority of space is used on /var/cache/lighttpd/uploads

22215756	/var/cache/lighttpd/uploads

Can you have a look which files are in

przemko · December 10, 2020, 8:35pm

There’re lot of files like this:

lighttpd-upload-007uSV	lighttpd-upload-APUL95	lighttpd-upload-KcGCqg	lighttpd-upload-V1ZUah	lighttpd-upload-fVeH6I	lighttpd-upload-pjlzM6
lighttpd-upload-00H0Ru	lighttpd-upload-APiDoz	lighttpd-upload-KcHQpX	lighttpd-upload-V1cBWd	lighttpd-upload-fVpXHq	lighttpd-upload-pjwPOL
lighttpd-upload-00IOZZ	lighttpd-upload-AQV9Vl	lighttpd-upload-KcOdSD	lighttpd-upload-V1m6FC	lighttpd-upload-fVu5Kd	lighttpd-upload-pkDLWx
lighttpd-upload-00Lpnx	lighttpd-upload-AQWsQO	lighttpd-upload-KcXQoL	lighttpd-upload-V1tSjc	lighttpd-upload-fVv3GE	lighttpd-upload-pkHyoi
lighttpd-upload-00P9SY	lighttpd-upload-AQlr8k	lighttpd-upload-KcqR6q	lighttpd-upload-V1tWNq	lighttpd-upload-fWeeIv	lighttpd-upload-pkf8Ze
lighttpd-upload-00SxeD	lighttpd-upload-AQptDx	lighttpd-upload-Kd1pQM	lighttpd-upload-V1yeZq	lighttpd-upload-fWpVla	lighttpd-upload-pklQb7
lighttpd-upload-00W3Vf	lighttpd-upload-ARPDMT	lighttpd-upload-KdJMhb	lighttpd-upload-V22O5q	lighttpd-upload-fWppVw	lighttpd-upload-pldqkR
lighttpd-upload-00dhS3	lighttpd-upload-ARjvTU	lighttpd-upload-KdjYHP

To much to post it.

Joulinar · December 10, 2020, 8:37pm

can you check time stamp on these files? maybe some leftover from the past

przemko · December 10, 2020, 8:42pm

Do You mean like this?

-rw------- 1 www-data www-data 1055736 Dec 10 13:42 lighttpd-upload-zxI5lq
-rw------- 1 www-data www-data 1051638 Dec 10 14:23 lighttpd-upload-zxVAjA
-rw------- 1 www-data www-data 1055736 Dec 10 13:46 lighttpd-upload-zxZ3LE
-rw------- 1 www-data www-data 1055736 Dec 10 14:05 lighttpd-upload-zxzGch
-rw------- 1 www-data www-data 1055736 Dec 10 14:14 lighttpd-upload-zy1FAX
-rw------- 1 www-data www-data 1055736 Dec 10 13:51 lighttpd-upload-zyU4VL
-rw------- 1 www-data www-data 1055736 Dec 10 13:46 lighttpd-upload-zylw4W
-rw------- 1 www-data www-data 1055736 Dec 10 13:47 lighttpd-upload-zyvupF
-rw------- 1 www-data www-data 1055736 Dec 10 14:08 lighttpd-upload-zzOfK7
-rw------- 1 www-data www-data 1055736 Dec 10 14:03 lighttpd-upload-zza1Il
-rw------- 1 www-data www-data 1055736 Dec 10 13:53 lighttpd-upload-zzlB8Y
-rw------- 1 www-data www-data 1055736 Dec 10 13:49 lighttpd-upload-zzmyrQ

I think it’s my foult because I play with my Nextcloud webdav protocol. I use it rclone webgui and try to munt it on my archlinux laptop in work.

Do You think move /rootfs to my HDD /mnt/Toshiba where I have /dietpi_userdata will help?

Joulinar · December 10, 2020, 9:24pm

moving RootFS will format the target device. So not a good idea https://dietpi.com/forum/t/services-failing-after-swapping-backup-drives/3869/6

you could try to remove these cache files to gain space on your SD card.

przemko · December 11, 2020, 6:10am

Hi, I remove those files and for now everything is OK but I’m afraid that it will be full after some time again. I must check in work that webdav, rclone and rsync connections I try yesterday.

Regards.

Joulinar · December 11, 2020, 6:20pm

usually cache files should be deleted once upload was done. Maybe you need to watch it closely for the next time

przemko · December 11, 2020, 6:58pm

Hi Joulinar today when I look at sd-card space it’s 96% used from 29GB. That /var/cache/lighttpd/uploads/ folder is also full. I delete every file again
I have notice when I reinstall nextcloud that there’s old lighttpd installation present maybe this is problem?

Regards,
Przemek

Joulinar · December 11, 2020, 9:36pm

what do you mean by an old lighttpd instance? lighttpd can be installed only once.

Maybe it would make sense to check on a NextCloud specialised board regarding the cache issue and why files did not get deleted automatically.

przemko · December 11, 2020, 10:11pm

I mean when I reinstall nextcloud at the end I get information that there’s old lighttpd configuration on system and will be saved as lighttpd…old or something like that.

OK, I will post on some nextcloud forum.
Thanks for Your time and help.
Regards.
Przemek