No space left on device

after restoring /etc/fstab and rebooting i am not able to connect via ssh (connection refused). How long does a fsck last? Could it be the reason for a refused connection?

do you see green led still flashing/active?

I pulled powercable and now its back again :slight_smile:
So I think it s best to shutdown, make a new image and use a new card?

I really appreciate your help.

Doing backups is always a good idea. You might be ok to continue running your system, if there are not that much changes, not that much content on your system and you are ok to restore an older image backup without issues.

A full fsck can indeed take a while, especially when there are many errors, which leads to many fix/check iterations. Checking RPi LEDs for activity, the green one for filesystem activity, is indeed a proper hint. Pulling the power plug while fsck is running, is not a good idea when aiming to repair that filesystem. However, yes probably flashing a new image is/was a good idea :wink:.

In case it helps other users who admin a system that runs out of space, I found that docker, as used by Teslamate, seems to leak about three images on each restart. Executing

docker system prune -a

once in a while seems to help. I do it in my shell script that updates Teslamate as the process can be very slow.

Just throwing this out there to help other users and feed search engines to help other users. I’d lost many GB on a filesystem that only had 30, so my Teslamate instance was broken until I could return and learn this tidbit. The prune seems to work reliably.

thx for your message, but it has nothing to do with original issue. :wink:

/tmp keeps getting full after 3-4 days. This causes write errors in e.g. dietpi-update. A reboot solves and cleans /tmp.
Usingls -l does not show any big file like 1.9G on tmp. How can I identify which causes the full /tmp?
What do you think of A weekly reboot via Cron.?

I rebooted today, it takes some time to investigate again…

to find the 20 biggest files

du -a /tmp | sort -n -r | head -n 20

nearly 2 days after reboot df -h shows /tmp is neary full

root@DietPi4 [19:45:34]~ # df -h
Dateisystem    Größe Benutzt Verf. Verw% Eingehängt auf
/dev/root       235G    113G  113G   51% /
devtmpfs        1,9G       0  1,9G    0% /dev
tmpfs           1,9G    1,7M  1,9G    1% /dev/shm
tmpfs           771M     46M  726M    6% /run
tmpfs           5,0M    4,0K  5,0M    1% /run/lock
tmpfs           1,9G    1,9G   52M   98% /tmp
tmpfs            50M    132K   50M    1% /var/log
/dev/mmcblk0p1  253M     52M  201M   21% /boot
tmpfs           386M    8,0K  386M    1% /run/user/999
/dev/sda1       113G     61G   52G   55% /mnt/128
/dev/sdb1       229G    160G   69G   70% /mnt/256
tmpfs           386M    8,0K  386M    1% /run/user/0

When i try to find files in /tmp with du -a /tmp | sort -n -r | head -n 20 all files are 0bytes?

0       /tmp/.XIM-unix
0       /tmp/.X11-unix
0       /tmp/.Test-unix
0       /tmp/systemd-private-77adb340e35749048d0528ae29553b5d-systemd-logind.service-ez7cyi/tmp
0       /tmp/systemd-private-77adb340e35749048d0528ae29553b5d-systemd-logind.service-ez7cyi
0       /tmp/systemd-private-77adb340e35749048d0528ae29553b5d-redis-server.service-1tvB7f/tmp
0       /tmp/systemd-private-77adb340e35749048d0528ae29553b5d-redis-server.service-1tvB7f
0       /tmp/systemd-private-77adb340e35749048d0528ae29553b5d-mjpg-streamer.service-bkS3Ff/tmp
0       /tmp/systemd-private-77adb340e35749048d0528ae29553b5d-mjpg-streamer.service-bkS3Ff
0       /tmp/systemd-private-77adb340e35749048d0528ae29553b5d-coturn.service-MyDD0i/tmp
0       /tmp/systemd-private-77adb340e35749048d0528ae29553b5d-coturn.service-MyDD0i
0       /tmp/.ICE-unix
0       /tmp/.font-unix
0       /tmp

Can you try it again? You are running it as root, right?

sync; du -csh /tmp/*

I am running it at root, correct:
Here´s the result, i dont see any files bigger than 0 bytes.

root@DietPi4 [14:23:19]~ # sync; du -csh /tmp/*
0       /tmp/0jaTwb3Gka
0       /tmp/0q9h2MTQSm
0       /tmp/1t8gHWsHYP
0       /tmp/1v8WPjKXRP
0       /tmp/7Y8hkDQycJ
0       /tmp/7zu5c0VCbq
0       /tmp/8Jrcu8Ga5R
0       /tmp/8uDOJYtE0C
0       /tmp/9elp1V357x
0       /tmp/ap11QoCzIC
0       /tmp/aV5VyVjUZ1
0       /tmp/aZ9vYWyFlm
0       /tmp/BCln6pbQdX
0       /tmp/bpjr6FEwZe
0       /tmp/CsasWQtx5k
0       /tmp/E1HJlp3h4x
0       /tmp/f7l7o3GNru
0       /tmp/gfQKSohWZg
0       /tmp/gPcv6gsai5
0       /tmp/iV6oxwH3iJ
0       /tmp/l2Dj7NdfXa
0       /tmp/L7Y1cYfZ8D
0       /tmp/lBTLMC4z7d
0       /tmp/lfyelNblKr
0       /tmp/lIbF1HBg9w
0       /tmp/lxYgS20dMj
0       /tmp/LygurYuAEy
0       /tmp/MkEEx4pAUd
0       /tmp/nEGkiKUKen
0       /tmp/NFjWh6ImQD
0       /tmp/nJ9BZdaQqk
0       /tmp/NoHZMmRVCI
0       /tmp/oSMIz4dKeb
0       /tmp/P3I0SZyn5e
0       /tmp/PQcyE39WGE
0       /tmp/qvN0XS0G2X
0       /tmp/R4VmZb7SmJ
0       /tmp/rc5BGovWW2
0       /tmp/rig9Jfor5M
0       /tmp/rIub6gWKnX
0       /tmp/RU9Bn3wKOQ
0       /tmp/S8nNLDnT5j
0       /tmp/sLG90TSmcn
0       /tmp/slldJuT3vj
0       /tmp/systemd-private-77adb340e35749048d0528ae29553b5d-coturn.service-ez281i
0       /tmp/systemd-private-77adb340e35749048d0528ae29553b5d-mjpg-streamer.service-TKG2Xi
0       /tmp/systemd-private-77adb340e35749048d0528ae29553b5d-redis-server.service-SpsxNi
0       /tmp/systemd-private-77adb340e35749048d0528ae29553b5d-systemd-logind.service-ez7cyi
0       /tmp/tI4ZqrIa2l
0       /tmp/TSJsEq7o6g
0       /tmp/UMHJJ6SZhO
0       /tmp/UNbwpnF6Ir
0       /tmp/uSWq9aEIXx
0       /tmp/Utfa5iVdVa
0       /tmp/vl4T2iCxLR
0       /tmp/vLctoKU4hq
0       /tmp/WWof8EkRKb
0       /tmp/WyaNVVN1BX
0       /tmp/X5LG2ktjL1
0       /tmp/XwBFseH8wO
0       /tmp/y0693u4Q57
0       insgesamt

df -h:

 df -h
Dateisystem    Größe Benutzt Verf. Verw% Eingehängt auf
/dev/root       235G    113G  113G   51% /
devtmpfs        1,9G       0  1,9G    0% /dev
tmpfs           1,9G    1,9M  1,9G    1% /dev/shm
tmpfs           771M     78M  693M   11% /run
tmpfs           5,0M    4,0K  5,0M    1% /run/lock
tmpfs           1,9G    1,9G     0  100% /tmp
tmpfs            50M     64K   50M    1% /var/log
/dev/mmcblk0p1  253M     52M  201M   21% /boot
tmpfs           386M    8,0K  386M    1% /run/user/999
/dev/sda1       113G     62G   51G   55% /mnt/128
/dev/sdb1       229G    160G   69G   70% /mnt/256
tmpfs           386M    8,0K  386M    1% /run/user/0

strange, can you share

ls -la /tmp/

ls -la results in:

root@DietPi4 [14:24:06]~ # ls -la /tmp/
insgesamt 4
drwxrwxrwt 11 root             root    1360 23. Feb 14:28 .
drwxr-xr-x 20 root             root    4096 16. Jan 11:25 ..
-rw-r--r--  1 squeezeboxserver nogroup    0 23. Feb 09:04 0jaTwb3Gka
-rw-r--r--  1 squeezeboxserver nogroup    0 23. Feb 02:04 0q9h2MTQSm
-rw-r--r--  1 squeezeboxserver nogroup    0 23. Feb 14:04 1t8gHWsHYP
-rw-r--r--  1 squeezeboxserver nogroup    0 23. Feb 07:04 1v8WPjKXRP
-rw-r--r--  1 squeezeboxserver nogroup    0 22. Feb 23:04 7Y8hkDQycJ
-rw-r--r--  1 squeezeboxserver nogroup    0 23. Feb 00:04 7zu5c0VCbq
-rw-r--r--  1 squeezeboxserver nogroup    0 23. Feb 10:04 8Jrcu8Ga5R
-rw-r--r--  1 squeezeboxserver nogroup    0 23. Feb 04:04 8uDOJYtE0C
-rw-r--r--  1 squeezeboxserver nogroup    0 23. Feb 11:04 9elp1V357x
-rw-r--r--  1 squeezeboxserver nogroup    0 23. Feb 11:47 ap11QoCzIC
-rw-r--r--  1 squeezeboxserver nogroup    0 23. Feb 13:04 aV5VyVjUZ1
-rw-r--r--  1 squeezeboxserver nogroup    0 23. Feb 14:04 aZ9vYWyFlm
-rw-r--r--  1 squeezeboxserver nogroup    0 23. Feb 04:04 BCln6pbQdX
-rw-r--r--  1 squeezeboxserver nogroup    0 22. Feb 23:04 bpjr6FEwZe
-rw-r--r--  1 squeezeboxserver nogroup    0 23. Feb 12:04 CsasWQtx5k
-rw-r--r--  1 squeezeboxserver nogroup    0 23. Feb 07:04 E1HJlp3h4x
-rw-r--r--  1 squeezeboxserver nogroup    0 23. Feb 12:04 f7l7o3GNru
drwxrwxrwt  2 root             root      40 20. Feb 21:28 .font-unix
-rw-r--r--  1 squeezeboxserver nogroup    0 23. Feb 13:04 gfQKSohWZg
-rw-r--r--  1 squeezeboxserver nogroup    0 23. Feb 07:04 gPcv6gsai5
drwxrwxrwt  2 root             root      40 20. Feb 21:28 .ICE-unix
-rw-r--r--  1 squeezeboxserver nogroup    0 23. Feb 11:04 iV6oxwH3iJ
-rw-r--r--  1 squeezeboxserver nogroup    0 23. Feb 09:04 l2Dj7NdfXa
-rw-r--r--  1 squeezeboxserver nogroup    0 22. Feb 23:04 L7Y1cYfZ8D
-rw-r--r--  1 squeezeboxserver nogroup    0 23. Feb 11:04 lBTLMC4z7d
-rw-r--r--  1 squeezeboxserver nogroup    0 22. Feb 22:04 lfyelNblKr
-rw-r--r--  1 squeezeboxserver nogroup    0 23. Feb 01:04 lIbF1HBg9w
-rw-r--r--  1 squeezeboxserver nogroup    0 22. Feb 22:04 lxYgS20dMj
-rw-r--r--  1 squeezeboxserver nogroup    0 23. Feb 12:04 LygurYuAEy
-rw-r--r--  1 squeezeboxserver nogroup    0 23. Feb 08:04 MkEEx4pAUd
-rw-r--r--  1 squeezeboxserver nogroup    0 23. Feb 06:04 nEGkiKUKen
-rw-r--r--  1 squeezeboxserver nogroup    0 23. Feb 08:04 NFjWh6ImQD
-rw-r--r--  1 squeezeboxserver nogroup    0 23. Feb 09:04 nJ9BZdaQqk
-rw-r--r--  1 squeezeboxserver nogroup    0 22. Feb 21:04 NoHZMmRVCI
-rw-r--r--  1 squeezeboxserver nogroup    0 23. Feb 03:04 oSMIz4dKeb
-rw-r--r--  1 squeezeboxserver nogroup    0 23. Feb 06:04 P3I0SZyn5e
-rw-r--r--  1 squeezeboxserver nogroup    0 22. Feb 21:04 PQcyE39WGE
-rw-r--r--  1 squeezeboxserver nogroup    0 23. Feb 02:04 qvN0XS0G2X
-rw-r--r--  1 squeezeboxserver nogroup    0 23. Feb 04:04 R4VmZb7SmJ
-rw-r--r--  1 squeezeboxserver nogroup    0 23. Feb 00:04 rc5BGovWW2
-rw-r--r--  1 squeezeboxserver nogroup    0 23. Feb 10:04 rig9Jfor5M
-rw-r--r--  1 squeezeboxserver nogroup    0 23. Feb 02:04 rIub6gWKnX
-rw-r--r--  1 squeezeboxserver nogroup    0 23. Feb 06:04 RU9Bn3wKOQ
-rw-r--r--  1 squeezeboxserver nogroup    0 23. Feb 05:04 S8nNLDnT5j
-rw-r--r--  1 squeezeboxserver nogroup    0 23. Feb 05:04 sLG90TSmcn
-rw-r--r--  1 squeezeboxserver nogroup    0 23. Feb 05:04 slldJuT3vj
drwx------  3 root             root      60 22. Feb 20:00 systemd-private-77adb340e35749048d0528ae29553b5d-coturn.service-ez281i
drwx------  3 root             root      60 22. Feb 20:00 systemd-private-77adb340e35749048d0528ae29553b5d-mjpg-streamer.service-TKG2Xi
drwx------  3 root             root      60 22. Feb 20:00 systemd-private-77adb340e35749048d0528ae29553b5d-redis-server.service-SpsxNi
drwx------  3 root             root      60 20. Feb 21:28 systemd-private-77adb340e35749048d0528ae29553b5d-systemd-logind.service-ez7cyi
drwxrwxrwt  2 root             root      40 20. Feb 21:28 .Test-unix
-rw-r--r--  1 squeezeboxserver nogroup    0 23. Feb 03:04 tI4ZqrIa2l
-rw-r--r--  1 squeezeboxserver nogroup    0 23. Feb 10:04 TSJsEq7o6g
-rw-r--r--  1 squeezeboxserver nogroup    0 23. Feb 11:47 UMHJJ6SZhO
-rw-r--r--  1 squeezeboxserver nogroup    0 23. Feb 01:04 UNbwpnF6Ir
-rw-r--r--  1 squeezeboxserver nogroup    0 23. Feb 13:04 uSWq9aEIXx
-rw-r--r--  1 squeezeboxserver nogroup    0 23. Feb 01:04 Utfa5iVdVa
-rw-r--r--  1 squeezeboxserver nogroup    0 22. Feb 21:04 vl4T2iCxLR
-rw-r--r--  1 squeezeboxserver nogroup    0 23. Feb 11:47 vLctoKU4hq
-rw-r--r--  1 squeezeboxserver nogroup    0 23. Feb 14:04 WWof8EkRKb
-rw-r--r--  1 squeezeboxserver nogroup    0 23. Feb 08:04 WyaNVVN1BX
drwxrwxrwt  2 root             root      40 20. Feb 21:28 .X11-unix
-rw-r--r--  1 squeezeboxserver nogroup    0 23. Feb 03:04 X5LG2ktjL1
drwxrwxrwt  2 root             root      40 20. Feb 21:28 .XIM-unix
-rw-r--r--  1 squeezeboxserver nogroup    0 22. Feb 22:04 XwBFseH8wO
-rw-r--r--  1 squeezeboxserver nogroup    0 23. Feb 00:04 y0693u4Q57

looks like your squeezeboxserver is filling up your /tmp

for squeezboxserver i remember i updated UPNPbridge recently and Spotty. for the ls -la command: isn´t the “0” between group and date the file size?

The zero is indeed the filesize, however in case of corrupted filesystem it is not weird to have seemingly zero-byte size files occupying the whole space. For example: https://unix.stackexchange.com/questions/97960/free-space-0-bytes-and-files-full-of-data-shown-as-empty

I recently used a new sdcard with the image of the previous one. Can a corrupt file system result of software or is it caused by sdcard? I have one more left, so I could use SanDisk again to see if tmp is full after 2 days.

In this case the filesystem is on ram, so it should not be connected to the sd card. Which board is this and how much memory does it have?
free -h

Mysterious, 1.9 GiB but nothing shown in the directory :thinking:. We have another command to list files and directory with sizes recursively: G_TREESIZE /tmp

That LMS files seem to be flags or hashes given via filename only, intended to be empty. That it does not clean them up is however not nice. Would be a good idea to put it into PrivateTmp=true to have them in an own private subfolder and cleared on service restart:

mkdir -p /etc/systemd/system/logitechmediaserver.service.d
echo -e '[Service]\nPrivateTmp=true' > /etc/systemd/system/logitechmediaserver.service.d/privateTmp.conf
systemctl daemon-reload
systemctl restart logitechmediaserver
rm /tmp/[^.]?????????

Ah, actually I’m not sure whether files in such private tmp subdirectories can be seen outside of the service’s environment. Do you actively stream something with mjpg-streamer? Probably it uses its /tmp in some circumstances. Could be verified via service restart, which clears its private tmp:

systemctl restart mjpg-streamer
df /tmp