after restoring /etc/fstab and rebooting i am not able to connect via ssh (connection refused). How long does a fsck last? Could it be the reason for a refused connection?
do you see green led still flashing/active?
I pulled powercable and now its back again
So I think it s best to shutdown, make a new image and use a new card?
I really appreciate your help.
Doing backups is always a good idea. You might be ok to continue running your system, if there are not that much changes, not that much content on your system and you are ok to restore an older image backup without issues.
A full fsck can indeed take a while, especially when there are many errors, which leads to many fix/check iterations. Checking RPi LEDs for activity, the green one for filesystem activity, is indeed a proper hint. Pulling the power plug while fsck is running, is not a good idea when aiming to repair that filesystem. However, yes probably flashing a new image is/was a good idea .
In case it helps other users who admin a system that runs out of space, I found that docker, as used by Teslamate, seems to leak about three images on each restart. Executing
docker system prune -a
once in a while seems to help. I do it in my shell script that updates Teslamate as the process can be very slow.
Just throwing this out there to help other users and feed search engines to help other users. I’d lost many GB on a filesystem that only had 30, so my Teslamate instance was broken until I could return and learn this tidbit. The prune seems to work reliably.
thx for your message, but it has nothing to do with original issue.
/tmp keeps getting full after 3-4 days. This causes write errors in e.g. dietpi-update. A reboot solves and cleans /tmp.
Usingls -l does not show any big file like 1.9G on tmp. How can I identify which causes the full /tmp?
What do you think of A weekly reboot via Cron.?
I rebooted today, it takes some time to investigate again…
to find the 20 biggest files
du -a /tmp | sort -n -r | head -n 20
nearly 2 days after reboot df -h shows /tmp is neary full
root@DietPi4 [19:45:34]~ # df -h
Dateisystem Größe Benutzt Verf. Verw% Eingehängt auf
/dev/root 235G 113G 113G 51% /
devtmpfs 1,9G 0 1,9G 0% /dev
tmpfs 1,9G 1,7M 1,9G 1% /dev/shm
tmpfs 771M 46M 726M 6% /run
tmpfs 5,0M 4,0K 5,0M 1% /run/lock
tmpfs 1,9G 1,9G 52M 98% /tmp
tmpfs 50M 132K 50M 1% /var/log
/dev/mmcblk0p1 253M 52M 201M 21% /boot
tmpfs 386M 8,0K 386M 1% /run/user/999
/dev/sda1 113G 61G 52G 55% /mnt/128
/dev/sdb1 229G 160G 69G 70% /mnt/256
tmpfs 386M 8,0K 386M 1% /run/user/0
When i try to find files in /tmp with du -a /tmp | sort -n -r | head -n 20 all files are 0bytes?
0 /tmp/.XIM-unix
0 /tmp/.X11-unix
0 /tmp/.Test-unix
0 /tmp/systemd-private-77adb340e35749048d0528ae29553b5d-systemd-logind.service-ez7cyi/tmp
0 /tmp/systemd-private-77adb340e35749048d0528ae29553b5d-systemd-logind.service-ez7cyi
0 /tmp/systemd-private-77adb340e35749048d0528ae29553b5d-redis-server.service-1tvB7f/tmp
0 /tmp/systemd-private-77adb340e35749048d0528ae29553b5d-redis-server.service-1tvB7f
0 /tmp/systemd-private-77adb340e35749048d0528ae29553b5d-mjpg-streamer.service-bkS3Ff/tmp
0 /tmp/systemd-private-77adb340e35749048d0528ae29553b5d-mjpg-streamer.service-bkS3Ff
0 /tmp/systemd-private-77adb340e35749048d0528ae29553b5d-coturn.service-MyDD0i/tmp
0 /tmp/systemd-private-77adb340e35749048d0528ae29553b5d-coturn.service-MyDD0i
0 /tmp/.ICE-unix
0 /tmp/.font-unix
0 /tmp
Can you try it again? You are running it as root, right?
sync; du -csh /tmp/*
I am running it at root, correct:
Here´s the result, i dont see any files bigger than 0 bytes.
root@DietPi4 [14:23:19]~ # sync; du -csh /tmp/*
0 /tmp/0jaTwb3Gka
0 /tmp/0q9h2MTQSm
0 /tmp/1t8gHWsHYP
0 /tmp/1v8WPjKXRP
0 /tmp/7Y8hkDQycJ
0 /tmp/7zu5c0VCbq
0 /tmp/8Jrcu8Ga5R
0 /tmp/8uDOJYtE0C
0 /tmp/9elp1V357x
0 /tmp/ap11QoCzIC
0 /tmp/aV5VyVjUZ1
0 /tmp/aZ9vYWyFlm
0 /tmp/BCln6pbQdX
0 /tmp/bpjr6FEwZe
0 /tmp/CsasWQtx5k
0 /tmp/E1HJlp3h4x
0 /tmp/f7l7o3GNru
0 /tmp/gfQKSohWZg
0 /tmp/gPcv6gsai5
0 /tmp/iV6oxwH3iJ
0 /tmp/l2Dj7NdfXa
0 /tmp/L7Y1cYfZ8D
0 /tmp/lBTLMC4z7d
0 /tmp/lfyelNblKr
0 /tmp/lIbF1HBg9w
0 /tmp/lxYgS20dMj
0 /tmp/LygurYuAEy
0 /tmp/MkEEx4pAUd
0 /tmp/nEGkiKUKen
0 /tmp/NFjWh6ImQD
0 /tmp/nJ9BZdaQqk
0 /tmp/NoHZMmRVCI
0 /tmp/oSMIz4dKeb
0 /tmp/P3I0SZyn5e
0 /tmp/PQcyE39WGE
0 /tmp/qvN0XS0G2X
0 /tmp/R4VmZb7SmJ
0 /tmp/rc5BGovWW2
0 /tmp/rig9Jfor5M
0 /tmp/rIub6gWKnX
0 /tmp/RU9Bn3wKOQ
0 /tmp/S8nNLDnT5j
0 /tmp/sLG90TSmcn
0 /tmp/slldJuT3vj
0 /tmp/systemd-private-77adb340e35749048d0528ae29553b5d-coturn.service-ez281i
0 /tmp/systemd-private-77adb340e35749048d0528ae29553b5d-mjpg-streamer.service-TKG2Xi
0 /tmp/systemd-private-77adb340e35749048d0528ae29553b5d-redis-server.service-SpsxNi
0 /tmp/systemd-private-77adb340e35749048d0528ae29553b5d-systemd-logind.service-ez7cyi
0 /tmp/tI4ZqrIa2l
0 /tmp/TSJsEq7o6g
0 /tmp/UMHJJ6SZhO
0 /tmp/UNbwpnF6Ir
0 /tmp/uSWq9aEIXx
0 /tmp/Utfa5iVdVa
0 /tmp/vl4T2iCxLR
0 /tmp/vLctoKU4hq
0 /tmp/WWof8EkRKb
0 /tmp/WyaNVVN1BX
0 /tmp/X5LG2ktjL1
0 /tmp/XwBFseH8wO
0 /tmp/y0693u4Q57
0 insgesamt
df -h:
df -h
Dateisystem Größe Benutzt Verf. Verw% Eingehängt auf
/dev/root 235G 113G 113G 51% /
devtmpfs 1,9G 0 1,9G 0% /dev
tmpfs 1,9G 1,9M 1,9G 1% /dev/shm
tmpfs 771M 78M 693M 11% /run
tmpfs 5,0M 4,0K 5,0M 1% /run/lock
tmpfs 1,9G 1,9G 0 100% /tmp
tmpfs 50M 64K 50M 1% /var/log
/dev/mmcblk0p1 253M 52M 201M 21% /boot
tmpfs 386M 8,0K 386M 1% /run/user/999
/dev/sda1 113G 62G 51G 55% /mnt/128
/dev/sdb1 229G 160G 69G 70% /mnt/256
tmpfs 386M 8,0K 386M 1% /run/user/0
strange, can you share
ls -la /tmp/
ls -la results in:
root@DietPi4 [14:24:06]~ # ls -la /tmp/
insgesamt 4
drwxrwxrwt 11 root root 1360 23. Feb 14:28 .
drwxr-xr-x 20 root root 4096 16. Jan 11:25 ..
-rw-r--r-- 1 squeezeboxserver nogroup 0 23. Feb 09:04 0jaTwb3Gka
-rw-r--r-- 1 squeezeboxserver nogroup 0 23. Feb 02:04 0q9h2MTQSm
-rw-r--r-- 1 squeezeboxserver nogroup 0 23. Feb 14:04 1t8gHWsHYP
-rw-r--r-- 1 squeezeboxserver nogroup 0 23. Feb 07:04 1v8WPjKXRP
-rw-r--r-- 1 squeezeboxserver nogroup 0 22. Feb 23:04 7Y8hkDQycJ
-rw-r--r-- 1 squeezeboxserver nogroup 0 23. Feb 00:04 7zu5c0VCbq
-rw-r--r-- 1 squeezeboxserver nogroup 0 23. Feb 10:04 8Jrcu8Ga5R
-rw-r--r-- 1 squeezeboxserver nogroup 0 23. Feb 04:04 8uDOJYtE0C
-rw-r--r-- 1 squeezeboxserver nogroup 0 23. Feb 11:04 9elp1V357x
-rw-r--r-- 1 squeezeboxserver nogroup 0 23. Feb 11:47 ap11QoCzIC
-rw-r--r-- 1 squeezeboxserver nogroup 0 23. Feb 13:04 aV5VyVjUZ1
-rw-r--r-- 1 squeezeboxserver nogroup 0 23. Feb 14:04 aZ9vYWyFlm
-rw-r--r-- 1 squeezeboxserver nogroup 0 23. Feb 04:04 BCln6pbQdX
-rw-r--r-- 1 squeezeboxserver nogroup 0 22. Feb 23:04 bpjr6FEwZe
-rw-r--r-- 1 squeezeboxserver nogroup 0 23. Feb 12:04 CsasWQtx5k
-rw-r--r-- 1 squeezeboxserver nogroup 0 23. Feb 07:04 E1HJlp3h4x
-rw-r--r-- 1 squeezeboxserver nogroup 0 23. Feb 12:04 f7l7o3GNru
drwxrwxrwt 2 root root 40 20. Feb 21:28 .font-unix
-rw-r--r-- 1 squeezeboxserver nogroup 0 23. Feb 13:04 gfQKSohWZg
-rw-r--r-- 1 squeezeboxserver nogroup 0 23. Feb 07:04 gPcv6gsai5
drwxrwxrwt 2 root root 40 20. Feb 21:28 .ICE-unix
-rw-r--r-- 1 squeezeboxserver nogroup 0 23. Feb 11:04 iV6oxwH3iJ
-rw-r--r-- 1 squeezeboxserver nogroup 0 23. Feb 09:04 l2Dj7NdfXa
-rw-r--r-- 1 squeezeboxserver nogroup 0 22. Feb 23:04 L7Y1cYfZ8D
-rw-r--r-- 1 squeezeboxserver nogroup 0 23. Feb 11:04 lBTLMC4z7d
-rw-r--r-- 1 squeezeboxserver nogroup 0 22. Feb 22:04 lfyelNblKr
-rw-r--r-- 1 squeezeboxserver nogroup 0 23. Feb 01:04 lIbF1HBg9w
-rw-r--r-- 1 squeezeboxserver nogroup 0 22. Feb 22:04 lxYgS20dMj
-rw-r--r-- 1 squeezeboxserver nogroup 0 23. Feb 12:04 LygurYuAEy
-rw-r--r-- 1 squeezeboxserver nogroup 0 23. Feb 08:04 MkEEx4pAUd
-rw-r--r-- 1 squeezeboxserver nogroup 0 23. Feb 06:04 nEGkiKUKen
-rw-r--r-- 1 squeezeboxserver nogroup 0 23. Feb 08:04 NFjWh6ImQD
-rw-r--r-- 1 squeezeboxserver nogroup 0 23. Feb 09:04 nJ9BZdaQqk
-rw-r--r-- 1 squeezeboxserver nogroup 0 22. Feb 21:04 NoHZMmRVCI
-rw-r--r-- 1 squeezeboxserver nogroup 0 23. Feb 03:04 oSMIz4dKeb
-rw-r--r-- 1 squeezeboxserver nogroup 0 23. Feb 06:04 P3I0SZyn5e
-rw-r--r-- 1 squeezeboxserver nogroup 0 22. Feb 21:04 PQcyE39WGE
-rw-r--r-- 1 squeezeboxserver nogroup 0 23. Feb 02:04 qvN0XS0G2X
-rw-r--r-- 1 squeezeboxserver nogroup 0 23. Feb 04:04 R4VmZb7SmJ
-rw-r--r-- 1 squeezeboxserver nogroup 0 23. Feb 00:04 rc5BGovWW2
-rw-r--r-- 1 squeezeboxserver nogroup 0 23. Feb 10:04 rig9Jfor5M
-rw-r--r-- 1 squeezeboxserver nogroup 0 23. Feb 02:04 rIub6gWKnX
-rw-r--r-- 1 squeezeboxserver nogroup 0 23. Feb 06:04 RU9Bn3wKOQ
-rw-r--r-- 1 squeezeboxserver nogroup 0 23. Feb 05:04 S8nNLDnT5j
-rw-r--r-- 1 squeezeboxserver nogroup 0 23. Feb 05:04 sLG90TSmcn
-rw-r--r-- 1 squeezeboxserver nogroup 0 23. Feb 05:04 slldJuT3vj
drwx------ 3 root root 60 22. Feb 20:00 systemd-private-77adb340e35749048d0528ae29553b5d-coturn.service-ez281i
drwx------ 3 root root 60 22. Feb 20:00 systemd-private-77adb340e35749048d0528ae29553b5d-mjpg-streamer.service-TKG2Xi
drwx------ 3 root root 60 22. Feb 20:00 systemd-private-77adb340e35749048d0528ae29553b5d-redis-server.service-SpsxNi
drwx------ 3 root root 60 20. Feb 21:28 systemd-private-77adb340e35749048d0528ae29553b5d-systemd-logind.service-ez7cyi
drwxrwxrwt 2 root root 40 20. Feb 21:28 .Test-unix
-rw-r--r-- 1 squeezeboxserver nogroup 0 23. Feb 03:04 tI4ZqrIa2l
-rw-r--r-- 1 squeezeboxserver nogroup 0 23. Feb 10:04 TSJsEq7o6g
-rw-r--r-- 1 squeezeboxserver nogroup 0 23. Feb 11:47 UMHJJ6SZhO
-rw-r--r-- 1 squeezeboxserver nogroup 0 23. Feb 01:04 UNbwpnF6Ir
-rw-r--r-- 1 squeezeboxserver nogroup 0 23. Feb 13:04 uSWq9aEIXx
-rw-r--r-- 1 squeezeboxserver nogroup 0 23. Feb 01:04 Utfa5iVdVa
-rw-r--r-- 1 squeezeboxserver nogroup 0 22. Feb 21:04 vl4T2iCxLR
-rw-r--r-- 1 squeezeboxserver nogroup 0 23. Feb 11:47 vLctoKU4hq
-rw-r--r-- 1 squeezeboxserver nogroup 0 23. Feb 14:04 WWof8EkRKb
-rw-r--r-- 1 squeezeboxserver nogroup 0 23. Feb 08:04 WyaNVVN1BX
drwxrwxrwt 2 root root 40 20. Feb 21:28 .X11-unix
-rw-r--r-- 1 squeezeboxserver nogroup 0 23. Feb 03:04 X5LG2ktjL1
drwxrwxrwt 2 root root 40 20. Feb 21:28 .XIM-unix
-rw-r--r-- 1 squeezeboxserver nogroup 0 22. Feb 22:04 XwBFseH8wO
-rw-r--r-- 1 squeezeboxserver nogroup 0 23. Feb 00:04 y0693u4Q57
looks like your squeezeboxserver is filling up your /tmp
for squeezboxserver i remember i updated UPNPbridge recently and Spotty. for the ls -la command: isn´t the “0” between group and date the file size?
The zero is indeed the filesize, however in case of corrupted filesystem it is not weird to have seemingly zero-byte size files occupying the whole space. For example: https://unix.stackexchange.com/questions/97960/free-space-0-bytes-and-files-full-of-data-shown-as-empty
I recently used a new sdcard with the image of the previous one. Can a corrupt file system result of software or is it caused by sdcard? I have one more left, so I could use SanDisk again to see if tmp is full after 2 days.
In this case the filesystem is on ram, so it should not be connected to the sd card. Which board is this and how much memory does it have?
free -h
Mysterious, 1.9 GiB but nothing shown in the directory . We have another command to list files and directory with sizes recursively: G_TREESIZE /tmp
That LMS files seem to be flags or hashes given via filename only, intended to be empty. That it does not clean them up is however not nice. Would be a good idea to put it into PrivateTmp=true to have them in an own private subfolder and cleared on service restart:
mkdir -p /etc/systemd/system/logitechmediaserver.service.d
echo -e '[Service]\nPrivateTmp=true' > /etc/systemd/system/logitechmediaserver.service.d/privateTmp.conf
systemctl daemon-reload
systemctl restart logitechmediaserver
rm /tmp/[^.]?????????
Ah, actually I’m not sure whether files in such private tmp subdirectories can be seen outside of the service’s environment. Do you actively stream something with mjpg-streamer? Probably it uses its /tmp in some circumstances. Could be verified via service restart, which clears its private tmp:
systemctl restart mjpg-streamer
df /tmp