i had to reboot the pi4 with 4GB RAM, tmp was cleared. The mjpg-streamer was an attempt(without reading) to get a usbwebcam working, i uninstalled it.
G_TREESIZE /tmp shows at the moment 0,0 bytes for 4 files (logind.service, redis-server, lms, coturn).
I will try PrivateTmp=true for LMS and will see what happens in the next two days.
(last command rm /tmp/[^.]??? does not work, file or folder not found).
The command was to remove the LMS files before reboot. Makes sense that they are not there anymore, especially with PrivateTmp=true (which moves them into a subdirectory).
tmp is full again, restart of lms has no succes,
but i found that nextcloud is logging every second, i was not aware of this.
I changed the loglevel in var/www/nextcloud/config/config.php
from 0 to 2.
Can i see in tmp if nextcloud has a PrivateTmp and could be the cause for full /tmp directory?
I answer myself, i found
Doctrine\DBAL\Exception\DriverException: An exception occurred while executing a query: SQLSTATE[HY000]: General error: 1021 Disk full (/tmp/#sql-temptable-3aa-11e5-ca.MAI); waiting for someone to free some space... (errno: 28 "No space left on device")
in the log were mostly keepassweb, recognize, music, workflowocr, i disabled these apps.
I reboot and keep an eye on nextcloud.
root@DietPi4 [21:02:58]/tmp # systemctl restart logitechmediaserver
root@DietPi4 [21:03:03]/tmp # df -h
Dateisystem Größe Benutzt Verf. Verw% Eingehängt auf
/dev/root 235G 113G 112G 51% /
devtmpfs 1,9G 0 1,9G 0% /dev
tmpfs 1,9G 1,7M 1,9G 1% /dev/shm
tmpfs 764M 78M 687M 11% /run
tmpfs 5,0M 4,0K 5,0M 1% /run/lock
tmpfs 1,9G 1,9G 0 100% /tmp
tmpfs 50M 104K 50M 1% /var/log
/dev/mmcblk0p1 253M 52M 201M 21% /boot
tmpfs 382M 8,0K 382M 1% /run/user/999
/dev/sda1 113G 62G 51G 55% /mnt/128
/dev/sdb1 229G 160G 69G 70% /mnt/256
tmpfs 382M 8,0K 382M 1% /run/user/0
root@DietPi4 [21:03:06]/tmp # G_TREESIZE /tmp
0,0 bytes /tmp/.XIM-unix
0,0 bytes /tmp/.X11-unix
0,0 bytes /tmp/.Test-unix
0,0 bytes /tmp/systemd-private-7437effbbf6943d58b62f89f700b21b3-systemd-logind.service-pKtWvi
0,0 bytes /tmp/systemd-private-7437effbbf6943d58b62f89f700b21b3-redis-server.service-GRG9Ai
0,0 bytes /tmp/systemd-private-7437effbbf6943d58b62f89f700b21b3-logitechmediaserver.service-oVWvSg
0,0 bytes /tmp/systemd-private-7437effbbf6943d58b62f89f700b21b3-coturn.service-S9OB5g
0,0 bytes /tmp/.ICE-unix
0,0 bytes /tmp/.font-unix
0,0 bytes /tmp
still strange that we don’t see what files are using /tmp
Does /tmp is still filling up? Maybe you can check before it is getting to 100% usage.
22 hours running, 32%tmp is used (500Mb of 1,9Gb)
──
root@DietPi4 [18:26:30]~ # df -h
Dateisystem Größe Benutzt Verf. Verw% Eingehängt auf
/dev/root 235G 113G 112G 51% /
devtmpfs 1,9G 0 1,9G 0% /dev
tmpfs 1,9G 1,6M 1,9G 1% /dev/shm
tmpfs 764M 79M 686M 11% /run
tmpfs 5,0M 4,0K 5,0M 1% /run/lock
tmpfs 1,9G 598M 1,3G 32% /tmp
tmpfs 50M 20K 50M 1% /var/log
/dev/mmcblk0p1 253M 52M 201M 21% /boot
tmpfs 382M 8,0K 382M 1% /run/user/999
/dev/sda1 113G 62G 51G 56% /mnt/128
/dev/sdb1 229G 160G 69G 70% /mnt/256
tmpfs 382M 8,0K 382M 1% /run/user/0
root@DietPi4 [18:29:27]/tmp # ls -al insgesamt 4
drwxrwxrwt 11 root root 220 27. Feb 18:25 .
drwxr-xr-x 20 root root 4096 27. Feb 05:12 ..
drwxrwxrwt 2 root root 40 26. Feb 21:22 .font-unix
drwxrwxrwt 2 root root 40 26. Feb 21:22 .ICE-unix
drwx------ 3 root root 60 26. Feb 21:22 systemd-private-9483f30167d642159b4029a36bc1dc58-coturn.service-uqK5Oh
drwx------ 3 root root 60 26. Feb 21:22 systemd-private-9483f30167d642159b4029a36bc1dc58-logitechmediaserver.service-40tfjh
drwx------ 3 root root 60 26. Feb 21:22 systemd-private-9483f30167d642159b4029a36bc1dc58-redis-server.service-cVM9wj
drwx------ 3 root root 60 26. Feb 21:22 systemd-private-9483f30167d642159b4029a36bc1dc58-systemd-logind.service-cIrAnh
drwxrwxrwt 2 root root 40 26. Feb 21:22 .Test-unix
drwxrwxrwt 2 root root 40 26. Feb 21:22 .X11-unix
drwxrwxrwt 2 root root 40 26. Feb 21:22 .XIM-unix
root@DietPi4 [18:29:31]/tmp #
I guess following doesn’t show anything still. Right?
G_TREESIZE /tmp
G_TREESIZE /tmp gives similar result asls -la
Yes contact might be similar but important question is if it gives a file size.
Isn’t the file size between group and date in ls -la?
root@DietPi4 [19:31:46]~ # G_TREESIZE /tmp
0,0 bytes /tmp/.XIM-unix
0,0 bytes /tmp/.X11-unix
0,0 bytes /tmp/.Test-unix
0,0 bytes /tmp/systemd-private-9483f30167d642159b4029a36bc1dc58-systemd-logind.service-cIrAnh
0,0 bytes /tmp/systemd-private-9483f30167d642159b4029a36bc1dc58-redis-server.service-cVM9wj
0,0 bytes /tmp/systemd-private-9483f30167d642159b4029a36bc1dc58-logitechmediaserver.service-40tfjh
0,0 bytes /tmp/systemd-private-9483f30167d642159b4029a36bc1dc58-coturn.service-uqK5Oh
0,0 bytes /tmp/.ICE-unix
0,0 bytes /tmp/.font-unix
0,0 bytes /tmp
ls -la is showing the size of files only. It did not show any information on folders or the content inside. This is what usually G_TREESIZE does. It sum up the space used by folders. But in your case, something strange is going on as you system is showing always 0. Doesn’t matter how much space is used.
Please try to restart or stop the other services with use a private tmp subdir:
systemctl stop coturn systemd-logind
systemctl restart redis-server # just restarting since Nextcloud strictly requires it
sleep 2
df -h /tmp
systemctl start coturn systemd-logind
Why cannot I see that nextcloud MySQL is using /tmp?
May be offtopic, There is one failure in nextcloud system check
The PHP OPcache module is not properly configured:
The OPcache interned strings buffer is nearly full. To assure that repeating strings can be effectively cached, it is recommended to apply opcache.interned_strings_buffer to your PHP configuration with a value higher than 8.
The second one in protocol
Fatal
no app in context
2022-02-26T21:22:14+0100
Doctrine\DBAL\Exception\DriverException: An exception occurred while executing a query: SQLSTATE[HY000]: General error: 1021 Disk full (/tmp/#sql-temptable-3aa-127b-78.MAI); waiting for someone to free some space... (errno: 28 "No space left on device").
root@DietPi4 [04:46:07]/tmp # df -h /tmp
Dateisystem Größe Benutzt Verf. Verw% Eingehängt auf
tmpfs 1,9G 1,1G 897M 54% /tmp
root@DietPi4 [04:46:23]/tmp # systemctl stop coturn systemd-logind
root@DietPi4 [04:46:47]/tmp # systemctl restart redis-server
root@DietPi4 [04:47:17]/tmp # sleep 2
root@DietPi4 [04:47:36]/tmp # df -h /tmp
Dateisystem Größe Benutzt Verf. Verw% Eingehängt auf
tmpfs 1,9G 1,1G 895M 54% /tmp
root@DietPi4 [04:47:56]/tmp # systemctl start coturn systemd-logind
you could try to restart all DietPi managed services
dietpi-services restart
Nope, before and after restarting all “dietpi managed services” /tmp is 95%full.
Should I do a fresh install or continue investigating?
Or add a Cron job and reboot every 2 days?
(I could try to see if backup works in a virtual machine)
You can’t restore a backup from a physical device on a virtual machine. They both use different architecture.
I meant to restore the sql-dump of nextcloud in a virtual machine, that’s most important for me.
I am not sure how to export/import the certificates for my domain yet.
Pihole and lms would be ok
The SQL dump is one thing. The Nextcloud data another. They would need to be migrated as well.
HTTPS certificates should not be an issue, as they could be recreated easily.
Pihole themselves offer a function to export/import settings.
I used lsof to display open files, but still do not understand where to look. maybe its a cron issue?
root@DietPi4 [10:30:22]~ # df -h
Dateisystem Größe Benutzt Verf. Verw% Eingehängt auf
/dev/root 235G 110G 116G 49% /
devtmpfs 1,9G 0 1,9G 0% /dev
tmpfs 1,9G 1,3M 1,9G 1% /dev/shm
tmpfs 764M 18M 747M 3% /run
tmpfs 5,0M 4,0K 5,0M 1% /run/lock
tmpfs 1,9G 321M 1,6G 17% /tmp
tmpfs 50M 40K 50M 1% /var/log
/dev/mmcblk0p1 253M 52M 201M 21% /boot
tmpfs 382M 8,0K 382M 1% /run/user/999
/dev/sda1 113G 62G 51G 56% /mnt/128
/dev/sdb1 229G 160G 69G 70% /mnt/256
tmpfs 382M 8,0K 382M 1% /run/user/0
root@DietPi4 [10:30:28]~ # lsof /tmp
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
mariadbd 797 mysql 6u REG 0,29 0 30 /tmp/ibUUhR4t (deleted)
mariadbd 797 mysql 7u REG 0,29 0 31 /tmp/ibHz3QDr (deleted)
mariadbd 797 mysql 8u REG 0,29 0 33 /tmp/ibe6Tf7r (deleted)
mariadbd 797 mysql 11u REG 0,29 0 34 /tmp/ibGSCdxs (deleted)
php-fpm7. 873 root 3u REG 0,29 0 38 /tmp/.ZendSem.M32GOb (deleted)
php-fpm7. 8139 www-data 3u REG 0,29 0 38 /tmp/.ZendSem.M32GOb (deleted)
php-fpm7. 8501 www-data 3u REG 0,29 0 38 /tmp/.ZendSem.M32GOb (deleted)
php-fpm7. 8507 www-data 3u REG 0,29 0 38 /tmp/.ZendSem.M32GOb (deleted)
php-fpm7. 8875 www-data 3u REG 0,29 0 38 /tmp/.ZendSem.M32GOb (deleted)
cron 13495 root 6u REG 0,29 335817057 3740 /tmp/#3740 (deleted)
dash 13497 root 1u REG 0,29 335817057 3740 /tmp/#3740 (deleted)
dash 13497 root 2u REG 0,29 335817057 3740 /tmp/#3740 (deleted)
run-parts 13500 root 1u REG 0,29 335817057 3740 /tmp/#3740 (deleted)
run-parts 13500 root 2u REG 0,29 335817057 3740 /tmp/#3740 (deleted)
Indeed it seems to be a cron job creating ~320 MiB large files (and removes them again). Check your cron jobs in /etc/cron.daily/ and /etc/cron.hourly/.