Tmpfs ran full after using Nextcloud backup app

tl;dr

I suspect Nextcloud backup app filled up my RAM and my swap space, because memory_limit in /etc/php/8.2/cli/php.ini was set to -1 (unlimitted). I set this probably by myself.
When I rebooted my /etc/fstab was just a blank file. How can I asure that the system reboots correctly when RAM and swap is full? Is there even a way beside then to delete /tmp?


I have searched the existing open and closed issues
Pretty much similar to this issue

Required Information

DietPi 9.0.2 master
bookworm 0
Linux RPi4 6.1.21-v8+ #1642 SMP PREEMPT Mon Apr  3 17:24:16 BST 2023 aarch64 GNU/Linux
RPi4 arm64
PHP Version: 8.2.7

Additional Information (if applicable)

  • Software title: Nextcloud
  • Was the software title installed freshly or updated/migrated? neither
  • Can this issue be replicated on a fresh installation of DietPi? probably

What actually happend:

I installed the Nextcloud backup app on my instance and made backup. After the copying it does normally a compression of the backup.
I saw an error in the nextcloud log, that there is no space left on some path, I think it said /dev/root but I’m not sure. (I didn’t copy the log and the log is now gone)
I checked with df -h and saw, that tmpfs, mounted at /tmp is full an has 0% space left.
(Main partition has more then 60GB space left)

What made it worse:

I searched then the forum and found the thread linked above and did a reboot, like mentioned and ran into the same problem like in the linked thread:

-bash: /boot/dietpi/func/dietpi-globals: Datei oder Verzeichnis nicht gefunden
[FAILED] DietPi-Login | Failed to load DietPi-Globals. Skipping DietPi login scripts...

My fstab was empty, but since I learned from former mistakes I had a backup of fstab. I connected the disk to another machine (since it was read-only when booted up) and replaced it, which fixed the error.
So /tmp is the swap file, right?
I assume the compression process of Nextcloud backup hat eaten up my RAM and also filled the swap completly.
When I rebooted, there was no space left to execute some shutdown scripts or something, and my /etc/fstab got wiped?
I also did fschk which found only the dirty bit set, because of the ungracefully shutdown?!

Why did this happen?

I checked my memory_limit for PHP:

root@RPi4:~# find /etc/php* -name "*.ini" -exec grep -H "memory_limit" {} \; 2>/dev/null
/etc/php/8.2/fpm/php.ini:memory_limit = 128M
/etc/php/8.2/fpm/conf.d/98-dietpi-nextcloud.ini:memory_limit=512M
/etc/php/8.2/cli/php.ini:memory_limit = -1
/etc/php/8.2/cli/conf.d/98-dietpi-nextcloud.ini:memory_limit=512M
/etc/php/8.2/phpdbg/php.ini:memory_limit = 256M
/etc/php/8.2/phpdbg/conf.d/98-dietpi-nextcloud.ini:memory_limit=512M
/etc/php/8.2/mods-available/dietpi-nextcloud.ini:memory_limit=512M

root@RPi4:~# cat /var/www/nextcloud/.user.ini | grep memory 
(no custom memory_limit set)

So interesting is this:

/etc/php/8.2/cli/php.ini:memory_limit = -1
/etc/php/8.2/cli/conf.d/98-dietpi-nextcloud.ini:memory_limit=512M

Idk if Nextcloud backup app uses php cli for compression, but it’s completly plausible, since it’s a kind of background task.
But the drop-in config for nextcloud should’ve overwritten the global value of -1 (no limit)?!

I checked default memory_limit for php cli and it looks like it is by default 128M.
So the only logical conclusion is, that I set the -1 by myself?! :upside_down_face: :melting_face:

Maybe somebody has alternative ideas what happened?

(I’ve also set now /etc/php/8.2/cli/php.ini:memory_limit to 512M)

This should never happen.

Usually rebooting a system should clear memory as well as /tmp

I mean it was still there, but just a blank file.

nope /tmp is the temp file system, SWAP is something else :stuck_out_tongue: You can show SWAP file as follow

swapon -s
1 Like

Ok then nextcloud filled up tmp but it makes no sense to me that then the system reported, it is no space left. I couldn’t execute any dietpi- script.
So maybe something else was going on, possibly even not connected to the netxtcloud-backup :thinking:

Well even /tmp is a temporary file system, it has a fixed size. Did you check df -h?

yes, the 1,9G were filled up 100%. Then I searched for it on the forum and found the thread linked above, but I just read the advice to reboot without reading a bit further and then I ran into the same problem like OP from the mentioned thread :sweat_smile:
I panicked a bit since this was the productive system and DNS resolution for the whole network depends on it.
So I didn’t gather much information, just tried to get it back up as fast as possible.

I digged a bit and found this issue on github, which sound similiar to mine:
https://github.com/nextcloud/backup/issues/485

I will make a dedicated Nextcloud temp dir and add this to /etc/php/8.2/cli/conf.d/98-dietpi-nextcloud.ini and see what happens when the next backup is triggered in 3 days.

I checked also if it is applied via

php -c /etc/php/8.2/cli/conf.d/98-dietpi-nextcloud.ini -c /var/www/nextcloud/.user.ini -i |  grep dir
sys_temp_dir => /nc_tmp => /nc_tmp
2 Likes