No space left on device

Troubleshooting
1

Hello Joulinar, Hi MichaIng,

i updated recently to buster, to 7.9.3 and now to 8.0.2.
I know mosquitto is not running, though i did the workaround.
Now i see that coturn service cannot be started.
When opening dietpi-software or dietpi-drivemanager, i get errormessages like:

/boot/dietpi/func/dietpi-globals: line 783: echo: write error: No space left on device

I recently installed lidarr, transmittion and Jackett. They do not download currently.
(The temperature has increased from 55°C upto 65°.)

I do not think, the SD card is out of space:
cat /var/tcat /var/tmp/dietpi/logs/fs_partition_resize.log

Removed /etc/systemd/system/local-fs.target.wants/dietpi-fs_partition_resize.service.
Disk /dev/mmcblk0: 238.3 GiB, 255869321216 bytes, 499744768 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xe8af6eb2

Old situation:

Device         Boot  Start       End   Sectors  Size Id Type
/dev/mmcblk0p1        8192    532479    524288  256M  c W95 FAT32 (LBA)
/dev/mmcblk0p2      532480 499744767 499212288  238G 83 Linux

/dev/mmcblk0p2:
New situation:
Disklabel type: dos
Disk identifier: 0xe8af6eb2

Device         Boot  Start       End   Sectors  Size Id Type
/dev/mmcblk0p1        8192    532479    524288  256M  c W95 FAT32 (LBA)
/dev/mmcblk0p2      532480 499744767 499212288  238G 83 Linux

The partition table has been altered.
resize2fs 1.44.5 (15-Dec-2018)
The filesystem is already 62401536 (4k) blocks long.  Nothing to do!
2

can you share

df -h
3

It seems tempfs is full.
(Idea: I started nextcloud new included backup and could not stop anymore.)

root@DietPi4 [23:41:51]~ # df -h
Dateisystem    Größe Benutzt Verf. Verw% Eingehängt auf
/dev/root       235G    123G  102G   55% /
devtmpfs        1,9G       0  1,9G    0% /dev
tmpfs           1,9G    1,6M  1,9G    1% /dev/shm
tmpfs           771M     81M  690M   11% /run
tmpfs           5,0M    4,0K  5,0M    1% /run/lock
tmpfs           1,9G    1,9G     0  100% /tmp
tmpfs            50M    4,3M   46M    9% /var/log
/dev/mmcblk0p1  253M     51M  202M   21% /boot
tmpfs           386M    8,0K  386M    1% /run/user/999
/dev/sdb1       229G     30G  200G   13% /mnt/256
/dev/sda1       113G     57G   57G   51% /mnt/128
tmpfs           386M    8,0K  386M    1% /run/user/0
4

reboot your system and try again

5 
-bash: /boot/dietpi/func/dietpi-globals: Datei oder Verzeichnis nicht gefunden
[FAILED] DietPi-Login | Failed to load DietPi-Globals. Skipping DietPi login scripts...

So no dietpi-software running…

But /tmp was cleared

root@DietPi4 [23:50:02]~ # df -h
Dateisystem    Größe Benutzt Verf. Verw% Eingehängt auf
/dev/root       235G    123G  102G   55% /
devtmpfs        1,9G       0  1,9G    0% /dev
tmpfs           1,9G       0  1,9G    0% /dev/shm
tmpfs           771M    8,7M  762M    2% /run
tmpfs           5,0M    4,0K  5,0M    1% /run/lock
tmpfs           386M    8,0K  386M    1% /run/user/0
6

you are missing the boot file system. It was present on your first output but missing now.

/dev/mmcblk0p1  253M     51M  202M   21% /boot

any kernel error message?

dmesg -l err,crit,alert,emerg
7 
dmesg -l err,crit,alert,emerg
[    0.545239] bcm2708_fb soc:fb: Unable to determine number of FBs. Disabling driver.
8

can you check following

journalctl -u boot.mount
9

There are no entries

root@DietPi4 [23:56:10]/etc # journalctl -u boot.mount
-- Journal begins at Tue 2022-01-11 23:49:40 CET, ends at Wed 2022-01-12 00:00:20 CET. --
-- No entries --
10

hmm looks like the device was skipped during boot. Are you able to mount it manually?

mount /dev/mmcblk0p1 /boot
ls -l /boot
11

Yes, that works,

root@DietPi4 [00:08:22]/etc # mount /dev/mmcblk0p1 /boot
root@DietPi4 [00:08:25]/etc # ls -l /boot
insgesamt 49999
-rwxr-xr-x 1 root root   26146  7. Dez 01:31  bcm2708-rpi-b.dtb
-rwxr-xr-x 1 root root   26417  7. Dez 01:31  bcm2708-rpi-b-plus.dtb
-rwxr-xr-x 1 root root   25757  7. Dez 01:31  bcm2708-rpi-b-rev1.dtb
-rwxr-xr-x 1 root root   26068  7. Dez 01:31  bcm2708-rpi-cm.dtb
-rwxr-xr-x 1 root root   25891  7. Dez 01:31  bcm2708-rpi-zero.dtb
-rwxr-xr-x 1 root root   27084  7. Dez 01:31  bcm2708-rpi-zero-w.dtb
-rwxr-xr-x 1 root root   27292  7. Dez 01:31  bcm2709-rpi-2-b.dtb
-rwxr-xr-x 1 root root   27441  7. Dez 01:31  bcm2710-rpi-2-b.dtb
-rwxr-xr-x 1 root root   28939  7. Dez 01:31  bcm2710-rpi-3-b.dtb
-rwxr-xr-x 1 root root   29558  7. Dez 01:31  bcm2710-rpi-3-b-plus.dtb
-rwxr-xr-x 1 root root   27429  7. Dez 01:31  bcm2710-rpi-cm3.dtb
-rwxr-xr-x 1 root root   28100  7. Dez 01:31  bcm2710-rpi-zero-2.dtb
-rwxr-xr-x 1 root root   49877  7. Dez 01:31  bcm2711-rpi-400.dtb
-rwxr-xr-x 1 root root   49833  7. Dez 01:31  bcm2711-rpi-4-b.dtb
-rwxr-xr-x 1 root root   50555  7. Dez 01:31  bcm2711-rpi-cm4.dtb
-rwxr-xr-x 1 root root   52456  7. Dez 01:31  bootcode.bin
-rwxr-xr-x 1 root root     121 28. Sep 21:31  cmdline.txt
-rwxr-xr-x 1 root root    2550 19. Dez 07:28  config.txt
-rwxr-xr-x 1 root root   18693  7. Dez 01:31  COPYING.linux
drwxr-xr-x 4 root root    3584 10. Jan 22:16  dietpi
-rwxr-xr-x 1 root root   18092 20. Dez 2020   dietpi-LICENSE.txt
-rwxr-xr-x 1 root root   13394 20. Dez 2020   dietpi-README.md
-rwxr-xr-x 1 root root   12984 10. Jan 22:16  dietpi.txt
-rwxr-xr-x 1 root root    3213  7. Dez 01:31  fixup4cd.dat
-rwxr-xr-x 1 root root    5411  7. Dez 01:31  fixup4.dat
-rwxr-xr-x 1 root root    8426  7. Dez 01:31  fixup4db.dat
-rwxr-xr-x 1 root root    8420  7. Dez 01:31  fixup4x.dat
-rwxr-xr-x 1 root root    3213  7. Dez 01:31  fixup_cd.dat
-rwxr-xr-x 1 root root    7279  7. Dez 01:31  fixup.dat
-rwxr-xr-x 1 root root   10269  7. Dez 01:31  fixup_db.dat
-rwxr-xr-x 1 root root   10269  7. Dez 01:31  fixup_x.dat
-rwxr-xr-x 1 root root   13312  1. Jan 1980   FSCK0000.REC
-rwxr-xr-x 1 root root   68096  1. Jan 1980   FSCK0001.REC
-rwxr-xr-x 1 root root  629248  1. Jan 1980   FSCK0002.REC
-rwxr-xr-x 1 root root   10752  1. Jan 1980   FSCK0003.REC
-rwxr-xr-x 1 root root   10240  1. Jan 1980   FSCK0004.REC
-rwxr-xr-x 1 root root  658432  1. Jan 1980   FSCK0005.REC
-rwxr-xr-x 1 root root     512  1. Jan 1980   FSCK0006.REC
-rwxr-xr-x 1 root root    2048  1. Jan 1980   FSCK0007.REC
-rwxr-xr-x 1 root root     145 11. Jan 2021   issue.txt
-rwxr-xr-x 1 root root 6355024  7. Dez 01:30  kernel7.img
-rwxr-xr-x 1 root root 6779872  7. Dez 01:30  kernel7l.img
-rwxr-xr-x 1 root root 7906048  7. Dez 01:30  kernel8.img
-rwxr-xr-x 1 root root 6006520  7. Dez 01:30  kernel.img
-rwxr-xr-x 1 root root    1594  7. Dez 01:31  LICENCE.broadcom
drwxr-xr-x 2 root root   20992  7. Dez 01:31  overlays
-rwxr-xr-x 1 root root  801564  7. Dez 01:31  start4cd.elf
-rwxr-xr-x 1 root root 3737928  7. Dez 01:31  start4db.elf
-rwxr-xr-x 1 root root 2241504  7. Dez 01:31  start4.elf
-rwxr-xr-x 1 root root 2993544  7. Dez 01:31  start4x.elf
-rwxr-xr-x 1 root root  801564  7. Dez 01:31  start_cd.elf
-rwxr-xr-x 1 root root 4809800  7. Dez 01:31  start_db.elf
-rwxr-xr-x 1 root root 2965696  7. Dez 01:31  start.elf
-rwxr-xr-x 1 root root 3717256  7. Dez 01:31  start_x.elf
drwxr-xr-x 2 root root     512 24. Jan 2021  'System Volume Information'
12

can you give another try to reboot and check of the volume has been mounted

13

After rebooting /boot ist not mounted, same error as above.

I shut down till tomorrow, too late now, thanks so far

14

ok try to export/save into a file whole output of journalctl

15

Good evening,
i restarted, and got the result of journalctl.
journalctl Pi4.txt (68.5 KB)

16

I see quite a lot the message about Read-only file system. Usually this indicates physical issues on your SD card. It seems RootFS partition is broken. Is following working?

> /forcefsck
reboot
# then after reboot
journalctl -t systemd-fsck

MichaIng
could you have a look if there is still a chance to get the SD card back working.

17

/forcefsck replys with read-only file system
journalctl -t systemd-fsck has no entries.

I have a 3 week old backup of sdcard, no big changes were made since then. I could use this on a new sdcard.
But how can I notice that the sdcard is going to die?

18

Can you manually remount the root drive writeable?

mount -o remount,rw /
> /forcefsck

There are a lot of FSCK remains in /boot, so repairing that one is a good idea as well, which can be done while mounted:

fsck /dev/mmcblk0p1

Since there is no failure or even attempt to mount /boot at boot. What is the content of:

cat /etc/fstab
19

remounting is possible, i created /forcefsck (it is -rw-r–r–)
the fsck of /dev/mmcblk0p1 gives differences

There are differences between boot sector and its backup.
This is mostly harmless. Differences: (offset:original/backup)
  65:01/00
1) Copy original to backup
2) Copy backup to original
3) No action
[123?q]? 2

*** Filesystem was changed ***
The changes have not yet been written, you can still choose to leave the
filesystem unmodified:
1) Write changes
2) Leave filesystem unchanged
[12?q]? 1
/dev/mmcblk0p1: 362 files, 104231/516190 clusters

After rebooting i get same error

-bash: /boot/dietpi/func/dietpi-globals: Datei oder Verzeichnis nicht gefunden
[FAILED] DietPi-Login | Failed to load DietPi-Globals. Skipping DietPi login scripts...

/etc/fstab has 0 bytes, I have a /etc/fstab.org, i will restore that one

20

if /etc/fstab is an empty, it could have been corrupted. That would explain why bootFS was not mounted. Copy back fstab.org might be a good idea