Odroid HC1 backup problems

Troubleshooting
1

Hello @All!

I installed DietPi on Odroid HC1 in order to replace my former ArchLinuxArm installation on Odroid HC1 because ALARM seems to be abadonware for armv7h now.

The Odroid HC1 is supposed to work as PiHole with unbound, local package mirror for linux distro, nfs shares, ntp-server and radicale-server.

All went surprisingly smooth, only some minor issues were encountered.
DietPi is so unbelievable small with just about 900MB and these dietpi-helpers are great!
Even a noob could be able to install and run it.

When installing PiHole with unbound from optimized packages google DNS is selected during install automatically. This was confusing.
Fortunately after all is installed the right DNS-Server (unbound 127:0:0:1:5335) is choosen automatically.
Only some settings are missing / suboptimal (double caching PiHole/unbound, DNSSEC validation forwarding to clients missing, resolv.conf issues)

But all is nothing worth without a working and relieable way to backup and recover your installation!
DietPi offers dietpi-backup - but that does not really work:

I set the AUTO_SETUP_BACKUP_RESTORE to ‘2’ => non-interactive restore (restore first found backup which lies on the connected SSD) but that will not work as the backup process still ‘warns’ about UUIDs and requires confimation to continue.
So no simple headless restore possible. Also it seems that the attached SSD to Odroid HC1 is not triggered when the backup-process searches for backups.

So I tried it on an Odroid XU4 I had fortunately available. There the backup was recovered from USB-stick - but the ownership of system directories were not adopted:
On the rercovered installation unbound did not start as /var/lib/unbound ownership was reset to ‘root’ but ‘unbound’, same for lighttpd in /var/lib/ and complete pi-hole logging.

Practically all ownerships and permissions in system directories were reset to defaults (root:root)! As far as I’ve read this is a known issue also rsync option ‘-p’ is set in the backup-script.

Usually I’am used to backup my former linux installations on SBCs using tar archives - but I found no way to flash the diet-pi bootloader to the SD-card by myself.

The last idea I have is to do a full install (which installs the bootloder too), delete all files and recover my DietPi installation from my tar archive. Not tested if it works till now.
Of course this is not very elegant, costs a lot of time and will not be very good for the sd-card.

If someone knows a way how to flash the DietPi bootloader I would be very thankful. Perhaps some little tool like sd-fusing.sh together with the needed files.

So I could just partition and format the sd-card, flash the bootloader and recover my backup from tar.

Greetings

Nebwie GerryK

2

The last time I did this there was no warning. Indeed there should be a warning when the UUIDs change, but when you backup to the same system from where you made the backup, the UUIDs do not change, so I conclide you are trying to restore a backup which was made from another system?
You could just flash dietpi, go through the installation and then use dietpi-backup manually to restore your backups.

This is recommended by the pihole devs, pihole cache does not interfer with the unbound cache.

I’m not sure what you mean with this. Can you explain futher?

3

This should be done by Unbound directly/by default.

4

Hello and thank you for your quick replies.

I’ve used the instructions for installing pihole + unbound from here (sorry, it is in german).

They suggest to create ‘/etc/dnsmasq.d/10-pihole-extra.conf’ and put ‘proxy-dnssec’ in there to provide DNS validation forwarding to clients.

They also say that having unbound and pihole DNS caching is redundant and superflous.

The idea of auto restore is to prevent having to first install and update everything just to overwrite it afterwards with your backup.

You are right: As the recovery process did not work on the odroid hc1 I had to do it with XU4 wich provides video output (odroid HC1 does not!).

I think cases in which you’ll want to recover your backup to the same system with the same UUIDs is rather rare - in most cases your SD-card has failed and you’ll have to do a full system recovery wich will than not work because of this appearing message about different UUIDs when done with AUTO_SETUP_BACKUP_RESTORE=2.

So it is most likely that you’ll have other UUIDs when doing a recovery of you backup.
Perhaps this message should just have a timeout to prevent failure when that ‘2’ mode?

I’am just looking for a way to do a recovery totally headless on HC1 (which does not have video output) in the way:

1.) write dietpi image to sd-card
2.) set AUTO_SETUP_BACKUP_RESTORE=2 in dietpi.txt
3.) put card in device, power on and wait till the previous installation is fully recovered without any need to interact. And of course permissions and ownerships of files and directories should not be altered.

Have fun!

5

You don’t need to do any of this, you can just use dietpi-software to install pihole and unbound with optimal settings.

OK I got you there. I boot from SSD and do not thought about this case. I only needed to restore when software updates broke something or when I screwed something up :sweat_smile:
In the rare case I have to use another storage device I will take the pill and do a fresh setup with a restore afterwards.
In your case you could also create a complete image of your whole working system and in case of emergency you flash this image to your new device.
But maybe there is still the problem with changed UUIDs, idk how they would interfer with this method :thinking:

Another method would be to set AUTO_SETUP_AUTOMATED=0 and leave AUTO_SETUP_BACKUP_RESTORE=2 so you have to SSH into it and trigger the first boot setup, and when you are in there you can confirm the changed UUIDs.

6

About unbound and pihole cache:

We recommend not changing the cache in Pi-hole regardless of which upstream DNS servers you use.
Multiple caches are not a problem. Browsers, clients, Pi-hole and unbound will all maintain a cache. The first one in the chain with an answer will provide it.

Taken from an answer from an official pihole team staff.

7

Yes, totally no need to follow external guides. DietPi will configure PiHole as well Unbound to work together. On our configuration Unbound will have dnssec enabled by default.

8

First of all thank you for your help and hints.
I understand that pihole and unbound are already optimized and there is no need to configure any further.

But the restoring of a dietpi-backup still fails for me after several more attempts:

The ownership and users of some files are not right, users vanished, services fail to start:

unbound.service                    loaded failed failed  Unbound DNS server
unbound[759]: [759:0] fatal error: could not open autotrust file for writing, /var/lib/unbound/root.key.759-0-63e0c8: Permission denied

/var/lib/unbound/ and /var/lib/unbound/root.key are mapped to user-id ‘973’ / group-id ‘973’ which do both not exist anymore.

Same for /var/lib/ntp which is now owned by user id 87 / group id 87 wich also do not exist.

chown -R unbound:unbound /var/lib/unbound fixed it for the moment and brings dhcp-functionality back after restart of unbound-service.

Seems like there have been some users got lost during backup:
User ID 973, 87, 974 (/var/log/pihole.log and /var/log/pihole-FTL.log - so no logs anymore for pihole), /var/log/ntpstats now belongs to user and group id ‘87’

Of course Pi-Hole could be recovered by using its teleporter function - but to reinstall / reconfigure all the rest (nfs-server, distro package mirror, radicale car/cardDAV-server, mpd…) every time a sd-card might fail again is… a lot of work.

As no one has a way to just flash diet-pis bootloader so I could untar an backup-archive I switched back to archlinuxarm.

9

I guess still an open question. The backup you are trying to restore, on which system it has been done? On the very same SBC? Or another device?

Our backup tool is using rsync and personally I have never seen such behaviour in the last couple of years. There seems to be something special on your device.

10

Well afaik Odroid HC1/HC2/MC1/XU4 are all the same.
The backup was made on HC1 and restord with HC1

I tried to make / restore backup several times now - and result was always issues with ownership / permissions after restoring leading to services not starting.

I’am used to rsync - cloning my installations on x86 devices and regulary backup my personal data with it. I also have never encountered such issues.

As I’am curious I build up an isolated testing environment with only router, pc and Odroid HC1 and will try again to backup / recover a dietpi installation… perhaps with -aAXH rsync options instead of -aH only.

Will report back…

11

Can you have a look into the backup? What permission are set inside the backup for the files in question?

And the system that was used as source to create the backup, is it still available?

12

So I made a fresh backup from a working dietpi with pi-hole and unbound installed and put it to the attached HDD of my Odroid HC1.

Excluded the mountpoint in dietpi-backup settings (/HDD/*) to prevent getting all my music and the package-mirror distro files on it (250GB size!) get backuped too as I just wanted to backup the os.

Installed a fresh dietpi on another SD-card.
Put it in Odroid HC1, finished minimal possible install and than started dietpi-backup.
Software did not find HDD so I’d to mount it manually to /HDD.

Addenum: Perhaps that was the mistake! If I would have mounted it somewhere else the following would not have happened?

BTW:
/mnt/dietpi_userdata is preselected in dietpi-backup.
What sense does it to backup an image on the same device it is installed to as the main intend of a backup is to recover a broken system?
If the sd-card fails the backup is gone too.

Started recovery process from dietpi-backup, choose HDD as source - and guess what:

The first thing dietpi-backup did was to completely erase everything from the HDD - except the dietpi-backup. :man_facepalming:

If I wouldn’t have everything backuped I would anathematize you now!

You should be much more carful with rsyncs ‘delete’ option - or at least give a warning.

The reason is clear:
Because /HDD was excluded rsync deletes everything on it as it ‘thinks’ it was ‘empty’ during backup and now ‘recovers’ this state.
No noob would ever expect this!

So you have to unmount everything from the system to be backuped before starting the procedure and use an USB media as target for the backup - no internal disk with valuable data if you have excluded it from backup.

The good news: This time - beside totally loss of all data on HDD :stuck_out_tongue_winking_eye: - the image recovered without issues and everything worked after boot.

As impressive DietPi and its software is - dietpi-backup is a little bit half-baked.

Using AUTO_SETUP_BACKUP_RESTORE leads to the problems described, ownership and userrights are lost due to dissapearing users (ntp, pihole, unbound, www-data).

And if you have some storage mediums mounted which you excluded during backup they will be erased during recovery.

Don’t get me wrong: It was fun and I’ve learned a lot.
Hope my report could help to make DietPi better.

Fortunately yes.

I’ve non of the broken backups recovered on sd-card anymore - just the dietpi-backup as directory on the HDD where everything else was deleted.

13

on DietPi you should mount your HDD to /mnt/whatever. This way the HDD would stay untouched. Because a restore means to restore everything and to delete stuff not being part of the backup. Means, all files will be removed not being stored inside the backup folder. We do full system backups and restores. Except mount point located within /mnt

This is a default value and can be adjusted on user demand. You could easily specify where your backup should be done to.

This mount point has no influence on the restore done during initial boot process.

14

I had /HDD/* excluded during backup like /mnt/* is.
So content in /HDD was not backed up by dietpi-backup (which was desired).

During recovery of backup from /HDD rsync deleted its data because is was considered as ‘empty’ (excluded) during backup. (which was not desired!)

Do I get it right that this would not’ve happened if I had mounted it to /mnt/HDD and recovered dietpi-backup from /mnt/HDD?

That was my /boot/dietpi/.dietpi-backup_inc_exc:

+ /mnt/dietpi_userdata/
- /mnt/*
- /media/
- /HDD/*

I’ve now mounted my HDD to /mnt/HDD and will try again…

15

[rant] deleted 12345

16

@MichaIng could you please have a look. Thx

17

I did a test backup and restore on my x86-system.
df- Th gives me (I deleted irrelevant paths from the output):

/dev/sda1      ext4      147G    4,3G  136G    4% /
/dev/sdb1      ext4      113G     24K  113G    1% /mnt/bup
/dev/sdc1      vfat       15G    610M   14G    5% /mnt/data
  • sda1 is the system partition
  • sdb1 is the flash drive where the backup get’s saved (/mnt/bup/dietpi-backup)
  • sdc1 is a flash drive with some test data to see if it gets deleted.

The backup filter is at default settings:

+ /mnt/dietpi_userdata/
- /mnt/*
- /media/

After the backup I look into the backup folder to make sure the /mnt/data path is not included:

dietpi@TestPi:~$ ls -la /mnt/bup/dietpi-backup/data/mnt/
insgesamt 12
drwxr-xr-x  3 root   root   4096 20. Nov 21:29 .
drwxr-xr-x 11 root   root   4096 20. Nov 21:43 ..
drwxrwxr-x  7 dietpi dietpi 4096 16. Aug 02:38 dietpi_userdata

Ok, this looks fine, df -Th gives me now:

/dev/sda1      ext4      147G    4,3G  136G    4% /
/dev/sdb1      ext4      113G    4,3G  108G    4% /mnt/bup
/dev/sdc1      vfat       15G    610M   14G    5% /mnt/data

Now I did the restore, which was really fast since nothing on the system was changed. This is the log:

 Restore log from 2022-11-20_21:49:02                                         │
│                                                                              │
│ 2022/11/20 21:49:02 [4309] building file list                                │
│ 2022/11/20 21:49:03 [4309] done                                              │
│ 2022/11/20 21:49:03 [4309] .d..t...... ./                                    │
│ 2022/11/20 21:49:04 [4309] >f.st...... root/.xsession-errors                 │
│ 2022/11/20 21:49:05 [4309] sent 2,434,382 bytes  received 43 bytes           │
│ 973,770.00 bytes/sec                                                         │
│ 2022/11/20 21:49:05 [4309] total size is 4,677,209,196  speedup is 1,921.28

I did a reboot, df -Th gives me now:

/dev/sda1      ext4      147G    4,3G  136G    4% /
/dev/sdc1      ext4      113G    4,3G  108G    4% /mnt/bup
/dev/sdb1      vfat       15G    610M   14G    5% /mnt/data

The data on /mnt/data is still there.

I also did restores in the past on my RPi systems, which always worked fine, without messing up permissions and ownerships.

So I’m not sure what caused your issues. Maybe you should have used - /HDD/ instead of the wildcard entry - /HDD/* like it is done with /media/. The wildcard exclude rule for - /mnt/* only exists because there is an include rule for /mnt/dietpi_userdata/, I think.

And at your second try (the deleted post) is it possible that you didn’t remove your wildcard exclude rule from the filter and this also messed something up again (because /HDD/* would be part of /mnt/HDD/ if it is handled as a wildcard, maybe?)
As you can see, I didn’t mess with the filters and mounted my data in /mnt/data and nothing was touched/deleted by the restore process.

18

I did some more testing today.
I tried to mount the test data flash drive to /HDD, but when I’m trying this I get

───────────────────────────┤ DietPi-Drive_Manager ├───────────────────────────┐
│                                                                              │
│ Invalid mount target location:                                               │
│  - /HDD                                                                      │
│                                                                              │
│ The drive will now be mounted to:                                            │
│  - /mnt/1E0E-BDA0                                                            │
│

So I conclude you bypassed the dietpi tools and mounted it manually. But then you can not expect other dietpi tools will work with your manually changed stuff.

I also did a backup with a modified filter (I reverted the test data mount to /mnt/data), I added
- /data/* and in another try - /mnt/data/* (I tried to recreate your scenario), but I have same results as in the first try. Nothing was deleted in ´/mnt/data`.

Which filesystem does your backup hard disk use?

19

Try to use:

- /HDD/

to exclude the whole directory instead of its content only.

But actually both should work.

I’ll try to replicate with mount at /HDD and - /HDD/* exclude entry, as @Jappe this is the only case you didn’t try yet, isn’t it?

20

Worked as expected :thinking::

root@VM-Bullseye:~# mkdir /HDD
root@VM-Bullseye:~# mount /dev/sdb1 /HDD
root@VM-Bullseye:~# > /HDD/testfile
root@VM-Bullseye:~# echo '- /HDD/*' >> /boot/dietpi/.dietpi-backup_inc_exc
root@VM-Bullseye:~# dietpi-backup 1
...
[  OK  ] DietPi-Backup: Backup | Completed
root@VM-Bullseye:~# l /HDD/
total 16K
drwx------ 2 root root 16K Oct 23 19:40 lost+found
-rw-r--r-- 1 root root   0 Nov 21 18:00 testfile
root@VM-Bullseye:~# l /mnt/dietpi-backup/data/HDD/
total 0
root@VM-Bullseye:~# dietpi-backup -1
...
[  OK  ] DietPi-Backup: Restore | Completed
root@VM-Bullseye:~# l /HDD/
total 16K
drwx------ 2 root root 16K Oct 23 19:40 lost+found
-rw-r--r-- 1 root root   0 Nov 21 18:00 testfile