Services Failing after Swapping Backup Drives

cdlenfert · 21 February 2020 19:09

This is admittedly some user error going on, but I’m hoping I can recover from it smoothly.

Right now my MariaDB service fails to run

root@DietPi:~# systemctl status mariadb
● mariadb.service - MariaDB 10.1.44 database server
   Loaded: loaded (/lib/systemd/system/mariadb.service; disabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Fri 2020-02-21 11:22:02 MST; 10min ag
o
     Docs: man:mysqld(8)
           https://mariadb.com/kb/en/library/systemd/
  Process: 14447 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_P
OSITION (code=exited, status=1/FAILURE)
  Process: 14355 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=
`/usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemctl set-environment _WSREP_START_POSITIO
N=$VAR || exit 1 (code=exited, status=0/SUCCESS)
  Process: 14351 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (
code=exited, status=0/SUCCESS)
  Process: 14348 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (c
ode=exited, status=0/SUCCESS)
 Main PID: 14447 (code=exited, status=1/FAILURE)

Feb 21 11:21:32 DietPi systemd[1]: Starting MariaDB 10.1.44 database server...
Feb 21 11:21:33 DietPi mysqld[14447]: 2020-02-21 11:21:33 1995419440 [Note] /usr/sbin/mysqld
 (mysqld 10.1.44-MariaDB-0+deb9u1) starting as process 14447 ...
Feb 21 11:22:02 DietPi systemd[1]: mariadb.service: Main process exited, code=exited
, status=1/FAILURE
Feb 21 11:22:02 DietPi systemd[1]: Failed to start MariaDB 10.1.44 database server.

Feb 21 11:22:02 DietPi systemd[1]: mariadb.service: Unit entered failed state.
Feb 21 11:22:02 DietPi systemd[1]: mariadb.service: Failed with result 'exit-code'.

I noticed there were some errors related to being able to write to the /var/lib/mysql directory

Feb 21 11:14:22 DietPi systemd[1]: Starting MariaDB 10.1.44 database server...
Feb 21 11:14:23 DietPi mysqld[13800]: 2020-02-21 11:14:23 1996009264 [Note] /usr/sbin/mysqld (mysqld 10.1.44-MariaDB-0+deb9u1) starting as process 13800 ...
Feb 21 11:14:23 DietPi mysqld[13800]: 2020-02-21 11:14:23 1996009264 [Warning] Can't create test file /var/lib/mysql/DietPi.lower-test
Feb 21 11:14:23 DietPi mysqld[13800]: [97B blob data]
Feb 21 11:14:23 DietPi mysqld[13800]: 2020-02-21 11:14:23 1996009264 [ERROR] Aborting
Feb 21 11:14:23 DietPi systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Feb 21 11:14:23 DietPi systemd[1]: Failed to start MariaDB 10.1.44 database server.

When I look in that directory I can see that /var/lib/mysql is symlinked to /mnt/dietpi_userdata/mysql

root@DietPi:/var/lib# ls -l
total 84
drwxr-xr-x 2 root   root   4096 Feb 20 20:04 alsa
drwxr-xr-x 5 root   root   4096 Feb 14 15:53 apt
drwxr-xr-x 2 root   root   4096 Mar  4  2019 dhcp
drwxr-xr-x 2 root   root   4096 Sep 10  2018 dhcpcd5
drw-rw---- 8 dietpi dietpi 4096 Mar 10  2019 dietpi
drwxr-xr-x 7 root   root   4096 Feb 14 16:00 dpkg
drwxr-xr-x 2 root   root   4096 Jul 24  2019 fail2ban
drwxr-xr-x 2 root   root   4096 Sep 27  2018 git
drwxr-xr-x 2 root   root   4096 Mar 12  2018 misc
lrwxrwxrwx 1 root   root     26 Mar 10  2019 mysql -> /mnt/dietpi_userdata/mysql
lrwxrwxrwx 1 root   root     26 Jan 21 13:06 mysql.bak -> /mnt/dietpi_userdata/mysql
drwxr-xr-x 2 root   root   4096 Nov 13  2018 ntp
drwxr-xr-x 2 root   root   4096 Mar  4  2019 pam
drwxr-xr-x 4 root   root   4096 Mar 10  2019 php
drwxr-xr-x 3 plex   plex   4096 May 22  2019 plexmediaserver
drwxr-xr-x 2 root   root   4096 Jul 24  2019 python
drwxr-x--- 2 redis  redis  4096 Feb 21 11:14 redis
drwxr-xr-x 2 root   root   4096 Mar  4  2019 resolvconf
drwxr-xr-x 3 root   root   4096 Mar 10  2019 samba
drwx--x--x 4 root   root   4096 Nov 13  2018 sudo
drwxr-xr-x 7 root   root   4096 Oct 24 12:27 systemd
drwxr-xr-x 3 root   root   4096 Oct 24 12:24 ucf
drwxr-xr-x 2 root   root   4096 Oct 24 12:16 usbutils

However the /mnt/dietpi_userdata/mysql directory is empty.

root@DietPi:/mnt/dietpi_userdata# ls -l
total 0

This isn’t really surprising to me since all of my userdata is stored on an external USB drive. However something about the setup of where userdata is stored got messed up recently on my setup. When I review the drives available in dietpi drive manager it shows my rootfs (sd card) as the location of my userdata, and if I try to change this to my USB drive, it appears to work, but when I run drive manager again, the setting is not preserved.

Is there a way I can force dietpi to use my external drive as my userdata location?

I also tried manually defining the mysql data location in /etc/mysql/mariadb.conf.d/50-server.cnf but that just lead to mariadb still failing, just without any helpful errors I could see.

Some other info about my drives and mount points that could be helpful:

root@DietPi:/boot# ls -lah /mnt
total 40K
drwxr-xr-x 10 root root 4.0K Feb 15 15:39 .
drwxr-xr-x 22 root root 4.0K Mar 16  2019 ..
drwxr-xr-x  3 root root 4.0K May 13  2019 0827b932-2eb4-46ba-aab0-e7525d6af812
drwxr-xr-x  4 root root 4.0K Feb 20 19:44 85fb25d2-7a10-4e41-bbc1-57477df55a4b
drwxr-xr-x  2 root root 4.0K Feb 15 15:39 dietpi_userdata
drwxr-xr-x  2 root root 4.0K Mar  4  2019 ftp_client
drwxr-xr-x  2 root root 4.0K Mar  4  2019 nfs_client
drwxr-xr-x  2 root root 4.0K Mar 10  2019 pogo1
drwxr-xr-x  2 root root 4.0K Mar 10  2019 samba
drwxr-xr-x  2 root root 4.0K Mar 19  2019 SSD128GB

root@DietPi:/boot# cat /etc/fstab 
# Please use "dietpi-drive_manager" to setup mounts
#----------------------------------------------------------------
# NETWORK
#----------------------------------------------------------------
//192.168.0.8/PogoPlug-2 /pogo1 cifs username=guest,password=,iocharset=utf8,uid=dietpi,gid=dietpi,file_mode=0770,dir_mode=0770,vers=3.0,_netdev,nofail,x-systemd.automount 0 0

#----------------------------------------------------------------
# TMPFS
#----------------------------------------------------------------
tmpfs /tmp tmpfs size=512M,noatime,lazytime,nodev,nosuid,mode=1777
tmpfs /DietPi tmpfs defaults,size=10m,noatime,nodev,nosuid,mode=1777 0 0
tmpfs /var/log tmpfs defaults,size=50m,noatime,nodev,nosuid,mode=1777 0 0

#----------------------------------------------------------------
# MISC: ecryptfs, vboxsf (VirtualBox shared folder), bind mounts
#----------------------------------------------------------------


#----------------------------------------------------------------
# SWAPFILE
#----------------------------------------------------------------
/var/swap none swap sw 0 0

#----------------------------------------------------------------
# PHYSICAL DRIVES
#----------------------------------------------------------------
PARTUUID=b8ef7a27-02 / ext4 noatime,lazytime,rw 0 1
PARTUUID=b8ef7a27-01 /boot vfat noatime,lazytime,rw 0 2
UUID=85fb25d2-7a10-4e41-bbc1-57477df55a4b /mnt/85fb25d2-7a10-4e41-bbc1-57477df55a4b ext4 noatime,lazytime,rw,nofail,noauto,x-systemd.automount

Should I manually transfer my userdata folder from the USB drive to the SD card and then use dietpi drive manger to reformat the USB and reassign the userdata from SD back to the USB?

Joulinar · 21 February 2020 20:52

Hi,

first I would recommend to move everything back to /mnt/dietpi_userdata/ on your SD card and try to restart MariaDB. If it still fails, let’s have this fixed before trying to move it on USB again.

cdlenfert · 22 February 2020 05:10

Thanks for your response. I’ve moved everything back to /mnt/dietpi_userdata.

When trying to restart the service I get this:

root@DietPi:/mnt/dietpi_userdata/mysql/mysql# journalctl -n 100
Feb 21 21:50:12 DietPi systemd[1]: Starting MariaDB 10.1.44 database server...
Feb 21 21:50:13 DietPi mysqld[24103]: 2020-02-21 21:50:13 1995718448 [Note] /usr/sbin/mysqld (mysqld 10.1.44-MariaDB-0+deb9u1) starting a
s process 24103 ...
Feb 21 21:50:13 DietPi mysqld[24103]: 2020-02-21 21:50:13 1995718448 [Warning] Can't create test file /var/lib/mysql/DietPi.lower-test
Feb 21 21:50:13 DietPi mysqld[24103]: [90B blob data]
Feb 21 21:50:13 DietPi mysqld[24103]: 2020-02-21 21:50:13 1995718448 [ERROR] Aborting
Feb 21 21:50:13 DietPi systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Feb 21 21:50:13 DietPi systemd[1]: Failed to start MariaDB 10.1.44 database server.
Feb 21 21:50:13 DietPi systemd[1]: mariadb.service: Unit entered failed state.
Feb 21 21:50:13 DietPi systemd[1]: mariadb.service: Failed with result 'exit-code'.

The warning about not being able to create the test file went away when I tested earlier, but now it’s back. I’m pretty sure it’s a permissions issue (at least in part). The following is what I had after the move:

root@DietPi:/mnt/dietpi_userdata# ls -l
total 36
drwxr-xr-x 2 root root 4096 Feb 21 12:54 downloads
drwxr-xr-x 2 root root 4096 Feb 21 12:56 Music
drwxr-x--- 5 root root 4096 Feb 21 13:43 mysql
drwxr-x--- 6 root root 4096 Feb 21 12:57 nextcloud_data
drwxr-xr-x 3 root root 4096 Feb 21 12:56 php_bak
drwxr-xr-x 2 root root 4096 Feb 21 12:54 Pictures
drwx------ 3 root root 4096 Feb 21 12:58 syncthing
drwxr-xr-x 2 root root 4096 Feb 21 12:57 syncthing_data
drwxr-xr-x 2 root root 4096 Feb 21 12:54 Video

I did another rsync preserving permissions this time and now have:

root@DietPi:/mnt/dietpi_userdata# ls -l
total 36
drwxrwxr-x 2 dietpi   dietpi   4096 Mar 10  2019 downloads
drwxrwxr-x 2 dietpi   dietpi   4096 Mar 15  2019 Music
drwxrwx--- 5 mysql    mysql    4096 Feb 15 15:39 mysql
drwxrwx--- 6 www-data www-data 4096 Mar 10  2019 nextcloud_data
drwxrwxr-x 3 dietpi   dietpi   4096 Mar 10  2019 php_bak
drwxrwxr-x 2 dietpi   dietpi   4096 Mar 10  2019 Pictures
drwx------ 3 dietpi   dietpi   4096 Feb 15 12:11 syncthing
drwxr-xr-x 2 dietpi   dietpi   4096 Feb 15 12:11 syncthing_data
drwxrwxr-x 2 dietpi   dietpi   4096 Mar 10  2019 Video

After that I also was able to start Mariadb. I also see that Nextcloud is in maintenance mode (do I let that resolve itself or do I need to disable it? - I did remove about 20GB of failed uploads that were in my userdata folder to make the restore to SD possible). Should I consider moving to the USB now? My real goal is to get to a more stable system running off a USB SSD as I mention here: https://dietpi.com/forum/t/request-recommendations-on-hardware-setup-and-restore-options/3849/1

Joulinar · 22 February 2020 08:03

Hi,

to disable NextCloud Maintenance Mode, just run the following

ncc maintenance:mode --off

once done, pls use dietpi drive manager to relocate whole /mnt/dietpi_userdata. There is an option inside drive manager to mark your USB device as user data location. This as well should copy the data.

cdlenfert · 22 February 2020 16:17

This all worked well. Thank you! I’d like to only use my USB SSD for roots and user data, or possibly use SD for only booting. My hope would be to gain performance and stability. Right now my Plex database is on the SD and that can’t be great. Not sure what my path forward would look like to retain my data and migrate to an SSD based setup.

Joulinar · 22 February 2020 23:54

yep thats possible. You can use the SD card to host the boot partition and your USB device will contain the RootFS (incl user data)

Steps to be done:

copy all your user data back to SD card using dietpi-drive_manager, to get the original status
un-mount the USB device
do a reboot to check if all working fine
do a full backup of your SD card
boot your system up, it should now running on SD card fully, while USB device is not mounted
go to dietpi-drive_manager
select the unmounted USB device
select Transfer RootFS
now read the following instruction carefully and confirm
format your USB device
last chance to cancel
once done, you will be ask to select a mount point like /mnt/usb. This is just temporary
as soon as the device is mounted, Drive Manager will stop all processes and start to copy your entire RootFS
time for a coffee
you will be ask to reboot your device once completed
during reboot you might see some services failing like MariaDB. This is happening because Drive Manager missed to copy /mnt/dietpi_userdata
don’t worry, let’s fix it
go to dietpi-drive_manager
select the unused partition /dev/mmcblk0p2 on your SD card
mount the partition as /mnt/sd2
exit Drive Manager
change directory to cd /mnt/sd2/mnt
let’s copy the missing data cp -p -r * /mnt/
reboot your system
now everything should working fine

If you encounter any issues on this, restore your backup to SD card

MichaIng
Is there a reason why Drive Manager is not copying dietpi_userdata during RootFS relocation?

MichaIng · 23 February 2020 13:41

Not really, probably because it is assumed to be on external drive. However as well the rsync does not exclude all tmpfs, most importantly /DietPi is synced as well.

I’ll clean this up by mounting the root partition to a temporary mount point and rsync the files from there. AFAIK rsync itself has an option to sync only files from the same drives, probably one to skip mounts automatically even when from same drive e.g. bind mounts. Much cleaner than excluding only known mounts and /mnt/* (which includes dietpi_userdata).

Joulinar · 23 February 2020 13:46

on my test, /mnt/ themselves was copied to my USB stick but without sub directories /mnt/*. Could that be because I mounted the USB device to /mnt/ before?? At least this was the offer of Drive Manager. Maybe I will some test later the day to have the USB device mounted somewhere else before moving RootFS

cdlenfert · 23 February 2020 18:06

Thanks @Joulinar. I’m glad to give this a try, but I’m curious about a couple of things I hope you might clarify.

“let’s copy the missing data cp -p -r dietpi_userdata /mnt/” - so the USB SSD will exist at / and the default location for userdata becomes /mnt/ right?
Right now I’m using a 32GB SD card for boot/rootfs and it seems like it would be a waste just for booting the Pi 3b. Would it be reasonable/possible to also have boot from the USB? If so, what does that process look like vs. the one you so nicely outlined? and I understand a one time setting change is required on the Pi, but I don’t understand if that means it will only be bootable from USB afterward, or if SD boot is still possible? What are other consequences of making the USB boot setting change?
as @MichaIng mentions “However as well the rsync does not exclude all tmpfs, most importantly /DietPi is synced as well.” - is there something I should do differently from the instrucitons to not sync more than what is needed? such as wait for a DietPi update with the new “transfer rootfs” script?

MichaIng · 23 February 2020 18:54

Following the instructions from Joulinar will work. Yes you can also boot the RPi 3 from USB. Is it 3+ or non-plus model?

cdlenfert · 23 February 2020 19:10

Cool. I have the non plus model. Just 3b

Joulinar · 23 February 2020 21:40

some USB Boot information and how to activate

https://www.raspberrypi.org/documentation/hardware/raspberrypi/bootmodes/msd.md

btw you could move boot partition to a much smaler SD card as well

cdlenfert · 24 February 2020 18:23

Thanks. I had read through that article on USB booting, but what I still don’t understand is if I set the one time programmable memory to enable booting from USB, can I still boot from SD if I wanted to (not saying that I would). I’m a little spooked by the OTP aspect since I have no experience with this sort of thing. I’m guessing it just allows USB boot, but does not kill SD boot capabilities.

Also if I plan to have everything on my USB SSD (boot and rootfs) does the process of getting there look something like this?

add program_usb_boot_mode=1 to the end of /boot/config.txt
reboot
copy userdata back to my SD card so everything is on the SD
create an image of my SD card on my Mac
burn the image onto my USB SSD
plug the USB SSD into my Pi 3b, boot
done!?

Will there be any issues with the SD and USB drive UUIDs being different? I would also plan to remove the old USB SSD from the Drive manager and FSTAB entries before trying to use it as the boot/rootfs drive (it was previously my backup and userdata drive).

Last question, can I use a samba shared drive as the location for my dietpi backups, or is that not possible because of permissions/ownership limitations for network shares?

Joulinar · 24 February 2020 23:16

unfortunately I’m not able to answer the question as I have a 3B+ only where USB boot is working ootb.
But I would expect to have both working once activated. But it might be better to verify and have this question raised on RPi Board. Just in case.

Yep if you swap from SD to USB, you need to adjust UUIDs within /etc/fstab

Recommendation would be to start with a clean install in our SSD to avoid issues or side effects

cdlenfert · 24 February 2020 23:59

I found some answers in the Raspberry forums here - https://www.raspberrypi.org/forums/viewtopic.php?t=175630

Looks like booting from SD is still a possibility, so I’ll plan to move forward with setting the OTP memory to allow USB booting on my Pi 3b.

I wouldn’t have a problem updating UUIDs in the /etc/fstab, but since you mention it would be recommended to start from a clean install, would that mean reinstalling and reconfiguring every program I’m running? Is there no clean way to keep what I’ve got? Would restoring from a backup to a fresh install on the USB SSD be better than trying to migrated to the USB SSD from the SD?

I’m currently running:

Plex
Nextcloud
PiHole
VPN

Joulinar · 25 February 2020 00:11

well you could give it a try to change UUIDs in the /etc/fstab. if it’s not working, you still would have the SD as backup.

cdlenfert · 25 February 2020 01:27

Sounds good. I’ll try it out and post the results. Appreciate all the help!

cdlenfert · 25 February 2020 20:59

Here’s how it all went down.

added program_usb_boot_mode=1 to the end of /boot/config.txt
rebooted
copied userdata back to my SD card so everything is on the SD
copied userdata back to my SD card so everything is on the SD
created an image of my SD card on my Mac
burned the image onto my USB SSD
reviewed /etc/fstab - since I burned the SD images to the USB, the PARTUUID numbers were the same - no change required
pluged the USB SSD into my Pi 3b and powered on

Unfortunately nothing happened after that. The Pi didn’t boot off the USB at all and the lights on the USB>SSD adapter never even blinked.

I’m not sure why. I know some SSDs are not compatible, but everything here seemed to be set up properly. More info on this further down.

Thinking I’d go for the next best setup I did the following:

removed the USB SSD
inserted the SD card
booted just fine
formatted the USB SSD
used Drive-Manager to transfer RootFS to the USB SSD
manually transferred user-data from the SD ext4 partition to the USB SSD
rebooted and everything works

I somewhat foolisly didn’t run vcgencmd otp_dump | grep 17: after setting the USB boot option in config.txt, because I did check the config.txt file and saw the option was appended properly before rebooting. However now when I run the vcgencmd command it doesn’t give me the output of 3020000a and instead outputs 1020000a, indicating that the USB boot option failed to be set in the OTP memory. If that’s the case, that explains one reason why USB booting would have failed. When I look at /boot/config.txt the program_usb_boot_mode=1 is no longer there . I suppose I could do the following to try it again.

make a new backup of the boot only SD card
restore the old full OS SD card image to the SD card
boot off the SD card
try to set the USB boot parameter again
confirm it is set
copy RootFS to the USB SSD again
try booting again

I also have a 1GB SD card that I’d be fine using for the boot drive. How would I go about copying the current boot drive over to it? I assume I’d again need to change FSTAB (or maybe I could create an image of the current boot SD which should be small now, and restore it to a smaller SD … or would it image the full 32GB of my card. I’ve used PiShrink before to get image sizes down and downgrade cards, but not sure that works with a boot partition only on Dietpi.) Maybe putting both SD cards in as external storage on a different Pi would allow me to rsync the boot drive and then just update FSTAB on the USB SSD to use the new SD card’s boot partition UUID. Decisions decisions.

Joulinar · 25 February 2020 21:09

Hi,

I guess this is what you need to move boot fs to a smaller SD card

https://dietpi.com/forum/t/move-boot-partition-to-smaller-sd-card/3687/7

cdlenfert · 27 February 2020 03:08

Quoting myself here:

I somewhat foolisly didn’t run vcgencmd otp_dump | grep 17: after setting the USB boot option in config.txt, because I did check the config.txt file and saw the option was appended properly before rebooting. However now when I run the vcgencmd command it doesn’t give me the output of 3020000a and instead outputs 1020000a, indicating that the USB boot option failed to be set in the OTP memory. If that’s the case, that explains one reason why USB booting would have failed. When I look at /boot/config.txt the program_usb_boot_mode=1 is no longer there >

I didn’t realize the USB boot option should be set in DietPi-Config > Advanced Options. I had tried editing /boot/config.txt as mentioned in the Raspberry Pi forum post. Once I used the DietPi-Config option and rebooted the USB option was set successfully. Now to try again and see if USB boot can work.