Services Failing after Swapping Backup Drives

Having issues with your DietPi installation, or, found a bug? Post it here.
cdlenfert
Posts: 28
Joined: Fri Mar 29, 2019 5:42 pm

Services Failing after Swapping Backup Drives

Post by cdlenfert »

This is admittedly some user error going on, but I'm hoping I can recover from it smoothly.

Right now my MariaDB service fails to run

Code: Select all

root@DietPi:~# systemctl status mariadb
● mariadb.service - MariaDB 10.1.44 database server
   Loaded: loaded (/lib/systemd/system/mariadb.service; disabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Fri 2020-02-21 11:22:02 MST; 10min ag
o
     Docs: man:mysqld(8)
           https://mariadb.com/kb/en/library/systemd/
  Process: 14447 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_P
OSITION (code=exited, status=1/FAILURE)
  Process: 14355 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=
`/usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemctl set-environment _WSREP_START_POSITIO
N=$VAR || exit 1 (code=exited, status=0/SUCCESS)
  Process: 14351 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (
code=exited, status=0/SUCCESS)
  Process: 14348 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (c
ode=exited, status=0/SUCCESS)
 Main PID: 14447 (code=exited, status=1/FAILURE)

Feb 21 11:21:32 DietPi systemd[1]: Starting MariaDB 10.1.44 database server...
Feb 21 11:21:33 DietPi mysqld[14447]: 2020-02-21 11:21:33 1995419440 [Note] /usr/sbin/mysqld
 (mysqld 10.1.44-MariaDB-0+deb9u1) starting as process 14447 ...
Feb 21 11:22:02 DietPi systemd[1]: mariadb.service: Main process exited, code=exited
, status=1/FAILURE
Feb 21 11:22:02 DietPi systemd[1]: Failed to start MariaDB 10.1.44 database server.

Feb 21 11:22:02 DietPi systemd[1]: mariadb.service: Unit entered failed state.
Feb 21 11:22:02 DietPi systemd[1]: mariadb.service: Failed with result 'exit-code'.

I noticed there were some errors related to being able to write to the /var/lib/mysql directory

Code: Select all

Feb 21 11:14:22 DietPi systemd[1]: Starting MariaDB 10.1.44 database server...
Feb 21 11:14:23 DietPi mysqld[13800]: 2020-02-21 11:14:23 1996009264 [Note] /usr/sbin/mysqld (mysqld 10.1.44-MariaDB-0+deb9u1) starting as process 13800 ...
Feb 21 11:14:23 DietPi mysqld[13800]: 2020-02-21 11:14:23 1996009264 [Warning] Can't create test file /var/lib/mysql/DietPi.lower-test
Feb 21 11:14:23 DietPi mysqld[13800]: [97B blob data]
Feb 21 11:14:23 DietPi mysqld[13800]: 2020-02-21 11:14:23 1996009264 [ERROR] Aborting
Feb 21 11:14:23 DietPi systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Feb 21 11:14:23 DietPi systemd[1]: Failed to start MariaDB 10.1.44 database server.
When I look in that directory I can see that /var/lib/mysql is symlinked to /mnt/dietpi_userdata/mysql

Code: Select all

root@DietPi:/var/lib# ls -l
total 84
drwxr-xr-x 2 root   root   4096 Feb 20 20:04 alsa
drwxr-xr-x 5 root   root   4096 Feb 14 15:53 apt
drwxr-xr-x 2 root   root   4096 Mar  4  2019 dhcp
drwxr-xr-x 2 root   root   4096 Sep 10  2018 dhcpcd5
drw-rw---- 8 dietpi dietpi 4096 Mar 10  2019 dietpi
drwxr-xr-x 7 root   root   4096 Feb 14 16:00 dpkg
drwxr-xr-x 2 root   root   4096 Jul 24  2019 fail2ban
drwxr-xr-x 2 root   root   4096 Sep 27  2018 git
drwxr-xr-x 2 root   root   4096 Mar 12  2018 misc
lrwxrwxrwx 1 root   root     26 Mar 10  2019 mysql -> /mnt/dietpi_userdata/mysql
lrwxrwxrwx 1 root   root     26 Jan 21 13:06 mysql.bak -> /mnt/dietpi_userdata/mysql
drwxr-xr-x 2 root   root   4096 Nov 13  2018 ntp
drwxr-xr-x 2 root   root   4096 Mar  4  2019 pam
drwxr-xr-x 4 root   root   4096 Mar 10  2019 php
drwxr-xr-x 3 plex   plex   4096 May 22  2019 plexmediaserver
drwxr-xr-x 2 root   root   4096 Jul 24  2019 python
drwxr-x--- 2 redis  redis  4096 Feb 21 11:14 redis
drwxr-xr-x 2 root   root   4096 Mar  4  2019 resolvconf
drwxr-xr-x 3 root   root   4096 Mar 10  2019 samba
drwx--x--x 4 root   root   4096 Nov 13  2018 sudo
drwxr-xr-x 7 root   root   4096 Oct 24 12:27 systemd
drwxr-xr-x 3 root   root   4096 Oct 24 12:24 ucf
drwxr-xr-x 2 root   root   4096 Oct 24 12:16 usbutils
However the /mnt/dietpi_userdata/mysql directory is empty.

Code: Select all

root@DietPi:/mnt/dietpi_userdata# ls -l
total 0
This isn't really surprising to me since all of my userdata is stored on an external USB drive. However something about the setup of where userdata is stored got messed up recently on my setup. When I review the drives available in dietpi drive manager it shows my rootfs (sd card) as the location of my userdata, and if I try to change this to my USB drive, it appears to work, but when I run drive manager again, the setting is not preserved.

Is there a way I can force dietpi to use my external drive as my userdata location?

I also tried manually defining the mysql data location in /etc/mysql/mariadb.conf.d/50-server.cnf but that just lead to mariadb still failing, just without any helpful errors I could see.

Some other info about my drives and mount points that could be helpful:

Code: Select all

root@DietPi:/boot# ls -lah /mnt
total 40K
drwxr-xr-x 10 root root 4.0K Feb 15 15:39 .
drwxr-xr-x 22 root root 4.0K Mar 16  2019 ..
drwxr-xr-x  3 root root 4.0K May 13  2019 0827b932-2eb4-46ba-aab0-e7525d6af812
drwxr-xr-x  4 root root 4.0K Feb 20 19:44 85fb25d2-7a10-4e41-bbc1-57477df55a4b
drwxr-xr-x  2 root root 4.0K Feb 15 15:39 dietpi_userdata
drwxr-xr-x  2 root root 4.0K Mar  4  2019 ftp_client
drwxr-xr-x  2 root root 4.0K Mar  4  2019 nfs_client
drwxr-xr-x  2 root root 4.0K Mar 10  2019 pogo1
drwxr-xr-x  2 root root 4.0K Mar 10  2019 samba
drwxr-xr-x  2 root root 4.0K Mar 19  2019 SSD128GB

Code: Select all

root@DietPi:/boot# cat /etc/fstab 
# Please use "dietpi-drive_manager" to setup mounts
#----------------------------------------------------------------
# NETWORK
#----------------------------------------------------------------
//192.168.0.8/PogoPlug-2 /pogo1 cifs username=guest,password=,iocharset=utf8,uid=dietpi,gid=dietpi,file_mode=0770,dir_mode=0770,vers=3.0,_netdev,nofail,x-systemd.automount 0 0

#----------------------------------------------------------------
# TMPFS
#----------------------------------------------------------------
tmpfs /tmp tmpfs size=512M,noatime,lazytime,nodev,nosuid,mode=1777
tmpfs /DietPi tmpfs defaults,size=10m,noatime,nodev,nosuid,mode=1777 0 0
tmpfs /var/log tmpfs defaults,size=50m,noatime,nodev,nosuid,mode=1777 0 0

#----------------------------------------------------------------
# MISC: ecryptfs, vboxsf (VirtualBox shared folder), bind mounts
#----------------------------------------------------------------


#----------------------------------------------------------------
# SWAPFILE
#----------------------------------------------------------------
/var/swap none swap sw 0 0

#----------------------------------------------------------------
# PHYSICAL DRIVES
#----------------------------------------------------------------
PARTUUID=b8ef7a27-02 / ext4 noatime,lazytime,rw 0 1
PARTUUID=b8ef7a27-01 /boot vfat noatime,lazytime,rw 0 2
UUID=85fb25d2-7a10-4e41-bbc1-57477df55a4b /mnt/85fb25d2-7a10-4e41-bbc1-57477df55a4b ext4 noatime,lazytime,rw,nofail,noauto,x-systemd.automount
Should I manually transfer my userdata folder from the USB drive to the SD card and then use dietpi drive manger to reformat the USB and reassign the userdata from SD back to the USB?
User avatar
Joulinar
Posts: 2023
Joined: Sat Nov 16, 2019 12:49 am

Re: Services Failing after Swapping Backup Drives

Post by Joulinar »

Hi,

first I would recommend to move everything back to /mnt/dietpi_userdata/ on your SD card and try to restart MariaDB. If it still fails, let's have this fixed before trying to move it on USB again.
Pls let us know if a solution is working. This could help others if they hit by similar situation. Your DietPi Team
cdlenfert
Posts: 28
Joined: Fri Mar 29, 2019 5:42 pm

Re: Services Failing after Swapping Backup Drives

Post by cdlenfert »

Thanks for your response. I've moved everything back to /mnt/dietpi_userdata.

When trying to restart the service I get this:

Code: Select all

root@DietPi:/mnt/dietpi_userdata/mysql/mysql# journalctl -n 100
Feb 21 21:50:12 DietPi systemd[1]: Starting MariaDB 10.1.44 database server...
Feb 21 21:50:13 DietPi mysqld[24103]: 2020-02-21 21:50:13 1995718448 [Note] /usr/sbin/mysqld (mysqld 10.1.44-MariaDB-0+deb9u1) starting a
s process 24103 ...
Feb 21 21:50:13 DietPi mysqld[24103]: 2020-02-21 21:50:13 1995718448 [Warning] Can't create test file /var/lib/mysql/DietPi.lower-test
Feb 21 21:50:13 DietPi mysqld[24103]: [90B blob data]
Feb 21 21:50:13 DietPi mysqld[24103]: 2020-02-21 21:50:13 1995718448 [ERROR] Aborting
Feb 21 21:50:13 DietPi systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Feb 21 21:50:13 DietPi systemd[1]: Failed to start MariaDB 10.1.44 database server.
Feb 21 21:50:13 DietPi systemd[1]: mariadb.service: Unit entered failed state.
Feb 21 21:50:13 DietPi systemd[1]: mariadb.service: Failed with result 'exit-code'.
The warning about not being able to create the test file went away when I tested earlier, but now it's back. I'm pretty sure it's a permissions issue (at least in part). The following is what I had after the move:

Code: Select all

root@DietPi:/mnt/dietpi_userdata# ls -l
total 36
drwxr-xr-x 2 root root 4096 Feb 21 12:54 downloads
drwxr-xr-x 2 root root 4096 Feb 21 12:56 Music
drwxr-x--- 5 root root 4096 Feb 21 13:43 mysql
drwxr-x--- 6 root root 4096 Feb 21 12:57 nextcloud_data
drwxr-xr-x 3 root root 4096 Feb 21 12:56 php_bak
drwxr-xr-x 2 root root 4096 Feb 21 12:54 Pictures
drwx------ 3 root root 4096 Feb 21 12:58 syncthing
drwxr-xr-x 2 root root 4096 Feb 21 12:57 syncthing_data
drwxr-xr-x 2 root root 4096 Feb 21 12:54 Video
I did another rsync preserving permissions this time and now have:

Code: Select all

root@DietPi:/mnt/dietpi_userdata# ls -l
total 36
drwxrwxr-x 2 dietpi   dietpi   4096 Mar 10  2019 downloads
drwxrwxr-x 2 dietpi   dietpi   4096 Mar 15  2019 Music
drwxrwx--- 5 mysql    mysql    4096 Feb 15 15:39 mysql
drwxrwx--- 6 www-data www-data 4096 Mar 10  2019 nextcloud_data
drwxrwxr-x 3 dietpi   dietpi   4096 Mar 10  2019 php_bak
drwxrwxr-x 2 dietpi   dietpi   4096 Mar 10  2019 Pictures
drwx------ 3 dietpi   dietpi   4096 Feb 15 12:11 syncthing
drwxr-xr-x 2 dietpi   dietpi   4096 Feb 15 12:11 syncthing_data
drwxrwxr-x 2 dietpi   dietpi   4096 Mar 10  2019 Video
After that I also was able to start Mariadb. I also see that Nextcloud is in maintenance mode (do I let that resolve itself or do I need to disable it? - I did remove about 20GB of failed uploads that were in my userdata folder to make the restore to SD possible). Should I consider moving to the USB now? My real goal is to get to a more stable system running off a USB SSD as I mention here: viewtopic.php?f=9&t=7291&p=22539#p22539
User avatar
Joulinar
Posts: 2023
Joined: Sat Nov 16, 2019 12:49 am

Re: Services Failing after Swapping Backup Drives

Post by Joulinar »

Hi,

to disable NextCloud Maintenance Mode, just run the following

Code: Select all

ncc maintenance:mode --off
once done, pls use dietpi drive manager to relocate whole /mnt/dietpi_userdata. There is an option inside drive manager to mark your USB device as user data location. This as well should copy the data.
Pls let us know if a solution is working. This could help others if they hit by similar situation. Your DietPi Team
cdlenfert
Posts: 28
Joined: Fri Mar 29, 2019 5:42 pm

Re: Services Failing after Swapping Backup Drives

Post by cdlenfert »

This all worked well. Thank you! I’d like to only use my USB SSD for roots and user data, or possibly use SD for only booting. My hope would be to gain performance and stability. Right now my Plex database is on the SD and that can’t be great. Not sure what my path forward would look like to retain my data and migrate to an SSD based setup.
User avatar
Joulinar
Posts: 2023
Joined: Sat Nov 16, 2019 12:49 am

Re: Services Failing after Swapping Backup Drives

Post by Joulinar »

yep thats possible. You can use the SD card to host the boot partition and your USB device will contain the RootFS (incl user data)

Steps to be done:
  1. copy all your user data back to SD card using dietpi-drive_manager, to get the original status
  2. un-mount the USB device
  3. do a reboot to check if all working fine
  4. do a full backup of your SD card
  5. boot your system up, it should now running on SD card fully, while USB device is not mounted
  6. go to dietpi-drive_manager
  7. select the unmounted USB device
  8. select Transfer RootFS
  9. now read the following instruction carefully and confirm
  10. format your USB device
  11. last chance to cancel 8)
  12. once done, you will be ask to select a mount point like /mnt/usb. This is just temporary
  13. as soon as the device is mounted, Drive Manager will stop all processes and start to copy your entire RootFS
  14. time for a coffee
  15. you will be ask to reboot your device once completed
  16. during reboot you might see some services failing like MariaDB. This is happening because Drive Manager missed to copy /mnt/dietpi_userdata :roll:
  17. don't worry, let's fix it
  18. go to dietpi-drive_manager
  19. select the unused partition /dev/mmcblk0p2 on your SD card
  20. mount the partition as /mnt/sd2
  21. exit Drive Manager
  22. change directory to cd /mnt/sd2/mnt
  23. let's copy the missing data cp -p -r * /mnt/
  24. reboot your system
  25. now everything should working fine
If you encounter any issues on this, restore your backup to SD card 8)

@MichaIng
Is there a reason why Drive Manager is not copying dietpi_userdata during RootFS relocation?
Pls let us know if a solution is working. This could help others if they hit by similar situation. Your DietPi Team
User avatar
MichaIng
Site Admin
Posts: 2261
Joined: Sat Nov 18, 2017 6:21 pm

Re: Services Failing after Swapping Backup Drives

Post by MichaIng »

Not really, probably because it is assumed to be on external drive. However as well the rsync does not exclude all tmpfs, most importantly /DietPi is synced as well.

I'll clean this up by mounting the root partition to a temporary mount point and rsync the files from there. AFAIK rsync itself has an option to sync only files from the same drives, probably one to skip mounts automatically even when from same drive e.g. bind mounts. Much cleaner than excluding only known mounts and /mnt/* (which includes dietpi_userdata).
User avatar
Joulinar
Posts: 2023
Joined: Sat Nov 16, 2019 12:49 am

Re: Services Failing after Swapping Backup Drives

Post by Joulinar »

on my test, /mnt/ themselves was copied to my USB stick but without sub directories /mnt/*. Could that be because I mounted the USB device to /mnt/ before?? At least this was the offer of Drive Manager. Maybe I will some test later the day to have the USB device mounted somewhere else before moving RootFS
Pls let us know if a solution is working. This could help others if they hit by similar situation. Your DietPi Team
cdlenfert
Posts: 28
Joined: Fri Mar 29, 2019 5:42 pm

Re: Services Failing after Swapping Backup Drives

Post by cdlenfert »

Thanks @Joulinar. I'm glad to give this a try, but I'm curious about a couple of things I hope you might clarify.
  • "let's copy the missing data cp -p -r dietpi_userdata /mnt/" - so the USB SSD will exist at / and the default location for userdata becomes /mnt/ right?
  • Right now I'm using a 32GB SD card for boot/rootfs and it seems like it would be a waste just for booting the Pi 3b. Would it be reasonable/possible to also have boot from the USB? If so, what does that process look like vs. the one you so nicely outlined? and I understand a one time setting change is required on the Pi, but I don't understand if that means it will only be bootable from USB afterward, or if SD boot is still possible? What are other consequences of making the USB boot setting change?
  • as @MichaIng mentions "However as well the rsync does not exclude all tmpfs, most importantly /DietPi is synced as well." - is there something I should do differently from the instrucitons to not sync more than what is needed? such as wait for a DietPi update with the new "transfer rootfs" script?
User avatar
MichaIng
Site Admin
Posts: 2261
Joined: Sat Nov 18, 2017 6:21 pm

Re: Services Failing after Swapping Backup Drives

Post by MichaIng »

Following the instructions from Joulinar will work. Yes you can also boot the RPi 3 from USB. Is it 3+ or non-plus model?
Post Reply