Mariadb error after upgrade

Hallo, after today’s upgrade on my MiniPC (Dietpi 8.0.2) i noticed that mariadb is down and my Nextcloud is not working.

 sudo systemctl status mariadb.service
● mariadb.service - MariaDB 10.5.12 database server
     Loaded: loaded (/lib/systemd/system/mariadb.service; disabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Fri 2022-01-21 16:57:08 CET; 13min ago
       Docs: man:mariadbd(8)
             https://mariadb.com/kb/en/library/systemd/
    Process: 9664 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
    Process: 9665 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
    Process: 9667 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`cd /usr/bin/..; /usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1 (code=exited, status=0/SUCCESS)
    Process: 9714 ExecStart=/usr/sbin/mariadbd $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
   Main PID: 9714 (code=exited, status=1/FAILURE)
     Status: "MariaDB server is down"
        CPU: 72ms

sty 21 16:57:08 DietPi mariadbd[9714]: 2022-01-21 16:57:08 0 [ERROR] InnoDB: The innodb_system data file 'ibdata1' must be writable
sty 21 16:57:08 DietPi mariadbd[9714]: 2022-01-21 16:57:08 0 [ERROR] Plugin 'InnoDB' init function returned error.
sty 21 16:57:08 DietPi mariadbd[9714]: 2022-01-21 16:57:08 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
sty 21 16:57:08 DietPi mariadbd[9714]: 2022-01-21 16:57:08 0 [Note] Plugin 'FEEDBACK' is disabled.
sty 21 16:57:08 DietPi mariadbd[9714]: 2022-01-21 16:57:08 0 [ERROR] Could not open mysql.plugin table: "Table 'mysql.plugin' doesn't exist". Some plugins may be not loaded
sty 21 16:57:08 DietPi mariadbd[9714]: 2022-01-21 16:57:08 0 [ERROR] Failed to initialize plugins.
sty 21 16:57:08 DietPi mariadbd[9714]: 2022-01-21 16:57:08 0 [ERROR] Aborting
sty 21 16:57:08 DietPi systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
sty 21 16:57:08 DietPi systemd[1]: mariadb.service: Failed with result 'exit-code'.
sty 21 16:57:08 DietPi systemd[1]: Failed to start MariaDB 10.5.12 database server.

Also have problems with Home Assistant and Bazarr. When I restart services they work OK but after a while they are down again.

[ INFO ] DietPi-Services | bazarr                inactive (dead)
[FAILED] DietPi-Services | ● home-assistant.service - Home Assistant (DietPi)
     Loaded: loaded (/etc/systemd/system/home-assistant.service; disabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Fri 2022-01-21 16:57:30 CET; 21min ago
    Process: 10041 ExecStart=/home/homeassistant/homeassistant-start.sh (code=exited, status=1/FAILURE)
   Main PID: 10041 (code=exited, status=1/FAILURE)
        CPU: 613ms

Jan 21 16:57:28 DietPi systemd[1]: Started Home Assistant (DietPi).
Jan 21 16:57:30 DietPi homeassistant[10041]: Traceback (most recent call last):
Jan 21 16:57:30 DietPi homeassistant[10041]:   File "/home/homeassistant/.pyenv/versions/3.9.7/bin/hass", line 8, in <module>
Jan 21 16:57:30 DietPi homeassistant[10041]:     sys.exit(main())
Jan 21 16:57:30 DietPi homeassistant[10041]:   File "/home/homeassistant/.pyenv/versions/3.9.7/lib/python3.9/site-packages/homeassistant/__main__.py", line 316, in main
Jan 21 16:57:30 DietPi homeassistant[10041]:     with open(fault_file_name, mode="a", encoding="utf8") as fault_file:
Jan 21 16:57:30 DietPi homeassistant[10041]: PermissionError: [Errno 13] Permission denied: '/mnt/dietpi_userdata/homeassistant/home-assistant.log.fault'
Jan 21 16:57:30 DietPi systemd[1]: home-assistant.service: Main process exited, code=exited, status=1/FAILURE
Jan 21 16:57:30 DietPi systemd[1]: home-assistant.service: Failed with result 'exit-code'.

Can somebody help with that?
Regards.

For MariaDB do following to get some more logs

sed -i 's/#log_error/log_error/' /etc/mysql/mariadb.conf.d/50-server.cnf
systemctl restart mariadb.service
journalctl -u mariadb
cat /var/log/mysql/error.log
readlink /var/lib/mysql
readlink -f /var/lib/mysql

HA seems to have issues to write some files

PermissionError: [Errno 13] Permission denied: '/mnt/dietpi_userdata/homeassistant/home-assistant.log.fault'

Have a look into /mnt/dietpi_userdata/homeassistant/ if all files belongs to HA user

Hi, Joulinar I think maybe that errors are because I install yesterday Qbittorrent on Rpi3 (with ubuntu server) with dockstarter. I try to connect to my Radarr and Sonarr and guy from support help me to make it work. He told me to add mounted nfs share /mnt/nfs_Toshiba to .env of docker as /storage on that ubuntu server. My qbittorent work OK now but after that change dockstarter menu ask me to change permissions and meybe it change permissions everywhere. Because remote path mapping is like that in Radarr (for example:

IP: 192.168.0.20
Remote path: /storage
/mnt/dietpi_userdata
  1. First command for mariadb show nothing.
dietpi@DietPi:~$ sudo journalctl -u mariadb
Journal file /var/log/journal/1ae29948b12f4e4e8708271d3a7d334a/user-1000.journal is truncated, ignoring file.
-- Journal begins at Fri 2022-01-21 17:30:45 CET, ends at Fri 2022-01-21 19:02:15 CET. --
sty 21 17:30:51 DietPi systemd[1]: Starting MariaDB 10.5.12 database server...
sty 21 17:30:51 DietPi mariadbd[997]: 2022-01-21 17:30:51 0 [Note] /usr/sbin/mariadbd (mysqld 10.5.12-MariaDB-0+deb11u1) starting as process 997 ...
sty 21 17:30:51 DietPi mariadbd[997]: 2022-01-21 17:30:51 0 [Warning] Can't create test file /var/lib/mysql/DietPi.lower-test
sty 21 17:30:51 DietPi mariadbd[997]: 2022-01-21 17:30:51 0 [ERROR] mariadbd: File '/var/lib/mysql/aria_log_control' not found (Errcode: 13 "Permission denied")
sty 21 17:30:51 DietPi mariadbd[997]: 2022-01-21 17:30:51 0 [ERROR] mariadbd: Got error 'Can't open file' when trying to use aria control file '/var/lib/mysql/aria_log_control'
sty 21 17:30:51 DietPi mariadbd[997]: 2022-01-21 17:30:51 0 [ERROR] Plugin 'Aria' init function returned error.
sty 21 17:30:51 DietPi mariadbd[997]: 2022-01-21 17:30:51 0 [ERROR] Plugin 'Aria' registration as a STORAGE ENGINE failed.
sty 21 17:30:51 DietPi mariadbd[997]: 2022-01-21 17:30:51 0 [ERROR] InnoDB: The innodb_system data file 'ibdata1' must be writable
sty 21 17:30:51 DietPi mariadbd[997]: 2022-01-21 17:30:51 0 [ERROR] InnoDB: The innodb_system data file 'ibdata1' must be writable
sty 21 17:30:51 DietPi mariadbd[997]: 2022-01-21 17:30:51 0 [ERROR] Plugin 'InnoDB' init function returned error.
sty 21 17:30:51 DietPi mariadbd[997]: 2022-01-21 17:30:51 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
sty 21 17:30:51 DietPi mariadbd[997]: 2022-01-21 17:30:51 0 [Note] Plugin 'FEEDBACK' is disabled.
sty 21 17:30:51 DietPi mariadbd[997]: 2022-01-21 17:30:51 0 [ERROR] Could not open mysql.plugin table: "Table 'mysql.plugin' doesn't exist". Some plugins may be not loaded
sty 21 17:30:51 DietPi mariadbd[997]: 2022-01-21 17:30:51 0 [ERROR] Failed to initialize plugins.
sty 21 17:30:51 DietPi mariadbd[997]: 2022-01-21 17:30:51 0 [ERROR] Aborting
sty 21 17:30:51 DietPi systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
sty 21 17:30:51 DietPi systemd[1]: mariadb.service: Failed with result 'exit-code'.
sty 21 17:30:51 DietPi systemd[1]: Failed to start MariaDB 10.5.12 database server.
sty 21 18:53:39 DietPi systemd[1]: Starting MariaDB 10.5.12 database server...
sty 21 18:53:39 DietPi mariadbd[18202]: 2022-01-21 18:53:39 0 [Note] /usr/sbin/mariadbd (mysqld 10.5.12-MariaDB-0+deb11u1) starting as process 18202 ...
sty 21 18:53:39 DietPi mariadbd[18202]: 2022-01-21 18:53:39 0 [Warning] Can't create test file /var/lib/mysql/DietPi.lower-test
sty 21 18:53:39 DietPi mariadbd[18202]: 2022-01-21 18:53:39 0 [ERROR] mariadbd: File '/var/lib/mysql/aria_log_control' not found (Errcode: 13 "Permission denied")
sty 21 18:53:39 DietPi mariadbd[18202]: 2022-01-21 18:53:39 0 [ERROR] mariadbd: Got error 'Can't open file' when trying to use aria control file '/var/lib/mysql/aria_log_control'
sty 21 18:53:39 DietPi mariadbd[18202]: 2022-01-21 18:53:39 0 [ERROR] Plugin 'Aria' init function returned error.
sty 21 18:53:39 DietPi mariadbd[18202]: 2022-01-21 18:53:39 0 [ERROR] Plugin 'Aria' registration as a STORAGE ENGINE failed.
sty 21 18:53:39 DietPi mariadbd[18202]: 2022-01-21 18:53:39 0 [ERROR] InnoDB: The innodb_system data file 'ibdata1' must be writable
sty 21 18:53:39 DietPi mariadbd[18202]: 2022-01-21 18:53:39 0 [ERROR] InnoDB: The innodb_system data file 'ibdata1' must be writable
sty 21 18:53:39 DietPi mariadbd[18202]: 2022-01-21 18:53:39 0 [ERROR] Plugin 'InnoDB' init function returned error.
sty 21 18:53:39 DietPi mariadbd[18202]: 2022-01-21 18:53:39 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
sty 21 18:53:39 DietPi mariadbd[18202]: 2022-01-21 18:53:39 0 [Note] Plugin 'FEEDBACK' is disabled.
sty 21 18:53:39 DietPi mariadbd[18202]: 2022-01-21 18:53:39 0 [ERROR] Could not open mysql.plugin table: "Table 'mysql.plugin' doesn't exist". Some plugins may be not loaded
sty 21 18:53:39 DietPi mariadbd[18202]: 2022-01-21 18:53:39 0 [ERROR] Failed to initialize plugins.
sty 21 18:53:39 DietPi mariadbd[18202]: 2022-01-21 18:53:39 0 [ERROR] Aborting
sty 21 18:53:39 DietPi systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
sty 21 18:53:39 DietPi systemd[1]: mariadb.service: Failed with result 'exit-code'.
sty 21 18:53:39 DietPi systemd[1]: Failed to start MariaDB 10.5.12 database server.
  1. cat /var/log/mysql/error.log = no such file
dietpi@DietPi:~$ readlink /var/lib/mysql
/mnt/dietpi_userdata/mysql
dietpi@DietPi:~$ readlink -f /var/lib/mysql
/mnt/Toshiba/dietpi_userdata/mysql

For Home assistant folder I have everywhere dietpi:dietpi. I change that for homeassistant:homeassistant and now works. Meybe I will reinstall Bazarr and that will recover wright permissions.

Regards.


Regards.

Yes looks like you destroyed your system with changing the permission on whole dietpi user data directory. Can you share following

ls -la /mnt/dietpi_userdata/mysql

I restore backup but don’t help.

dietpi@DietPi:~$ ls -la /mnt/dietpi_userdata/mysql
razem 188588
drwxr-xr-x  5 dietpi dietpi      4096 01-20 11:12 .
drwxrwxr-x 22 dietpi dietpi      4096 01-20 21:03 ..
-rw-rw----  1 dietpi dietpi    114688 01-21 11:49 aria_log.00000001
-rw-rw----  1 dietpi dietpi        52 01-21 11:49 aria_log_control
-rw-r--r--  1 dietpi dietpi         0 10-10 12:54 debian-10.5.flag
-rw-rw----  1 dietpi dietpi     14560 01-20 11:12 ib_buffer_pool
-rw-rw----  1 dietpi dietpi  79691776 01-21 11:49 ibdata1
-rw-rw----  1 dietpi dietpi 100663296 01-21 11:49 ib_logfile0
-rw-rw----  1 dietpi dietpi  12582912 01-20 11:12 ibtmp1
-rw-rw----  1 dietpi dietpi         0 10-10 12:55 multi-master.info
drwx------  2 dietpi dietpi      4096 10-10 12:55 mysql
-rw-rw----  1 dietpi dietpi        16 10-10 12:55 mysql_upgrade_info
drwx------  2 dietpi dietpi     12288 10-11 09:45 nextcloud
drwx------  2 dietpi dietpi      4096 10-10 12:55 performance_schema
dietpi@DietPi:~$

Regards.

User data, located on an external disk, are excluded from backup in a default environment. Therefore restore did not change anything.

You need to change all files and folders to user: group mysql

chown -R mysql:mysql /mnt/dietpi_userdata/mysql

Thanks, that works. Only problem I noticed now is with Bazarr and Lidarr (that are two apps I don’t have with Nginx Proxy Manager - they works only with local IP address).

 DietPi-Services
─────────────────────────────────────────────────────
 Mode: restart 

[  OK  ] DietPi-Services | restart : avahi-daemon
[  OK  ] DietPi-Services | restart : proftpd
[  OK  ] DietPi-Services | restart : nfs-kernel-server
[  OK  ] DietPi-Services | restart : redis-server
[  OK  ] DietPi-Services | restart : mariadb
[  OK  ] DietPi-Services | restart : php7.4-fpm
[  OK  ] DietPi-Services | restart : lighttpd
[  OK  ] DietPi-Services | restart : plexmediaserver
[  OK  ] DietPi-Services | restart : emby-server
[  OK  ] DietPi-Services | restart : jackett
[  OK  ] DietPi-Services | restart : sonarr
[  OK  ] DietPi-Services | restart : radarr
[  OK  ] DietPi-Services | restart : lidarr
[  OK  ] DietPi-Services | restart : bazarr
[  OK  ] DietPi-Services | restart : home-assistant
[  OK  ] DietPi-Services | restart : noip2
[  OK  ] DietPi-Services | restart : docker
[  OK  ] DietPi-Services | restart : cron
dietpi@DietPi:~$ sudo systemctl status bazarr.service 
● bazarr.service - Bazarr Daemon (DietPi)
     Loaded: loaded (/etc/systemd/system/bazarr.service; disabled; vendor preset: enabled)
     Active: inactive (dead)

sty 21 19:58:18 DietPi Bazarr[29916]:   File "/opt/bazarr/bazarr/../libs/peewee.py", line 190, in reraise
sty 21 19:58:18 DietPi Bazarr[29916]:     raise value.with_traceback(tb)
sty 21 19:58:18 DietPi Bazarr[29916]:   File "/opt/bazarr/bazarr/../libs/peewee.py", line 3144, in execute_sql
sty 21 19:58:18 DietPi Bazarr[29916]:     cursor.execute(sql, params or ())
sty 21 19:58:18 DietPi Bazarr[29916]: peewee.OperationalError: attempt to write a readonly database
sty 21 19:58:19 DietPi Bazarr[29889]: Python 3.9.x is unsupported. Current version is 3.9.2. Keep in mind that even if it works, you're on your own.
sty 21 19:58:19 DietPi Bazarr[29889]: Bazarr starting...
sty 21 19:58:19 DietPi Bazarr[29889]: Bazarr exited.
sty 21 19:58:19 DietPi systemd[1]: bazarr.service: Succeeded.
sty 21 19:58:19 DietPi systemd[1]: bazarr.service: Consumed 1.858s CPU time.

dietpi@DietPi:~$ sudo systemctl status lidarr.service 
● lidarr.service - Lidarr Daemon (DietPi)
     Loaded: loaded (/etc/systemd/system/lidarr.service; disabled; vendor preset: enabled)
     Active: active (running) since Fri 2022-01-21 19:58:14 CET; 12min ago
   Main PID: 29881 (Lidarr)
      Tasks: 11 (limit: 9354)
     Memory: 34.2M
        CPU: 1.102s
     CGroup: /system.slice/lidarr.service
             └─29881 /opt/lidarr/Lidarr -nobrowser -data=/mnt/dietpi_userdata/lidarr

sty 21 19:58:15 DietPi Lidarr[29881]: [Info] Bootstrap: Starting Lidarr - /opt/lidarr/Lidarr.dll - Version 0.8.1.2135
sty 21 19:58:15 DietPi Lidarr[29881]: [Info] AppFolderInfo: Data directory is being overridden to [/mnt/dietpi_userdata/lidarr]
sty 21 19:58:15 DietPi Lidarr[29881]: [Trace] DiskProviderBase: Directory '/mnt/dietpi_userdata/lidarr' isn't writable. Access to the path '/mnt/dietpi_userdata/lidarr/lidarr_write_test.txt' is denied.
sty 21 19:58:15 DietPi Lidarr[29881]: [Fatal] ConsoleApp: EPIC FAIL!
sty 21 19:58:15 DietPi Lidarr[29881]: [v0.8.1.2135] NzbDrone.Common.Exceptions.LidarrStartupException: Lidarr failed to start: AppFolder /mnt/dietpi_userdata/lidarr is not writable
sty 21 19:58:15 DietPi Lidarr[29881]:    at NzbDrone.Common.EnvironmentInfo.AppFolderFactory.Register() in D:\a\1\s\src\NzbDrone.Common\EnvironmentInfo\AppFolderFactory.cs:line 55
sty 21 19:58:15 DietPi Lidarr[29881]:    at NzbDrone.Host.Bootstrap.Start(StartupContext startupContext, IUserAlert userAlert, Action`1 startCallback) in D:\a\1\s\src\NzbDrone.Host\Bootstrap.cs:line 39
sty 21 19:58:15 DietPi Lidarr[29881]:    at NzbDrone.Console.ConsoleApp.Main(String[] args) in D:\a\1\s\src\NzbDrone.Console\ConsoleApp.cs:line 43
sty 21 19:58:15 DietPi Lidarr[29881]: Press enter to exit...
sty 21 19:58:16 DietPi Lidarr[29881]: Non-recoverable failure, waiting for user intervention...

Looks like Lidarr is working but it’s not.
Will reinstall that two apps help to restore permissions?

Probably. Just give it a try

OK, is there a command to list only installed apps?

dietpi-software list | grep " =2 "

Thank You very much for help. Everything is working now.
Can You tell me also if I want to upgrade Radarr to v4 it’s good to change in Radarr settings from master to develop?

theoretically it should be quite simple to switch Radarr

#do a backup first
systemctl stop radarr.service
cd /tmp/
rm -rf /opt/radarr/
wget https://github.com/Radarr/Radarr/releases/download/v4.0.2.5836/Radarr.develop.4.0.2.5836.linux-core-arm64.tar.gz
tar -xvzf Radarr.develop.4.0.2.5836.linux-core-arm64.tar.gz
mv Radarr /opt/radarr
rm Radarr.develop.4.0.2.5836.linux-core-arm64.tar.gz
chown -R radarr:dietpi /opt/radarr
systemctl restart radarr.service

Download URL would need to be adjusted depending on system type (arm, arm64, x64)

Thank You very much. :smiley::+1:
Regards.