MariaDB fails to start after upgrade

bbkhdt · 15 December 2021 17:00

Hi,

After I upgraded DietPi to 7.9.3 and did a reboot I cannot access my nextcloud instance. Using dietpi-services start MariaDB fails to start.

Checking ncc maintenance:mode gives the following backtrace

An unhandled exception has been thrown:
Doctrine\DBAL\Exception: Failed to connect to the database: An exception occurred in the driver: SQLSTATE[HY000] [2002] No such file or directory in /var/www/nextcloud/lib/private/DB/Connection.php:87
Stack trace:
#0 /var/www/nextcloud/3rdparty/doctrine/dbal/src/Connection.php(1519): OC\DB\Connection->connect()
#1 /var/www/nextcloud/3rdparty/doctrine/dbal/src/Connection.php(1041): Doctrine\DBAL\Connection->getWrappedConnection()
#2 /var/www/nextcloud/lib/private/DB/Connection.php(236): Doctrine\DBAL\Connection->executeQuery('SELECT * FROM `...', Array, Array, NULL)
#3 /var/www/nextcloud/3rdparty/doctrine/dbal/src/Query/QueryBuilder.php(345): OC\DB\Connection->executeQuery('SELECT * FROM `...', Array, Array)
#4 /var/www/nextcloud/lib/private/DB/QueryBuilder/QueryBuilder.php(287): Doctrine\DBAL\Query\QueryBuilder->execute()
#5 /var/www/nextcloud/lib/private/AppConfig.php(344): OC\DB\QueryBuilder\QueryBuilder->execute()
#6 /var/www/nextcloud/lib/private/AppConfig.php(109): OC\AppConfig->loadConfigValues()
#7 /var/www/nextcloud/lib/private/AppConfig.php(300): OC\AppConfig->getApps()
#8 /var/www/nextcloud/lib/private/legacy/OC_App.php(971): OC\AppConfig->getValues(false, 'installed_versi...')
#9 /var/www/nextcloud/lib/private/Server.php(687): OC_App::getAppVersions()
#10 /var/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(160): OC\Server->OC\{closure}(Object(OC\Server))
#11 /var/www/nextcloud/3rdparty/pimple/pimple/src/Pimple/Container.php(118): OC\AppFramework\Utility\SimpleContainer->OC\AppFramework\Utility\{closure}(Object(Pimple\Container))
#12 /var/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(127): Pimple\Container->offsetGet('OC\\Memcache\\Fac...')
#13 /var/www/nextcloud/lib/private/ServerContainer.php(136): OC\AppFramework\Utility\SimpleContainer->query('OC\\Memcache\\Fac...', false)
#14 /var/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(56): OC\ServerContainer->query('OC\\Memcache\\Fac...')
#15 /var/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(182): OC\AppFramework\Utility\SimpleContainer->get('OC\\Memcache\\Fac...')
#16 /var/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(160): OC\AppFramework\Utility\SimpleContainer->OC\AppFramework\Utility\{closure}(Object(OC\Server))
#17 /var/www/nextcloud/3rdparty/pimple/pimple/src/Pimple/Container.php(114): OC\AppFramework\Utility\SimpleContainer->OC\AppFramework\Utility\{closure}(Object(Pimple\Container))
#18 /var/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(127): Pimple\Container->offsetGet('OCP\\ICacheFacto...')
#19 /var/www/nextcloud/lib/private/ServerContainer.php(136): OC\AppFramework\Utility\SimpleContainer->query('OCP\\ICacheFacto...', false)
#20 /var/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(56): OC\ServerContainer->query('OCP\\ICacheFacto...')
#21 /var/www/nextcloud/lib/private/Server.php(1064): OC\AppFramework\Utility\SimpleContainer->get('OCP\\ICacheFacto...')
#22 /var/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(160): OC\Server->OC\{closure}(Object(OC\Server))
#23 /var/www/nextcloud/3rdparty/pimple/pimple/src/Pimple/Container.php(118): OC\AppFramework\Utility\SimpleContainer->OC\AppFramework\Utility\{closure}(Object(Pimple\Container))
#24 /var/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(127): Pimple\Container->offsetGet('OCP\\Lock\\ILocki...')
#25 /var/www/nextcloud/lib/private/ServerContainer.php(136): OC\AppFramework\Utility\SimpleContainer->query('OCP\\Lock\\ILocki...', false)
#26 /var/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(56): OC\ServerContainer->query('OCP\\Lock\\ILocki...')
#27 /var/www/nextcloud/lib/private/Server.php(2024): OC\AppFramework\Utility\SimpleContainer->get('OCP\\Lock\\ILocki...')
#28 /var/www/nextcloud/lib/private/Files/View.php(118): OC\Server->getLockingProvider()
#29 /var/www/nextcloud/lib/private/Server.php(435): OC\Files\View->__construct()
#30 /var/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(160): OC\Server->OC\{closure}(Object(OC\Server))
#31 /var/www/nextcloud/3rdparty/pimple/pimple/src/Pimple/Container.php(118): OC\AppFramework\Utility\SimpleContainer->OC\AppFramework\Utility\{closure}(Object(Pimple\Container))
#32 /var/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(127): Pimple\Container->offsetGet('OC\\Files\\Node\\H...')
#33 /var/www/nextcloud/lib/private/ServerContainer.php(136): OC\AppFramework\Utility\SimpleContainer->query('OC\\Files\\Node\\H...', false)
#34 /var/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(56): OC\ServerContainer->query('OC\\Files\\Node\\H...')
#35 /var/www/nextcloud/lib/private/Server.php(1387): OC\AppFramework\Utility\SimpleContainer->get('OC\\Files\\Node\\H...')
#36 /var/www/nextcloud/lib/base.php(594): OC\Server->boot()
#37 /var/www/nextcloud/lib/base.php(1089): OC::init()
#38 /var/www/nextcloud/console.php(48): require_once('/var/www/nextcl...')
#39 /var/www/nextcloud/occ(11): require_once('/var/www/nextcl...')
#40 {main}

I need some help interpreting and acting on this.

Joulinar · 15 December 2021 17:13

Pls can you share following

systemctl restart mariadb.service
journalctl -u mariadb
cat /var/log/mysql/error.log
readlink /var/lib/mysql
readlink -f /var/lib/mysql

bbkhdt · 15 December 2021 18:13

Hi, thanks for the imediate reply!

so systemctl restart mariadb.service gives me this:

Job for mariadb.service failed because the control process exited with error code.
See "systemctl status mariadb.service" and "journalctl -xe" for details.

Then systemctl status mariadb.service gives me:

● mariadb.service - MariaDB 10.3.31 database server
   Loaded: loaded (/lib/systemd/system/mariadb.service; disabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Wed 2021-12-15 17:34:57 GMT; 11min ago
     Docs: man:mysqld(8)
           https://mariadb.com/kb/en/library/systemd/
  Process: 30858 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld (code=exited, status=0/SUCCESS)
  Process: 30859 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START_POSITION (code=exited, status=0/SUCCESS)
  Process: 30861 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR= ||   VAR=`cd /usr/bin/..; /usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemctl set-environment _WSREP_STA
RT_POSITION=$VAR || exit 1 (code=exited, status=0/SUCCESS)
  Process: 30917 ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION (code=exited, status=1/FAILURE)
 Main PID: 30917 (code=exited, status=1/FAILURE)
   Status: "MariaDB server is down"

Dec 15 17:28:56 DietPi systemd[1]: Starting MariaDB 10.3.31 database server...
Dec 15 17:30:26 DietPi mysqld[30917]: 2021-12-15 17:30:26 0 [Note] /usr/sbin/mysqld (mysqld 10.3.31-MariaDB-0+deb10u1) starting as process 30917 ...
Dec 15 17:33:27 DietPi mysqld[30917]: 2021-12-15 17:33:27 0 [Warning] Can't create test file /var/lib/mysql/DietPi.lower-test
Dec 15 17:34:57 DietPi mysqld[30917]: [87B blob data]
Dec 15 17:34:57 DietPi mysqld[30917]: 2021-12-15 17:34:57 0 [ERROR] Aborting
Dec 15 17:34:57 DietPi systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Dec 15 17:34:57 DietPi systemd[1]: mariadb.service: Failed with result 'exit-code'.
Dec 15 17:34:57 DietPi systemd[1]: Failed to start MariaDB 10.3.31 database server.

Then journalctl -u mariadb prints:

-- Logs begin at Wed 2021-12-15 13:07:23 GMT, end at Wed 2021-12-15 17:48:02 GMT. --
Dec 15 13:09:26 DietPi systemd[1]: Starting MariaDB 10.3.31 database server...
Dec 15 13:10:57 DietPi mysqld[721]: 2021-12-15 13:10:57 0 [Note] /usr/sbin/mysqld (mysqld 10.3.31-MariaDB-0+deb10u1) starting as process 721 ...
Dec 15 13:13:57 DietPi mysqld[721]: 2021-12-15 13:13:57 0 [Warning] Can't create test file /var/lib/mysql/DietPi.lower-test
Dec 15 13:15:28 DietPi mysqld[721]: [87B blob data]
Dec 15 13:15:28 DietPi mysqld[721]: 2021-12-15 13:15:28 0 [ERROR] Aborting
Dec 15 13:15:28 DietPi systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Dec 15 13:15:28 DietPi systemd[1]: mariadb.service: Failed with result 'exit-code'.
Dec 15 13:15:28 DietPi systemd[1]: Failed to start MariaDB 10.3.31 database server.
Dec 15 14:27:10 DietPi systemd[1]: Starting MariaDB 10.3.31 database server...
Dec 15 14:28:02 DietPi mysqld[1788]: 2021-12-15 14:28:02 0 [Note] /usr/sbin/mysqld (mysqld 10.3.31-MariaDB-0+deb10u1) starting as process 1788 ...
Dec 15 14:31:03 DietPi mysqld[1788]: 2021-12-15 14:31:03 0 [Warning] Can't create test file /var/lib/mysql/DietPi.lower-test
Dec 15 14:32:33 DietPi mysqld[1788]: [87B blob data]
Dec 15 14:32:33 DietPi mysqld[1788]: 2021-12-15 14:32:33 0 [ERROR] Aborting
Dec 15 14:32:33 DietPi systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Dec 15 14:32:33 DietPi systemd[1]: mariadb.service: Failed with result 'exit-code'.
Dec 15 14:32:33 DietPi systemd[1]: Failed to start MariaDB 10.3.31 database server.
Dec 15 14:37:06 DietPi systemd[1]: Starting MariaDB 10.3.31 database server...
Dec 15 14:38:02 DietPi mysqld[5288]: 2021-12-15 14:38:02 0 [Note] /usr/sbin/mysqld (mysqld 10.3.31-MariaDB-0+deb10u1) starting as process 5288 ...
Dec 15 14:41:02 DietPi mysqld[5288]: 2021-12-15 14:41:02 0 [Warning] Can't create test file /var/lib/mysql/DietPi.lower-test
Dec 15 14:42:32 DietPi mysqld[5288]: [87B blob data]
Dec 15 14:42:32 DietPi mysqld[5288]: 2021-12-15 14:42:32 0 [ERROR] Aborting
Dec 15 14:42:32 DietPi systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Dec 15 14:42:32 DietPi systemd[1]: mariadb.service: Failed with result 'exit-code'.
Dec 15 14:42:32 DietPi systemd[1]: Failed to start MariaDB 10.3.31 database server.
Dec 15 16:48:21 DietPi systemd[1]: Starting MariaDB 10.3.31 database server...
Dec 15 16:49:52 DietPi mysqld[23577]: 2021-12-15 16:49:52 0 [Note] /usr/sbin/mysqld (mysqld 10.3.31-MariaDB-0+deb10u1) starting as process 23577 ...
Dec 15 16:52:52 DietPi mysqld[23577]: 2021-12-15 16:52:52 0 [Warning] Can't create test file /var/lib/mysql/DietPi.lower-test
Dec 15 16:54:23 DietPi mysqld[23577]: [87B blob data]
Dec 15 16:54:23 DietPi mysqld[23577]: 2021-12-15 16:54:23 0 [ERROR] Aborting
Dec 15 16:54:23 DietPi systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Dec 15 16:54:23 DietPi systemd[1]: mariadb.service: Failed with result 'exit-code'.
Dec 15 16:54:23 DietPi systemd[1]: Failed to start MariaDB 10.3.31 database server.
Dec 15 17:28:56 DietPi systemd[1]: Starting MariaDB 10.3.31 database server...
Dec 15 17:30:26 DietPi mysqld[30917]: 2021-12-15 17:30:26 0 [Note] /usr/sbin/mysqld (mysqld 10.3.31-MariaDB-0+deb10u1) starting as process 30917 ...
Dec 15 17:33:27 DietPi mysqld[30917]: 2021-12-15 17:33:27 0 [Warning] Can't create test file /var/lib/mysql/DietPi.lower-test
Dec 15 17:34:57 DietPi mysqld[30917]: [87B blob data]
Dec 15 17:34:57 DietPi mysqld[30917]: 2021-12-15 17:34:57 0 [ERROR] Aborting
Dec 15 17:34:57 DietPi systemd[1]: mariadb.service: Main process exited, code=exited, status=1/FAILURE
Dec 15 17:34:57 DietPi systemd[1]: mariadb.service: Failed with result 'exit-code'.
Dec 15 17:34:57 DietPi systemd[1]: Failed to start MariaDB 10.3.31 database server.

Then cat /var/log/mysql/error.log prints nothing

Then readlink /var/lib/mysql prints:

/mnt/dietpi_userdata/mysql

Then readlink -f /var/lib/mysql prints nothing again.

Joulinar · 15 December 2021 18:16

Do you use an external disk to store your data?

bbkhdt · 15 December 2021 18:29

Yes, I’m using an Odroid HC1 with a Sata-SSD

Joulinar · 15 December 2021 18:52

And the attached drive is working correctly. Means you are able to access it?

bbkhdt · 15 December 2021 21:07

Uhm, no. I can’t access it. It took a long time to detect devices and I was unable to mount the drive. Do you think this is a coincidence? Or do you think the upgrade has something to do with that.

Joulinar · 15 December 2021 21:17

Usually an update is not able to damage your drive. Can you check if it is connected correctly? Does it have its own PSU? All cables are correct? You can check for kernel error message

dmesg -l err,crit,alert,emerg

bbkhdt · 16 December 2021 11:39

Ok, so I shut down the device, cut power for a minute and rebooted the system. Now the drive is accessible again.

Somehow it got unmounted. I deleted the previous mount point, created a new one with the same name so that paths match (using dietpi-drive-manager). Then I restarted all servives via dietpi-services restart. Now MariaDB starts and my NextCloud instance is accessible again.

Thanks for your help! I will keep an eye on the drive. I hope it’s not dying. It’s relatively new. Anyway this issue is a good motivation for finally automating the Dietpi and NextCloud backup routine.