nextcloud sad now - internal server error

Troubleshooting
1

Hi everyone, I am very new to dietpi, been playing with it for a few days…
I was able to get nextcloud installed and working with a letsencrypt cert, and I moved the userdata via the drive manager to an external ssd.

This worked for a while, then overnight, I can no longer reach nextcloud.

Internal Server Error

The server encountered an internal error and was unable to complete your request.
Please contact the server administrator if this error reappears multiple times, please include the technical details below in your report.
More details can be found in the server log.

I think I have lost my connection to the mariadb, but I’m not sure how or why.
I created a bug report: 00cadb3c-f270-4f9f-8153-74a9ac34ff36

Is there a way to reinstall nextcloud without losing my data?

2

Hi,

Let’s check status of MariaDB

systemctl restart mariadb.service
journalctl -u mariadb
cat /var/log/mysql/error.log
readlink /var/lib/mysql
readlink -f /var/lib/mysql
3

journalctl -u mariadb

Feb 13 18:30:09 DietPi systemd[1]: Starting MariaDB 10.3.27 database server...
Feb 13 18:30:10 DietPi mysqld[507]: 2021-02-13 18:30:10 0 [Note] /usr/sbin/mysqld (mysqld 10.3.27-MariaDB-0+deb10u1) starting as process 507 ...
Feb 13 18:30:10 DietPi mysqld[507]: 2021-02-13 18:30:10 0 [Warning] Could not increase number of max_open_files to more than 16384 (request: 32186)
Feb 13 18:30:12 DietPi systemd[1]: mariadb.service: Main process exited, code=killed, status=11/SEGV
Feb 13 18:30:12 DietPi systemd[1]: mariadb.service: Failed with result 'signal'.
Feb 13 18:30:12 DietPi systemd[1]: Failed to start MariaDB 10.3.27 database server.
Feb 13 18:30:18 DietPi systemd[1]: mariadb.service: Service RestartSec=5s expired, scheduling restart.
Feb 13 18:30:18 DietPi systemd[1]: mariadb.service: Scheduled restart job, restart counter is at 1.
Feb 13 18:30:18 DietPi systemd[1]: Stopped MariaDB 10.3.27 database server.
Feb 13 18:30:18 DietPi systemd[1]: Starting MariaDB 10.3.27 database server...
Feb 13 18:30:18 DietPi mysqld[1347]: 2021-02-13 18:30:18 0 [Note] /usr/sbin/mysqld (mysqld 10.3.27-MariaDB-0+deb10u1) starting as process 1347 ...
Feb 13 18:30:18 DietPi mysqld[1347]: 2021-02-13 18:30:18 0 [Warning] Could not increase number of max_open_files to more than 16384 (request: 32186)
Feb 13 18:30:20 DietPi systemd[1]: Started MariaDB 10.3.27 database server.
Feb 13 18:30:20 DietPi systemd[1]: mariadb.service: Main process exited, code=killed, status=11/SEGV
Feb 13 18:30:20 DietPi systemd[1]: mariadb.service: Failed with result 'signal'.
Feb 13 18:30:25 DietPi systemd[1]: mariadb.service: Service RestartSec=5s expired, scheduling restart.
Feb 13 18:30:25 DietPi systemd[1]: mariadb.service: Scheduled restart job, restart counter is at 2.
Feb 13 18:30:25 DietPi systemd[1]: Stopped MariaDB 10.3.27 database server.
Feb 13 18:30:25 DietPi systemd[1]: Starting MariaDB 10.3.27 database server...
Feb 13 18:30:25 DietPi mysqld[1522]: 2021-02-13 18:30:25 0 [Note] /usr/sbin/mysqld (mysqld 10.3.27-MariaDB-0+deb10u1) starting as process 1522 ...
Feb 13 18:30:25 DietPi mysqld[1522]: 2021-02-13 18:30:25 0 [Warning] Could not increase number of max_open_files to more than 16384 (request: 32186)
Feb 13 18:30:26 DietPi systemd[1]: Started MariaDB 10.3.27 database server.
Feb 13 18:30:27 DietPi systemd[1]: mariadb.service: Main process exited, code=killed, status=11/SEGV
Feb 13 18:30:27 DietPi systemd[1]: mariadb.service: Failed with result 'signal'.
Feb 13 18:30:32 DietPi systemd[1]: mariadb.service: Service RestartSec=5s expired, scheduling restart.
Feb 13 18:30:32 DietPi systemd[1]: mariadb.service: Scheduled restart job, restart counter is at 3.
Feb 13 18:30:32 DietPi systemd[1]: Stopped MariaDB 10.3.27 database server.
Feb 13 18:30:32 DietPi systemd[1]: Starting MariaDB 10.3.27 database server...
Feb 13 18:30:32 DietPi mysqld[1627]: 2021-02-13 18:30:32 0 [Note] /usr/sbin/mysqld (mysqld 10.3.27-MariaDB-0+deb10u1) starting as process 1627 ...
Feb 13 18:30:32 DietPi mysqld[1627]: 2021-02-13 18:30:32 0 [Warning] Could not increase number of max_open_files to more than 16384 (request: 32186)
Feb 13 18:30:34 DietPi systemd[1]: Started MariaDB 10.3.27 database server.
Feb 13 18:30:34 DietPi systemd[1]: mariadb.service: Main process exited, code=killed, status=11/SEGV

error.log

Server version: 10.3.27-MariaDB-0+deb10u1
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=0
max_threads=153
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467430 Kbytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x49000
2021-02-13 18:50:32 0 [Note] InnoDB: Waiting for purge to start
/usr/sbin/mysqld(my_print_stacktrace+0x30)[0x5580af13b8]
/usr/sbin/mysqld(handle_fatal_signal+0x508)[0x5580656758]
linux-vdso.so.1(__kernel_rt_sigreturn+0x0)[0x7faff297c0]
2021-02-13 18:50:32 0 [Note] InnoDB: 10.3.27 started; log sequence number 108806939; transaction id 294378
2021-02-13 18:50:32 0 [Note] InnoDB: Loading buffer pool(s) from /mnt/dietpi_userdata/mysql/ib_buffer_pool
2021-02-13 18:50:32 0 [Note] Plugin 'FEEDBACK' is disabled.
2021-02-13 18:50:32 0 [Note] Recovering after a crash using tc.log
2021-02-13 18:50:32 0 [Note] Starting crash recovery...
2021-02-13 18:50:32 0 [Note] Crash recovery finished.
2021-02-13 18:50:32 0 [Note] Server socket created on IP: '127.0.0.1'.
/usr/sbin/mysqld(+0x8fee2c)[0x558081ee2c]
/usr/sbin/mysqld(+0x9c755c)[0x55808e755c]
/usr/sbin/mysqld(+0x94f99c)[0x558086f99c]
/usr/sbin/mysqld(+0x94fb58)[0x558086fb58]
/usr/sbin/mysqld(+0xa90ba8)[0x55809b0ba8]
/usr/sbin/mysqld(+0xa91284)[0x55809b1284]
/usr/sbin/mysqld(+0x95e7d8)[0x558087e7d8]
/usr/sbin/mysqld(+0x917284)[0x5580837284]
/usr/sbin/mysqld(+0x991d34)[0x55808b1d34]
/usr/sbin/mysqld(+0x992054)[0x55808b2054]
/usr/sbin/mysqld(+0x9924ac)[0x55808b24ac]
2021-02-13 18:50:32 0 [Note] Reading of all Master_info entries succeeded
2021-02-13 18:50:32 0 [Note] Added new Master_info '' to hash table
2021-02-13 18:50:32 0 [Note] /usr/sbin/mysqld: ready for connections.
Version: '10.3.27-MariaDB-0+deb10u1'  socket: '/run/mysqld/mysqld.sock'  port: 3306  Debian 10
/lib/aarch64-linux-gnu/libpthread.so.0(+0x77e4)[0x7fafed37e4]
2021-02-13 18:50:32 0 [Note] InnoDB: Buffer pool(s) load completed at 210213 18:50:32
/lib/aarch64-linux-gnu/libc.so.6(+0xcfadc)[0x7faf9e2adc]
The manual page at https://mariadb.com/kb/en/how-to-produce-a-full-stack-trace-for-mysqld/ contains
information that should help you find out what is causing the crash.
Writing a core file...
Working directory at /mnt/dietpi_userdata/mysql
Resource Limits:
Limit                     Soft Limit           Hard Limit           Units
Max cpu time              unlimited            unlimited            seconds
Max file size             unlimited            unlimited            bytes
Max data size             unlimited            unlimited            bytes
Max stack size            8388608              unlimited            bytes
Max core file size        0                    unlimited            bytes
Max resident set          unlimited            unlimited            bytes
Max processes             3546                 3546                 processes
Max open files            16384                16384                files
Max locked memory         65536                65536                bytes
Max address space         unlimited            unlimited            bytes
Max file locks            unlimited            unlimited            locks
Max pending signals       3546                 3546                 signals
Max msgqueue size         819200               819200               bytes
Max nice priority         0                    0
Max realtime priority     0                    0
Max realtime timeout      unlimited            unlimited            us
Core pattern: core

2021-02-13 18:50:38 0 [Note] InnoDB: Using Linux native AIO
2021-02-13 18:50:38 0 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-02-13 18:50:38 0 [Note] InnoDB: Uses event mutexes
2021-02-13 18:50:38 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2021-02-13 18:50:38 0 [Note] InnoDB: Number of pools: 1
2021-02-13 18:50:38 0 [Note] InnoDB: Using generic crc32 instructions
2021-02-13 18:50:38 0 [Note] InnoDB: Initializing buffer pool, total size = 128M, instances = 1, chunk size = 128M
2021-02-13 18:50:38 0 [Note] InnoDB: Completed initialization of buffer pool
2021-02-13 18:50:38 0 [Note] InnoDB: If the mysqld execution user is authorized, page cleaner thread priority can be changed. See the man page of setpriority().
2021-02-13 18:50:38 0 [Note] InnoDB: Starting crash recovery from checkpoint LSN=108805484
2021-02-13 18:50:38 0 [Note] InnoDB: 1 transaction(s) which must be rolled back or cleaned up in total 1 row operations to undo
2021-02-13 18:50:38 0 [Note] InnoDB: Trx id counter is 294377
2021-02-13 18:50:38 0 [Note] InnoDB: Starting final batch to recover 7 pages from redo log.
2021-02-13 18:50:39 0 [Note] InnoDB: 128 out of 128 rollback segments are active.
2021-02-13 18:50:39 0 [Note] InnoDB: Starting in background the rollback of recovered transactions
2021-02-13 18:50:39 0 [ERROR] InnoDB: Corruption in the tablespace. Bitmap shows insert buffer records to page [page id: space=122, page number=22] though the page type is 0, which is not an index leaf page. We try to resolve the problem by skipping the insert buffer merge for this page. Please run CHECK TABLE on your tables to determine if they are corrupt after this.
InnoDB: Discarding record
 PHYSICAL RECORD: n_fields 7; 1-byte offsets; info bits 0
 0: len 4; hex 0000007a; asc    z;;
 1: len 1; hex 00; asc  ;;
 2: len 4; hex 00000016; asc     ;;
 3: len 22; hex 2021-02-13 18:50:39 0 [Note] InnoDB: Removed temporary tablespace data file: "ibtmp1"
2021-02-13 18:50:39 0 [Note] InnoDB: Creating shared tablespace for temporary tables
000000010c0f03fc002e860800088000860800088000; asc        2021-02-13 18:50:39 0 [Note] InnoDB: Setting file './ibtmp1' size to 12 MB. Physically writing the file full; Please wait ...
  .          2021-02-13 18:50:39 0 [Note] InnoDB: File './ibtmp1' size is now 12 MB.
  ;;
 4: len 0; hex ; asc ;;
 5: len 8; hex 8000000000000000; asc         ;;
 6: len 8; hex 8000000000000283; asc         ;;

InnoDB: from the insert buffer!

210213 18:50:39 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed,
something is definitely wrong and this may fail.

Server version: 10.3.27-MariaDB-0+deb10u1
key_buffer_size=134217728
read_buffer_size=131072
max_used_connections=0
max_threads=153
thread_count=0
It is possible that mysqld could use up to
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 467430 Kbytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x49000
2021-02-13 18:50:39 0 [Note] InnoDB: Waiting for purge to start
/usr/sbin/mysqld(my_print_stacktrace+0x30)[0x557bd313b8]
/usr/sbin/mysqld(handle_fatal_signal+0x508)[0x557b896758]
linux-vdso.so.1(__kernel_rt_sigreturn+0x0)[0x7f8f8247c0]
2021-02-13 18:50:39 0 [Note] InnoDB: 10.3.27 started; log sequence number 108806939; transaction id 294378
2021-02-13 18:50:39 0 [Note] InnoDB: Loading buffer pool(s) from /mnt/dietpi_userdata/mysql/ib_buffer_pool
2021-02-13 18:50:39 0 [Note] Plugin 'FEEDBACK' is disabled.
2021-02-13 18:50:39 0 [Note] Recovering after a crash using tc.log
2021-02-13 18:50:39 0 [Note] Starting crash recovery...
2021-02-13 18:50:39 0 [Note] Crash recovery finished.
2021-02-13 18:50:39 0 [Note] Server socket created on IP: '127.0.0.1'.
/usr/sbin/mysqld(+0x8fee2c)[0x557ba5ee2c]
/usr/sbin/mysqld(+0x9c755c)[0x557bb2755c]
/usr/sbin/mysqld(+0x94f99c)[0x557baaf99c]
/usr/sbin/mysqld(+0x94fb58)[0x557baafb58]
/usr/sbin/mysqld(+0xa90ba8)[0x557bbf0ba8]
/usr/sbin/mysqld(+0xa91284)[0x557bbf1284]
/usr/sbin/mysqld(+0x95e7d8)[0x557babe7d8]
/usr/sbin/mysqld(+0x917284)[0x557ba77284]
/usr/sbin/mysqld(+0x991d34)[0x557baf1d34]
/usr/sbin/mysqld(+0x992054)[0x557baf2054]
/usr/sbin/mysqld(+0x9924ac)[0x557baf24ac]
2021-02-13 18:50:39 0 [Note] Reading of all Master_info entries succeeded
2021-02-13 18:50:39 0 [Note] Added new Master_info '' to hash table
2021-02-13 18:50:39 0 [Note] /usr/sbin/mysqld: ready for connections.
Version: '10.3.27-MariaDB-0+deb10u1'  socket: '/run/mysqld/mysqld.sock'  port: 3

readlink -f /var/lib/mysql

/mnt/dietpi_userdata/mysql
4

What happens to my nextcloud data if I reinstall the software using the dietpi reinstall command?

5

Also noticed I can’t exit out of maintenance mode, not sure if that is the cause of the problem or a symptom of the actual problem.

root@DietPi:/var/www/nextcloud/lib/private/DB# ncc maintenance:mode --off

An unhandled exception has been thrown:
Doctrine\DBAL\DBALException: Failed to connect to the database: An exception occurred in driver: SQLSTATE[HY000] [2002] No such file or directory in /var/www/nextcloud/lib/private/DB/Connection.php:72
Stack trace:
#0 /var/www/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(1449): OC\DB\Connection->connect()
#1 /var/www/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Connection.php(892): Doctrine\DBAL\Connection->getWrappedConnection()
#2 /var/www/nextcloud/lib/private/DB/Connection.php(202): Doctrine\DBAL\Connection->executeQuery('SELECT * FROM `...', Array, Array, NULL)
#3 /var/www/nextcloud/3rdparty/doctrine/dbal/lib/Doctrine/DBAL/Query/QueryBuilder.php(206): OC\DB\Connection->executeQuery('SELECT * FROM `...', Array, Array)
#4 /var/www/nextcloud/lib/private/DB/QueryBuilder/QueryBuilder.php(217): Doctrine\DBAL\Query\QueryBuilder->execute()
#5 /var/www/nextcloud/lib/private/AppConfig.php(345): OC\DB\QueryBuilder\QueryBuilder->execute()
#6 /var/www/nextcloud/lib/private/AppConfig.php(110): OC\AppConfig->loadConfigValues()
#7 /var/www/nextcloud/lib/private/AppConfig.php(301): OC\AppConfig->getApps()
#8 /var/www/nextcloud/lib/private/legacy/OC_App.php(949): OC\AppConfig->getValues(false, 'installed_versi...')
#9 /var/www/nextcloud/lib/private/Server.php(668): OC_App::getAppVersions()
#10 /var/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(155): OC\Server->OC\{closure}(Object(OC\Server))
#11 /var/www/nextcloud/3rdparty/pimple/pimple/src/Pimple/Container.php(118): OC\AppFramework\Utility\SimpleContainer->OC\AppFramework\Utility\{closure}(Object(Pimple\Container))
#12 /var/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(122): Pimple\Container->offsetGet('OC\\Memcache\\Fac...')
#13 /var/www/nextcloud/lib/private/ServerContainer.php(156): OC\AppFramework\Utility\SimpleContainer->query('OC\\Memcache\\Fac...', true)
#14 /var/www/nextcloud/lib/private/Server.php(1677): OC\ServerContainer->query('OC\\Memcache\\Fac...')
#15 /var/www/nextcloud/lib/private/Server.php(1017): OC\Server->getMemCacheFactory()
#16 /var/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(155): OC\Server->OC\{closure}(Object(OC\Server))
#17 /var/www/nextcloud/3rdparty/pimple/pimple/src/Pimple/Container.php(118): OC\AppFramework\Utility\SimpleContainer->OC\AppFramework\Utility\{closure}(Object(Pimple\Container))
#18 /var/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(122): Pimple\Container->offsetGet('OCP\\Lock\\ILocki...')
#19 /var/www/nextcloud/lib/private/ServerContainer.php(156): OC\AppFramework\Utility\SimpleContainer->query('OCP\\Lock\\ILocki...', true)
#20 /var/www/nextcloud/lib/private/Server.php(1977): OC\ServerContainer->query('OCP\\Lock\\ILocki...')
#21 /var/www/nextcloud/lib/private/Files/View.php(118): OC\Server->getLockingProvider()
#22 /var/www/nextcloud/lib/private/Server.php(395): OC\Files\View->__construct()
#23 /var/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(155): OC\Server->OC\{closure}(Object(OC\Server))
#24 /var/www/nextcloud/3rdparty/pimple/pimple/src/Pimple/Container.php(118): OC\AppFramework\Utility\SimpleContainer->OC\AppFramework\Utility\{closure}(Object(Pimple\Container))
#25 /var/www/nextcloud/lib/private/AppFramework/Utility/SimpleContainer.php(122): Pimple\Container->offsetGet('OC\\Files\\Node\\H...')
#26 /var/www/nextcloud/lib/private/ServerContainer.php(156): OC\AppFramework\Utility\SimpleContainer->query('OC\\Files\\Node\\H...', true)
#27 /var/www/nextcloud/lib/private/Server.php(1324): OC\ServerContainer->query('OC\\Files\\Node\\H...')
#28 /var/www/nextcloud/lib/base.php(595): OC\Server->boot()
#29 /var/www/nextcloud/lib/base.php(1091): OC::init()
#30 /var/www/nextcloud/console.php(49): require_once('/var/www/nextcl...')
#31 /var/www/nextcloud/occ(11): require_once('/var/www/nextcl...')
#32 {main}root@DietPi:/var/www/nextcloud/lib/private/DB#
6

hmm there seems to be something really wrong on your database.

[ERROR] InnoDB: Corruption in the tablespace.
[ERROR] mysqld got signal 11 ;

Not sure if a simple reinstallation would fix something

What kind of data you are storing in NectCloud? Just documents? For 1 or 2 user? Or something more complex like calendar?

As well you could check your system for kernel messages dmesg -l err,crit,alert,emerg

As well you can check for sufficient free space df -h