Nextcloud no longer reachable

I previously had Nextcloud working and I can’t say for sure when it stopped. Many updates to DietPi since the last time I used it. I’m admittedly not great at troubleshooting. I’ll share what info I know and dig up anything else that could be helpful. Here’s my setup:
Device and DietPi version

  • Pi 3b


  • DietPi v6.26.3

Software

I noticed when I did a dietpi backup that the command

ncc maintenance:mode --on

failed.

I also noticed that after the backup succeeded and services were re-enabled that mariadb failed to start.

systemctl status mariadb

tells me it’s in fact not running: “Failed to start MariaDB 10.1.38 database server.”

I had thought that Pi-hole might be blocking it, but the fact that required services aren’t running, maybe that’s not the case. Pi-hole does a pretty confusing thing where it shows a block page and the block reason “does not appear on any blacklist” instead of a basic 404 message when a page is not available.

Anyway I did a few other configuration changes that I might be able to track down so that the following work:

  • VPN routes through Pi-Hole so clients don’t see ads


  • Added DNS Host Map on my router that points my dynamic DNS address to my DietPi devices IP (allowing access over the DYNDNS address inside my LAN

Should I try reinstalling Nextcloud? If I recall there was a pretty good amount of setup and configuration required and I’d like to avoid duplicating that effort, but I’d like even more to have it working again :slight_smile: Thanks for any ideas and please let me know if you have any questions and I can provide more details. Love the things I’m able to do thanks to DietPi!!

I read on Nextcloud’s troubleshooting FAQ page that Lightppd is not recommended and when I do dietpi-software I see that Lightppd is set as the “Webserver preference”. Not sure to do with that info quite yet, but it seems possibly significant.

Lighttpd works pretty well with Nextcloud. There have just not been made any official tests, hence they recommend apache2 or nginx which is officially assured to work.

Since MariaDB server fails to start, it has nothing to do with Pi-hole, the webserver etc. Please paste:
journalctl -u mariadb

Thanks for confirming it’s not Pi-hole. Here’s the output of the command.

# journalctl -u mariadb
-- Logs begin at Thu 2016-11-03 11:16:42 MDT, end at Sat 2019-12-14 20:05:19 MST
. --
Dec 08 17:54:43 DietPi systemd[1]: Starting MariaDB 10.1.38 database server...
Dec 08 17:54:45 DietPi mysqld[1332]: 2019-12-08 17:54:45 1996369712 [Note] /usr/
sbin/mysqld (mysqld 10.1.38-MariaDB-0+deb9u1) starting as process 1332 ...
Dec 08 17:54:51 DietPi systemd[1]: mariadb.service: Main process exited,
 code=exited, status=1/FAILURE
Dec 08 17:54:51 DietPi systemd[1]: Failed to start MariaDB 10.1.38 datab
ase server.
Dec 08 17:54:51 DietPi systemd[1]: mariadb.service: Unit entered failed 
state.
Dec 08 17:54:51 DietPi systemd[1]: mariadb.service: Failed with result '
exit-code'.
Dec 12 12:15:11 DietPi systemd[1]: Starting MariaDB 10.1.38 database server...
Dec 12 12:15:13 DietPi mysqld[1760]: 2019-12-12 12:15:13 1996263216 [Note] /usr/
sbin/mysqld (mysqld 10.1.38-MariaDB-0+deb9u1) starting as process 1760 ...
Dec 12 12:15:19 DietPi systemd[1]: mariadb.service: Main process exited,
 code=exited, status=1/FAILURE
Dec 12 12:15:19 DietPi systemd[1]: Failed to start MariaDB 10.1.38 datab
ase server.
Dec 12 12:15:19 DietPi systemd[1]: mariadb.service: Unit entered failed 
state.
Dec 12 12:15:19 DietPi systemd[1]: mariadb.service: Failed with result '
exit-code'.

Ah the binary logs to file, please paste:
cat /var/log/mysql/error.log
If it’s empty (hourly RAMlog clear), please restart the service to trigger fresh entries:
systemctl restart mariadb

Initially the log file was empty and I had to restart the mariadb service to get the following log data.

cat /var/log/mysql/error.log 
2019-12-15  7:39:19 1995521840 [Note] InnoDB: innodb_empty_free_list_algorithm has been changed to legacy because of small buffer pool size. In order to use backoff, increase buffer pool at least up to 20MB.

2019-12-15  7:39:19 1995521840 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2019-12-15  7:39:19 1995521840 [Note] InnoDB: The InnoDB memory heap is disabled
2019-12-15  7:39:19 1995521840 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2019-12-15  7:39:19 1995521840 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2019-12-15  7:39:19 1995521840 [Note] InnoDB: Compressed tables use zlib 1.2.8
2019-12-15  7:39:19 1995521840 [Note] InnoDB: Using Linux native AIO
2019-12-15  7:39:19 1995521840 [Note] InnoDB: Using generic crc32 instructions
2019-12-15  7:39:19 1995521840 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2019-12-15  7:39:19 1995521840 [Note] InnoDB: Completed initialization of buffer pool
2019-12-15  7:39:19 1995521840 [Note] InnoDB: Highest supported file format is Barracuda.
2019-12-15  7:39:19 1995521840 [Note] InnoDB: Starting crash recovery from checkpoint LSN=1146481047
2019-12-15  7:39:20 1995521840 [Note] InnoDB: Restoring possible half-written data pages from the doublewrite buffer...
InnoDB: Set innodb_force_recovery to ignore this error.
2019-12-15  7:39:20 1995521840 [ERROR] Plugin 'InnoDB' init function returned error.
2019-12-15  7:39:20 1995521840 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2019-12-15  7:39:20 1995521840 [Note] Plugin 'FEEDBACK' is disabled.
2019-12-15  7:39:20 1995521840 [ERROR] Unknown/unsupported storage engine: InnoDB
2019-12-15  7:39:20 1995521840 [ERROR] Aborting

Error in my_thread_global_end(): 1 threads didn't exit

I’m still not sure what to try next here. Anything on the log output I should look into first?

Hi,

your system is trying to do a crash recovery. And it seems this is not working.

Maybe you could try to set the following option to get the DB back online.

nano /etc/mysql/mariadb.conf.d/50-server.cnf

pls add the following line within [mysqld] section (right at the beginning)

innodb_force_recovery=1

save the file and try to restart MariaDB

systemctl restart mariadb

once done pls past the output of /var/log/mysql/error.log again

I just tried this and after adding the force recovery line fright after [mysqld] (following line) and restarting the mariadb service, it still fails but the log file is empty.

ok let’s try to remove the innodb_force_recovery and restart again.

I cleared force recovery line and restarted the mariadb service again. Still no output to the mysql log. I then found a Stack Exchange thread about trying to clear the innodb log files so I made a backup and did that.

root@DietPi:~# rm /var/lib/mysql/ib_logfile0
root@DietPi:~# rm /var/lib/mysql/ib_logfile1

Then I tried restarting the mariadb service again and this time it started. That was exciting, but I still couldn’t reach Nextcloud. I tried restarting the Pi and that didn’t bring Nextcloud back, but Mariadb was running fine after the reboot.

Here is the status of the Mariadb service:

root@DietPi:~# systemctl status mariadb
● mariadb.service - MariaDB 10.1.38 database server
   Loaded: loaded (/lib/systemd/system/mariadb.service; disabled; vendor preset:
 enabled)
   Active: active (running) since Tue 2020-01-21 13:18:52 MST; 3s ago
     Docs: man:mysqld(8)
           https://mariadb.com/kb/en/library/systemd/
  Process: 2079 ExecStartPost=/bin/sh -c systemctl unset-environment _WSREP_STAR
T_POSITION (code=exited, status=0/SUCCESS)
  Process: 2077 ExecStartPost=/etc/mysql/debian-start (code=exited, status=0/SUC
CESS)
  Process: 2023 ExecStartPre=/bin/sh -c [ ! -e /usr/bin/galera_recovery ] && VAR
= ||   VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ]   && systemctl set-environme
nt _WSREP_START_POSITION=$VAR || exit 1 (code=exited, status=0/SUCCESS)
  Process: 2018 ExecStartPre=/bin/sh -c systemctl unset-environment _WSREP_START
_POSITION (code=exited, status=0/SUCCESS)
  Process: 2015 ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/ru
n/mysqld (code=exited, status=0/SUCCESS)
 Main PID: 2048 (mysqld)
   Status: "Taking your SQL requests now..."
    Tasks: 26 (limit: 4915)
   CGroup: /system.slice/mariadb.service
           └─2048 /usr/sbin/mysqld

Jan 21 13:18:51 DietPi mysqld[2048]: InnoDB: for more information.
Jan 21 13:18:51 DietPi mysqld[2048]: 2020-01-21 13:18:51 1995968304 [Note] InnoD
B: 128 rollback segment(s) are active.
Jan 21 13:18:51 DietPi mysqld[2048]: 2020-01-21 13:18:51 1995968304 [Note] InnoD
B: Waiting for purge to start
Jan 21 13:18:51 DietPi mysqld[2048]: 2020-01-21 13:18:51 1995968304 [Note] InnoD
B:  Percona XtraDB (http://www.percona.com) 5.6.42-84.2 started; log sequence nu
mber 1070602161
Jan 21 13:18:51 DietPi mysqld[2048]: 2020-01-21 13:18:51 1283453760 [Note] InnoD
B: Dumping buffer pool(s) not yet started
Jan 21 13:18:51 DietPi mysqld[2048]: 2020-01-21 13:18:51 1995968304 [Note] Plugi
n 'FEEDBACK' is disabled.
Jan 21 13:18:51 DietPi mysqld[2048]: 2020-01-21 13:18:51 1995968304 [Note] Serve
r socket created on IP: '::'.
Jan 21 13:18:51 DietPi mysqld[2048]: 2020-01-21 13:18:51 1995968304 [Note] /usr/
sbin/mysqld: ready for connections.
Jan 21 13:18:51 DietPi mysqld[2048]: Version: '10.1.38-MariaDB-0+deb9u1'  socket
: '/var/run/mysqld/mysqld.sock'  port: 3306  Raspbian 9.0
Jan 21 13:18:52 DietPi systemd[1]: Started MariaDB 10.1.38 database server.

Still nothing in the logs (/var/log/mysql/error.log)

what happen if you run

ncc maintenance:mode --off

Thanks for your assistance. It’s greatly appreciated!

The command told me Maintenance mode was already off.

root@DietPi:~# ncc maintenance:mode --off
Maintenance mode already disabled

I belive restoring the mariadb service did solve the issue however and I was unfortunately looking at the wrong URL to get to the server. Forgot the /nextcloud directory :roll_eyes:

Nextcloud appears to be working once again!! Thank you!

Ok no problem. I marked the issue as solved now.