Nextcloud no longer reachable Topic is solved

[cdlenfert]

I previously had Nextcloud working and I can't say for sure when it stopped. Many updates to DietPi since the last time I used it. I'm admittedly not great at troubleshooting. I'll share what info I know and dig up anything else that could be helpful. Here's my setup:
Device and DietPi version

• Pi 3b

• DietPi v6.26.3

Software

• Plex Media Server

• Nextcloud

• Pi-hole

• Nextcloud

• 3rd party VPN service - https://github.com/hwdsl2/setup-ipsec-vpn

I noticed when I did a dietpi backup that the command

Code: Select all

ncc maintenance:mode --on

failed.

I also noticed that after the backup succeeded and services were re-enabled that mariadb failed to start.

Code: Select all

systemctl status mariadb

tells me it's in fact not running: "Failed to start MariaDB 10.1.38 database server."

I had thought that Pi-hole might be blocking it, but the fact that required services aren't running, maybe that's not the case. Pi-hole does a pretty confusing thing where it shows a block page and the block reason "does not appear on any blacklist" instead of a basic 404 message when a page is not available.

Anyway I did a few other configuration changes that I might be able to track down so that the following work:

• VPN routes through Pi-Hole so clients don't see ads

• Added DNS Host Map on my router that points my dynamic DNS address to my DietPi devices IP (allowing access over the DYNDNS address inside my LAN

Should I try reinstalling Nextcloud? If I recall there was a pretty good amount of setup and configuration required and I'd like to avoid duplicating that effort, but I'd like even more to have it working again Thanks for any ideas and please let me know if you have any questions and I can provide more details. Love the things I'm able to do thanks to DietPi!!

[cdlenfert]

I read on Nextcloud's troubleshooting FAQ page that Lightppd is not recommended and when I do dietpi-software I see that Lightppd is set as the "Webserver preference". Not sure to do with that info quite yet, but it seems possibly significant.

[MichaIng]

Lighttpd works pretty well with Nextcloud. There have just not been made any official tests, hence they recommend apache2 or nginx which is officially assured to work.

Since MariaDB server fails to start, it has nothing to do with Pi-hole, the webserver etc. Please paste:
journalctl -u mariadb

[cdlenfert]

Thanks for confirming it's not Pi-hole. Here's the output of the command.

Code: Select all

# journalctl -u mariadb
-- Logs begin at Thu 2016-11-03 11:16:42 MDT, end at Sat 2019-12-14 20:05:19 MST
. --
Dec 08 17:54:43 DietPi systemd[1]: Starting MariaDB 10.1.38 database server...
Dec 08 17:54:45 DietPi mysqld[1332]: 2019-12-08 17:54:45 1996369712 [Note] /usr/
sbin/mysqld (mysqld 10.1.38-MariaDB-0+deb9u1) starting as process 1332 ...
Dec 08 17:54:51 DietPi systemd[1]: mariadb.service: Main process exited,
 code=exited, status=1/FAILURE
Dec 08 17:54:51 DietPi systemd[1]: Failed to start MariaDB 10.1.38 datab
ase server.
Dec 08 17:54:51 DietPi systemd[1]: mariadb.service: Unit entered failed 
state.
Dec 08 17:54:51 DietPi systemd[1]: mariadb.service: Failed with result '
exit-code'.
Dec 12 12:15:11 DietPi systemd[1]: Starting MariaDB 10.1.38 database server...
Dec 12 12:15:13 DietPi mysqld[1760]: 2019-12-12 12:15:13 1996263216 [Note] /usr/
sbin/mysqld (mysqld 10.1.38-MariaDB-0+deb9u1) starting as process 1760 ...
Dec 12 12:15:19 DietPi systemd[1]: mariadb.service: Main process exited,
 code=exited, status=1/FAILURE
Dec 12 12:15:19 DietPi systemd[1]: Failed to start MariaDB 10.1.38 datab
ase server.
Dec 12 12:15:19 DietPi systemd[1]: mariadb.service: Unit entered failed 
state.
Dec 12 12:15:19 DietPi systemd[1]: mariadb.service: Failed with result '
exit-code'.

[MichaIng]

Ah the binary logs to file, please paste:
cat /var/log/mysql/error.log
If it's empty (hourly RAMlog clear), please restart the service to trigger fresh entries:
systemctl restart mariadb

[cdlenfert]

Initially the log file was empty and I had to restart the mariadb service to get the following log data.

Code: Select all

cat /var/log/mysql/error.log 
2019-12-15  7:39:19 1995521840 [Note] InnoDB: innodb_empty_free_list_algorithm has been changed to legacy because of small buffer pool size. In order to use backoff, increase buffer pool at least up to 20MB.

2019-12-15  7:39:19 1995521840 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2019-12-15  7:39:19 1995521840 [Note] InnoDB: The InnoDB memory heap is disabled
2019-12-15  7:39:19 1995521840 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2019-12-15  7:39:19 1995521840 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2019-12-15  7:39:19 1995521840 [Note] InnoDB: Compressed tables use zlib 1.2.8
2019-12-15  7:39:19 1995521840 [Note] InnoDB: Using Linux native AIO
2019-12-15  7:39:19 1995521840 [Note] InnoDB: Using generic crc32 instructions
2019-12-15  7:39:19 1995521840 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2019-12-15  7:39:19 1995521840 [Note] InnoDB: Completed initialization of buffer pool
2019-12-15  7:39:19 1995521840 [Note] InnoDB: Highest supported file format is Barracuda.
2019-12-15  7:39:19 1995521840 [Note] InnoDB: Starting crash recovery from checkpoint LSN=1146481047
2019-12-15  7:39:20 1995521840 [Note] InnoDB: Restoring possible half-written data pages from the doublewrite buffer...
InnoDB: Set innodb_force_recovery to ignore this error.
2019-12-15  7:39:20 1995521840 [ERROR] Plugin 'InnoDB' init function returned error.
2019-12-15  7:39:20 1995521840 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2019-12-15  7:39:20 1995521840 [Note] Plugin 'FEEDBACK' is disabled.
2019-12-15  7:39:20 1995521840 [ERROR] Unknown/unsupported storage engine: InnoDB
2019-12-15  7:39:20 1995521840 [ERROR] Aborting

Error in my_thread_global_end(): 1 threads didn't exit

[cdlenfert]

I’m still not sure what to try next here. Anything on the log output I should look into first?

[Joulinar]

Hi,

your system is trying to do a crash recovery. And it seems this is not working.

Maybe you could try to set the following option to get the DB back online.

Code: Select all

nano /etc/mysql/mariadb.conf.d/50-server.cnf

pls add the following line within [mysqld] section (right at the beginning)

Code: Select all

innodb_force_recovery=1

save the file and try to restart MariaDB

Code: Select all

systemctl restart mariadb

once done pls past the output of /var/log/mysql/error.log again

[cdlenfert]

I just tried this and after adding the force recovery line fright after [mysqld] (following line) and restarting the mariadb service, it still fails but the log file is empty.

[Joulinar]

ok let's try to remove the innodb_force_recovery and restart again.