Nextcloud - Internal Server Error (after apt-upgrade)

Troubleshooting
1

Hi,

just encountered an Internal Server Error when visiting my Nextcloud site.
This happens after an apt upgrade, running DietPi v6.34.3

There are some similar listings here, but my symptoms seem to be different.
All my services are up and running, no error.logs

Apt upgrade mentioned php and mariadb, so I guess, the problem grew up from here.

lighttpd is in use, shows no errors. No errors from php.

MariaDB is up: 2021-02-22 19:47:35 547931471872 [Note] /usr/sbin/mysqld: ready for connections.
Version: '10.1.48-MariaDB-0+deb9u1'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  Debian 9.13

Jellyfin is running fine.

One thing I observe are these different outputs.
The latter is the correct one. Should these two be similar?

root@DietPi:~# readlink /var/lib/mysql
/mnt/dietpi_userdata/mysql

root@DietPi:~# readlink -f /var/lib/mysql
/mnt/usb_1/dietpi_userdata/mysql

Is there anybody with an idea what went wrong?

Thanks & regards
Frans


Other output:

root@DietPi:~# dietpi-services status

 DietPi-Services
─────────────────────────────────────────────────────
 Mode: status

[  OK  ] DietPi-Services | nmbd			active (running) since Mon 2021-02-22 19:47:30 CET; 4min 21s ago
[  OK  ] DietPi-Services | smbd			active (running) since Mon 2021-02-22 19:47:31 CET; 4min 21s ago
[  OK  ] DietPi-Services | redis-server		active (running) since Mon 2021-02-22 19:47:32 CET; 4min 20s ago
[  OK  ] DietPi-Services | mariadb		active (running) since Mon 2021-02-22 19:47:36 CET; 4min 15s ago
[  OK  ] DietPi-Services | php7.3-fpm		active (running) since Mon 2021-02-22 19:47:36 CET; 4min 15s ago
[  OK  ] DietPi-Services | lighttpd		active (running) since Mon 2021-02-22 19:47:36 CET; 4min 15s ago
[  OK  ] DietPi-Services | jellyfin		active (running) since Mon 2021-02-22 19:47:36 CET; 4min 15s ago
[  OK  ] DietPi-Services | cron			active (running) since Mon 2021-02-22 19:47:37 CET; 4min 15s ago
[  OK  ] DietPi-Services | dropbear		active (running) since Mon 2021-02-22 19:47:30 CET; 4min 22s ago
[  OK  ] DietPi-Services | fail2ban		active (running) since Mon 2021-02-22 19:47:17 CET; 4min 35s ago
[  OK  ] DietPi-Services | dietpi-ramlog	active (exited) since Mon 2021-02-22 19:47:14 CET; 4min 37s ago
[  OK  ] DietPi-Services | dietpi-preboot	active (exited) since Mon 2021-02-22 19:47:15 CET; 4min 37s ago
[  OK  ] DietPi-Services | dietpi-boot		active (exited) since Mon 2021-02-22 19:47:30 CET; 4min 22s ago
[  OK  ] DietPi-Services | dietpi-postboot	active (exited) since Mon 2021-02-22 19:47:30 CET; 4min 22s ago
[ INFO ] DietPi-Services | dietpi-wifi-monitor	inactive (dead)




root@DietPi:~# dmesg -l err,crit,alert,emerg
[    0.152616] tv_vout: major number 254 for disp
[    0.156005] genirq: Setting trigger mode 8 for irq 241 failed (gic_set_type+0x0/0x148)
[    0.156027] genirq: Setting trigger mode 8 for irq 242 failed (gic_set_type+0x0/0x148)
[    0.156106] genirq: Setting trigger mode 8 for irq 241 failed (gic_set_type+0x0/0x148)
[    0.156127] genirq: Setting trigger mode 8 for irq 242 failed (gic_set_type+0x0/0x148)
[    0.424040] kvm [1]: kvm_arch_timer: can't find DT node
[    0.424096] kvm [1]: error initializing Hyp mode: -19
[    0.448966] mtdoops: mtd device (mtddev=name/number) must be supplied
[    0.916008] sd: error -110 whilst initialising MMC card
[    1.120452] ppmgr_driver_probe called
[    1.121148] amlvideo_init called
[    1.121148] amlvideo_create_instance called
[    1.121151] v4l2_dev.name=:amlvideo-000
[    1.122676] cectx c810023c.aocec: aml_cec_probe(): can't find hdmirx
[    3.329395] sd 1:0:0:0: [sdb] No Caching mode page found
[    3.333018] sd 1:0:0:0: [sdb] Assuming drive cache: write through
[    3.511491] sd 2:0:0:0: [sdc] No Caching mode page found
[    3.511496] sd 2:0:0:0: [sdc] Assuming drive cache: write through




root@DietPi:~# journalctl -u mariadb
-- Logs begin at Thu 2016-11-03 18:16:42 CET, end at Mon 2021-02-22 19:52:43 CET. --
Feb 22 19:47:32 DietPi systemd[1]: Starting MariaDB 10.1.48 database server...
Feb 22 19:47:34 DietPi mysqld[1069]: 2021-02-22 19:47:34 547931471872 [Note] /usr/sbin/mysqld (mysqld 10.1.48-MariaDB-0+deb9u1) starting as process 1069 ...
Feb 22 19:47:36 DietPi systemd[1]: Started MariaDB 10.1.48 database server.




root@DietPi:~# cat /var/log/mysql/error.log
2021-02-22 19:47:34 547931471872 [Note] InnoDB: innodb_empty_free_list_algorithm has been changed to legacy because of small buffer pool size. In order to use backoff, increase buffer pool at least up to 20MB.

2021-02-22 19:47:34 547931471872 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2021-02-22 19:47:34 547931471872 [Note] InnoDB: The InnoDB memory heap is disabled
2021-02-22 19:47:34 547931471872 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2021-02-22 19:47:34 547931471872 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2021-02-22 19:47:34 547931471872 [Note] InnoDB: Compressed tables use zlib 1.2.8
2021-02-22 19:47:34 547931471872 [Note] InnoDB: Using Linux native AIO
2021-02-22 19:47:34 547931471872 [Note] InnoDB: Using generic crc32 instructions
2021-02-22 19:47:34 547931471872 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2021-02-22 19:47:34 547931471872 [Note] InnoDB: Completed initialization of buffer pool
2021-02-22 19:47:34 547931471872 [Note] InnoDB: Highest supported file format is Barracuda.
2021-02-22 19:47:35 547931471872 [Note] InnoDB: 128 rollback segment(s) are active.
2021-02-22 19:47:35 547931471872 [Note] InnoDB: Waiting for purge to start
2021-02-22 19:47:35 547931471872 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.49-89.0 started; log sequence number 6534359889
2021-02-22 19:47:35 547318919216 [Note] InnoDB: Dumping buffer pool(s) not yet started
2021-02-22 19:47:35 547931471872 [Note] Plugin 'FEEDBACK' is disabled.
2021-02-22 19:47:35 547931471872 [Note] Server socket created on IP: '127.0.0.1'.
2021-02-22 19:47:35 547931471872 [Note] /usr/sbin/mysqld: ready for connections.
Version: '10.1.48-MariaDB-0+deb9u1'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  Debian 9.13




root@DietPi:/var/log/lighttpd# tail -f -n 20 error.log
2021-02-22 15:43:48: (log.c.217) server started
2021-02-22 15:43:48: (server.c.1295) WARNING: unknown config-key: dir-listing.active (ignored)
2021-02-22 16:02:55: (connections-glue.c.166) SSL: -1 5 0 Success
2021-02-22 16:03:10: (connections-glue.c.166) SSL: -1 5 0 Success
2021-02-22 16:03:33: (connections-glue.c.166) SSL: -1 5 0 Success




root@DietPi:/var/log# tail -f -n 20 php7.3-fpm.log
[22-Feb-2021 15:43:47] NOTICE: fpm is running, pid 1336
[22-Feb-2021 15:43:47] NOTICE: ready to handle connections
[22-Feb-2021 15:43:47] NOTICE: systemd monitor interval set to 10000ms
2

Hi,

first, the link to the directories are correct. It indicate you have moved dietpi userdata to an usb device :wink:

root@DietPi:~# readlink /var/lib/mysql
/mnt/dietpi_userdata/mysql

root@DietPi:~# readlink -f /var/lib/mysql
/mnt/usb_1/dietpi_userdata/mysql

Ok back to your issue:

Jellyfin is using it’s own web server running on it’s own port. Therefore lighttpd is not used on Jellyfin.

For Nextcloud, as there are no issues indicated on the usual log files, can you have a look to nextcloud.log. It’s contains quite a strange format and might be hard to read. Probably you are able to upload the file to the board :slight_smile:

cat /mnt/dietpi_userdata/nextcloud_data/nextcloud.log
3

Hi,

thanks for listening!

And for the hint- this is the last strip of nextcloud.log.
There is one complain “Is the matching PHP module installed and enabled?”, so I guess, this match has been broken after the apt-upgrade.
From what I could find out, this seems to be a php memcache issue.

My research has been too ambiguous to find a fix.
Dietpi seems to run php 7.3 while I cen see, there are 7.4 and 8.0 directories in /etc/php/ too.
7.3 has much more mods-availabe than 7.4 and 8.0
Do you know the correct binding screw to fix this?

Thanks, Frans

{“reqId”:“TsKqDJL1cd36uXmUZECI”,“level”:3,“time”:“2021-02-23T06:47:07+00:00”,“remoteAddr”:“90.12.238.32”,“user”:"–",“app”:“core”,“method”:“GET”,“url”:"/nextcloud/",“message”:{“Exception”:“OC\HintException”,“Message”:“Memcache \OC\Memcache\APCu not available for local cache”,“Code”:0,“Trace”:[{“file”:"/var/www/nextcloud/lib/private/Server.php",“line”:669,“function”:"__construct",“class”:“OC\Memcache\Factory”,“type”:"->",“args”:[“48e14acda47afa5249382931bde9c8c7”,{“class”:“OC\Log”},"\OC\Memcache\APCu","\OC\Memcache\APCu","\OC\Memcache\Redis"]},{“file”:"/var/www/nextcloud/3rdparty/pimple/pimple/src/Pimple/Container.php",“line”:118,“function”:“OC\{closure}”,“class”:“OC\Server”,“type”:"->",“args”:["*** sensitive parameters replaced "]},{“file”:"/var/www/nextcloud/lib/private/ServerContainer.php",“line”:124,“function”:“offsetGet”,“class”:“Pimple\Container”,“type”:"->",“args”:[“OC\Memcache\Factory”]},{“file”:"/var/www/nextcloud/lib/private/Server.php",“line”:1700,“function”:“query”,“class”:“OC\ServerContainer”,“type”:"->",“args”:[“OC\Memcache\Factory”]},{“file”:"/var/www/nextcloud/lib/private/Server.php",“line”:633,“function”:“getMemCacheFactory”,“class”:“OC\Server”,“type”:"->",“args”:[]},{“file”:"/var/www/nextcloud/3rdparty/pimple/pimple/src/Pimple/Container.php",“line”:118,“function”:“OC\{closure}”,“class”:“OC\Server”,“type”:"->",“args”:[" sensitive parameters replaced "]},{“file”:"/var/www/nextcloud/lib/private/ServerContainer.php",“line”:124,“function”:“offsetGet”,“class”:“Pimple\Container”,“type”:"->",“args”:[“OCP\IURLGenerator”]},{“file”:"/var/www/nextcloud/lib/private/Server.php",“line”:1673,“function”:“query”,“class”:“OC\ServerContainer”,“type”:"->",“args”:[“OCP\IURLGenerator”]},{“file”:"/var/www/nextcloud/lib/private/Server.php",“line”:1148,“function”:“getURLGenerator”,“class”:“OC\Server”,“type”:"->",“args”:[]},{“file”:"/var/www/nextcloud/3rdparty/pimple/pimple/src/Pimple/Container.php",“line”:118,“function”:“OC\{closure}”,“class”:“OC\Server”,“type”:"->",“args”:[" sensitive parameters replaced ***"]},{“file”:"/var/www/nextcloud/lib/private/ServerContainer.php",“line”:124,“function”:“offsetGet”,“class”:“Pimple\Container”,“type”:"->",“args”:[“OC\Template\JSCombiner”]},{“file”:"/var/www/nextcloud/lib/private/TemplateLayout.php",“line”:344,“function”:“query”,“class”:“OC\ServerContainer”,“type”:"->",“args”:[“OC\Template\JSCombiner”]},{“file”:"/var/www/nextcloud/lib/private/TemplateLayout.php",“line”:173,“function”:“findJavascriptFiles”,“class”:“OC\TemplateLayout”,“type”:"::",“args”:[[“core/js/dist/main”,“js/merged-template-prepend”,“search/js/search”,“core/l10n/de_DE”,“search/l10n/de_DE”,“search/js/searchprovider”,“js/files/fileinfo”,“js/files/client”]]},{“file”:"/var/www/nextcloud/lib/private/legacy/OC_Template.php",“line”:185,“function”:"__construct",“class”:“OC\TemplateLayout”,“type”:"->",“args”:[“error”,""]},{“file”:"/var/www/nextcloud/lib/private/Template/Base.php",“line”:132,“function”:“fetchPage”,“class”:“OC_Template”,“type”:"->",“args”:[]},{“file”:"/var/www/nextcloud/lib/private/legacy/OC_Template.php",“line”:334,“function”:“printPage”,“class”:“OC\Template\Base”,“type”:"->",“args”:[]},{“file”:"/var/www/nextcloud/index.php",“line”:55,“function”:“printExceptionErrorPage”,“class”:“OC_Template”,“type”:"::",“args”:[{“class”:“OC\HintException”},500]}],“File”:"/var/www/nextcloud/lib/private/Memcache/Factory.php",“Line”:98,“Hint”:“Is the matching PHP module installed and enabled?”,“CustomMessage”:"–"},“userAgent”:“Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/88.0.4324.96 Safari/537.36”,“version”:“19.0.5.2”}
{“reqId”:“MY1cljFV3onq1xx4grRA”,“level”:3,“time”:“2021-02-23T06:47:11+00:00”,“remoteAddr”:“90.12.238.32”,“user”:"–",“app”:“PHP”,“method”:“PROPFIND”,“url”:"/nextcloud/remote.php/dav/files/xyz/",“message”:“Error: Class ‘OCA\DAV\Connector\Sabre\ExceptionLoggerPlugin’ not found at /var/www/nextcloud/remote.php#61”,“userAgent”:“Mozilla/5.0 (Macintosh) mirall/3.1.1git (build 4316) (Nextcloud)”,“version”:“19.0.5.2”}

4

usually there should be a folder for 7.3 only if I’m not mistaken.

root@DietPi3:/etc/php# ls -la
total 12
drwxr-xr-x  3 root root 4096 Feb 22 14:41 .
drwxr-xr-x 70 root root 4096 Feb 23 13:11 ..
drwxr-xr-x  5 root root 4096 Feb 22 14:41 7.3
root@DietPi3:/etc/php#

MichaIng
can you have a look pls as well

5

root@DietPi:~# ls -la /etc/php
total 36
drwxr-xr-x 9 root root 4096 Feb 22 10:32 .
drwxr-xr-x 86 root root 4096 Feb 22 10:43 …
drwxr-xr-x 3 root root 4096 Feb 22 10:32 5.6
drwxr-xr-x 5 root root 4096 Feb 17 2019 7.0
drwxr-xr-x 3 root root 4096 May 18 2019 7.1
drwxr-xr-x 3 root root 4096 May 18 2019 7.2
drwxr-xr-x 5 root root 4096 May 18 2019 7.3
drwxr-xr-x 3 root root 4096 Aug 12 2020 7.4
drwxr-xr-x 3 root root 4096 Feb 22 10:32 8.0

6

interesting, you installed version 5.6 same time as 8.0. At least if I’m looking to date/time folders are created/changed :thinking:

If I’m not mistaken, PHP8 is not yet available on Debian Buster and/or Bullseye repo. Quite interesting how that show up on your system.

drwxr-xr-x 3 root root 4096 Feb 22 10:32 5.6
drwxr-xr-x 3 root root 4096 Feb 22 10:32 8.0

can you check what versions are installed on your system

dpkg -l php*fpm
7

Debian Stretch systems have Ondrej’s PHP repository added, which serves all PHP versions from 5.6 til 8.0. The additional directories have been created by the php-apcu (and probably php-redis) package, which was by design one package for all PHP versions. Meanwhile, it has been split, so to get rid of that few obsolete files:

apt install php7.3-apcu
apt purge php-apcu
phpenmod apcu

And in case the same for the other modules found inside those additional dirs.

8

Hi,

this is the output of dpkg:

dpkg -l php*fpm
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name Version Architecture Description
++±==================================-======================-======================-========================================================================
un php-fpm (no description available)
ii php7.3-fpm 7.3.27-7+0~20210220.80 arm64 server-side, HTML-embedded scripting language (FPM-CGI binary)


apt install php7.3-apcu
apt purge php-apcu
phpenmod apcu

fixed the internal server error.
Nextcloud is up and running again :smiley:

Thank you Joulinar & MichaIng for fixing my machine.
I better take more care when using apt upgrade.

Thanks & regards
Frans

9

MichaIng
Thx for sharing, didn’t know that

fhals
pls keep in mind you are still running Debian Stretch. This might be a dead end road for NextCloud. There was a report about not being able to update to NextCloud 21 anymore as MariaDB is on a to low version and you will not be able to updated MariaDB on Stretch.

10

Does this mean, there is no future Nextcloud upgrade path on DietPi?

11

This has nothing to do with DietPi directly. You are running an older image version of Debian Stretch as base OS. DietPi is just some scripting on top :wink:

But to verify, simply check on Nextcloud for an available update. The last days version 21 has been released.
At least on a DEMO system, running Stretch, I was able to install NextCloud 21 without issues.

12

Thank you for the clarification.
I will post any success here :wink:

13

Yes, officially the MariaDB version shipped with Debian Stretch is not supported anymore by Nextcloud 21, but it’s still possible to install it. The admin panel shows the somehow wrong info then :smiley::

MariaDB version “10.1.48-MariaDB-0+deb9u1” is used. Nextcloud 21 will no longer support this version and requires MariaDB 10.2 or higher.

When installing NC20, and switching to beta branch (as it’s rolled out in waves), NC21 is offered by the web UI and run through nicely.

Means, for now we can keep allowing fresh Nextcloud installs on Stretch, but I guess with NC22 at least we need to hardcode a lower version for Stretch.