Nextcloud Installation

Having issues with your DietPi installation, or, found a bug? Post it here.
twentyninehairs
Posts: 14
Joined: Wed Feb 01, 2017 9:38 am

Re: Nextcloud Installation

Post by twentyninehairs »

Large file transfers are still taking the server down for something like 10min at a time while part files are combining. All those files are completing, and nothing is ever corrupted, but I still don't like that my server is sometimes down.

The Pi's core processing capabilities are more than powerful enough to run Nextcloud, the only problem with it really is the USB bus bottleneck. I never tested lighttpd, but I don't see how that would change this issue.

I tried using WiFi instead of a wired ethernet connection thinking maybe that is on a different bus. There is too much wireless noise where I live to keep a constant connection on the 2.4Ghz range even when the Pi is sitting next to the access point. According to the tests I ran, it didn't seem to help, but I was not really able to run adequate testing due to all the noise. Sad, but true.

I can't see any way of fixing this without decreasing transfer speed to/from the storage hard drive. I thought about maybe connecting the hard drive to a USB 1.0 hub, then connecting that hub to a USB 2.0 hub, and connecting the 2.0 hub to the Pi. I believe that should reduce transfer speed to around 800k if it even works at all, but I think that is too slow. I'm comfortable with a minimum of 2mb/sec with 4-8mb/sec ideal. Using cgroups is the only other solution I can think of, and I haven't been able to implement that within a reasonable amount of time.

So, I think I am going to give up on the Pi, and use an ODROID-C2 instead.
User avatar
WarHawk
Posts: 608
Joined: Thu Jul 20, 2017 8:55 am

Re: Nextcloud Installation

Post by WarHawk »

twentyninehairs wrote: Tue Jan 09, 2018 3:36 am @WarHawk If you installed nextcloud with the dietpi install script, you only need to use the command:

Code: Select all

ncc preview:pre-generate
I do not have that plugin, but I'm guessing you could do that by editing the main crontab script with the command

Code: Select all

crontab -e
Then try putting this at the end of the file, and save it.

Code: Select all

*/10 * * * * ncc preview:pre-generate
Please don't confuse the issue I'm trying to solve, however. The problem I'm having at the moment is that a windows client cannot sync large files to the server at all due to USB hard drive I/O saturating the USB bus.

It doesn't sync large files at all. I would not care if the transfers are slow, but it doesn't work at all. I would really like to solve this. When large chunks of files are written to a USB hard drive, it looks like the ethernet adapter is choked out of packets through that bus, and clients cannot access the server while those chunks are written. So a client will timeout attempting to communicate about that file being uploaded, and disconnects. Then when the server is finished writing a chunk, it becomes available again, and the client starts over attempting to send the same file. It never completes sending that file.

I'm trying to keep the USB bus from saturating by imposing I/O limits. Digging into this further today, I found that dietpi is already using cgroups for some I/O.

Does anybody have any suggestions to limit I/O to a USB disk? I do not have any experience with cgroups, so I will have some difficulty knowing what best practices would be to modify a system already in place.
Nice...I had to install the Preview Generator app from the admin apps page(I don't think it's installed stock on the DietPi install)

When I ran the

Code: Select all

occ preview:pre-generate
it would run for a second...then stop

I got it to go thru by installing screen, then running:

Code: Select all

occ preview:generate-all
in the /var/www/nextcloud directory (here occ resides)
Then detatch from the screen session and let it run in the background
It has taken a LOOOOOOOOOOOONG time to go thru several hundred gigs of photos...
User avatar
WarHawk
Posts: 608
Joined: Thu Jul 20, 2017 8:55 am

Re: Nextcloud Installation

Post by WarHawk »

I think I it to work without crashing

Code: Select all

sudo -u www-data php /var/www/nextcloud/occ preview:pre-generate
Guess I will add

Code: Select all

*/10     * * * * www-data php /var/www/nextcloud/occ preview:pre-generate
to /etc/crontab
User avatar
MichaIng
Site Admin
Posts: 2293
Joined: Sat Nov 18, 2017 6:21 pm

Re: Nextcloud Installation

Post by MichaIng »

@WarHawk
Note that with Nextcloud on DietPi you can use ncc, which is a globals function placed with DietPi as shortcut for: sudo -u www-data php /var/www/nextcloud/occ

With ownCloud, the shortcut is occ. Those are different to allow ownCloud and Nextcloud being installed in parallel ;).

But AFAIK these global functions are not available within crontab. Cron is executed in a non-interactive sub shell, thus /etc/profile and /etc/bash.bashrc are not sourced there, which is required for ncc/occ the be available. So there you need to use the full sudo + command path, as you did.

Instead of adding the job directly to /etc/crontab, I suggest you add it directly to www-data users crontab, to skip using sudo:
crontab -u www-data -e
There add:
*/10 * * * * php /var/www/nextcloud/occ preview:pre-generate

E.g. this is my www-data crontab:

Code: Select all

2018-12-08 01:21:38 root@micha:/var/log# crontab -u www-data -l
*/15 * * * * php /var/www/nextcloud/cron.php
10 * * * * php /var/www/nextcloud/occ preview:pre-generate -vvv &>> /var/log/micha-pre-generate.log
Nextcloud cron job is added by DietPi-Software installer. I just run pre-generation every hour and do debug logging, from ancient time to debug/test preview generation times and such ;).
User avatar
WarHawk
Posts: 608
Joined: Thu Jul 20, 2017 8:55 am

Re: Nextcloud Installation

Post by WarHawk »

Seems I am getting the same timout as they were in the threads

Does the preview:generate-all delete all the previews and re-generate them?

Uggh...I will figure this darn thing out...LOL
User avatar
MichaIng
Site Admin
Posts: 2293
Joined: Sat Nov 18, 2017 6:21 pm

Re: Nextcloud Installation

Post by MichaIng »

preview:generate-all does not delete any previews, AFAIK, it just re-scans all files and "requests" all previews for them. The internal (core, not app) generator then generates them, if not existent.

So the app just does the same, recursively to all files, that would happen, if you manually browse through all directories/files open details, gallery app and such, to force Nextcloud core to generate previews in all usual sizes/ratios.
Seems I am getting the same timout as they were in the threads
You mean the cron job just stops, even it has not yet finished? Not 100% sure about PHP timeouts, but not that you need to edit the ini file in /etc/php/7.0/cli/php.ini|conf.d to apply changes for occ/cron.php commands.
User avatar
WarHawk
Posts: 608
Joined: Thu Jul 20, 2017 8:55 am

Re: Nextcloud Installation

Post by WarHawk »

MichaIng wrote: Mon Dec 10, 2018 2:15 pm preview:generate-all does not delete any previews, AFAIK, it just re-scans all files and "requests" all previews for them. The internal (core, not app) generator then generates them, if not existent.

So the app just does the same, recursively to all files, that would happen, if you manually browse through all directories/files open details, gallery app and such, to force Nextcloud core to generate previews in all usual sizes/ratios.
Seems I am getting the same timout as they were in the threads
You mean the cron job just stops, even it has not yet finished? Not 100% sure about PHP timeouts, but not that you need to edit the ini file in /etc/php/7.0/cli/php.ini|conf.d to apply changes for occ/cron.php commands.
Nice...because I know the preview:generate-all is working...I even have it set to -vvv output to a file and I can see it slowly chopping thru...it does have to go thru over 300gigs of files and whatnot so it's reaaaaaaly slow...I guess once it generates them I can then set it to go thru every 10 minutes...if I did it now...it would have multiple instances and lock up the OPiPC...so far it's doing well...but with the preview:pre-generate stops after about 5 seconds of running (seems like the developer also had issues with it as well)
As long as it doesn't re-generate all of them only parses what has been done and continues on...then I think the preview:generate-all is the way I should go.

The CRON job I added to run the preview:pre-generate stops, I never see it going in htop, but when I setup the preview:generate-all it runs and runs and runs...but if I have it set to start too soon, it ends up making multiple threads and bogs the SBC down until it locks up...so I set it to start only once a day until it chops thru the initial generation of the files

crontab -e -u www-data

Code: Select all

0 22  * * * php /var/www/nextcloud/occ preview:generate-all -vvv &>> /var/log/nextcloud/files.created.txt
All I can say is, it's MUCH faster loading the pages with the previews pre-generated...

Do you have your's set for AJAX or CRON?
User avatar
MichaIng
Site Admin
Posts: 2293
Joined: Sat Nov 18, 2017 6:21 pm

Re: Nextcloud Installation

Post by MichaIng »

generate-all will be definitely slower, since it re-scans all files. pre-generate will only walk through the files that were added since last execution, so should be much faster.

But yeah, if there is no solution to have it running more than 5 minutes, that's a problem. Although, since you run it every 10 minutes, even if you add too much images at once, to complete in 5 minutes, the second or third cron execution will do ;).

Just checked my logs: It runs longer then 5 minutes in my case, if I add much pictures. Which webserver you use? Lighttpd with php-fpm (default on DietPi)?
I use Apache with mod_php, so things might be different. Worth investigating how this can be solved. If found, we can add a fix to DietPi install script.
User avatar
WarHawk
Posts: 608
Joined: Thu Jul 20, 2017 8:55 am

Re: Nextcloud Installation

Post by WarHawk »

Apache2...

Not sure how to roll back to Lighttpd since it is already set up for Apache2

Correction...it's back to Cron rather than Ajax...I'll leave that alone...I have a stock install but a non-standard build of dietpi

I am running MariaDB though

nextcloud:/home/warhawk# mysql_upgrade
This installation of MySQL is already upgraded to 10.1.37-MariaDB, use --force if you still need to run mysql_upgrade
Here is everything from my "settings" page
Nextcloud
Version: 14.0.4.2
Apps installed: 35
Apps updates available: 0

PHP
Version: 7.0.30
Memory Limit: 512 MB
Max Execution Time: 3600
Upload max size: 511 MB

Database
Type: mysql
Version: 10.1.37
Size: 131.6 MB
I will say this..was having fits for a while...power failure...ended up using the dietpi-drive_manager to scan and fix my external drive...it had a few corruptions...after I did that...smooth sailing
User avatar
MichaIng
Site Admin
Posts: 2293
Joined: Sat Nov 18, 2017 6:21 pm

Re: Nextcloud Installation

Post by MichaIng »

@WarHawk
No need to switch to Lighttpd, that will just cause trouble on configs and you would need to reinstall PHP and Nextcloud as well (at least the configuration steps). As said, I use Apache here as well and did since starting with Nextcloud (ownCloud that time).

You said the developer (you mean rullzer?) has this issue as well? Do you have a link? Will run some tests here again and check PHP configs.
Post Reply