Nextcloud Installation

Having issues with your DietPi installation, or, found a bug? Post it here.
twentyninehairs
Posts: 14
Joined: Wed Feb 01, 2017 9:38 am

Nextcloud Installation

Post by twentyninehairs »

I used the menu interface to install nextcloud with apache. It seems that apache does not have a prefork module installed, and I have an issue with forked processes consuming all my cpu/memory causing the server to crash.

Was this installation intended to be installed with lighttp?

Why wouldn't a prefork module be installed? Do I simply need to install and configure? I don't mind modifying my system, I'm just asking for a best known method to resolve a runaway apache prefork issue. Just trying to avoid pitfalls in the interest of saving time.
twentyninehairs
Posts: 14
Joined: Wed Feb 01, 2017 9:38 am

Re: Nextcloud Installation

Post by twentyninehairs »

With the little time I had over the last couple days, I poked at it a little, and found that the prefork module is actually enabled.

The reason I thought it was not installed is due to the following output:

Code: Select all

root@nextcloudpi:/etc/apache2/conf-available# apachectl -l                                                                    
Compiled in modules:                                                                                                          
  core.c                                                                                                                      
  mod_so.c                                                                                                                    
  mod_watchdog.c                                                                                                              
  http_core.c                                                                                                                 
  mod_log_config.c                                                                                                            
  mod_logio.c                                                                                                                 
  mod_version.c                                                                                                               
  mod_unixd.c
I ran some command, (I don't remember what it was), and found that it was actually installed, but I guess it isn't listed here. I'm not sure why.

After researching the prefork subject, and reading from this article here, that mpm_worker gives the best performance and memory useage, I disabled php7.0, then enabled mpm_worker, then attempted to enable php7.0 again.

It appears that php7.0 will not run with the installed version of mpm_worker?

Code: Select all

root@nextcloudpi:~# a2enmod php7.0                                                                                            
Considering dependency mpm_prefork for php7.0:                                                                                
Considering conflict mpm_event for mpm_prefork:                                                                               
Considering conflict mpm_worker for mpm_prefork:                                                                              
ERROR: Module mpm_worker is enabled - cannot proceed due to conflicts. It needs to be disabled first!                         
ERROR: Could not enable dependency mpm_prefork for php7.0, aborting
So I re-enabled mpm_prefork with php7.0 just exactly the way it was in the original installation.

After learning best configuration practices from here, I created the file: /etc/apache2/conf-available/performancetune.conf with the following contents:

Code: Select all

<IfModule mpm_prefork_module>                                                                                                                                 
StartServers 2                                                                                                                                                
MinSpareServers 2                                                                                                                                             
MaxSpareServers 5                                                                                                                                             
MaxClients 150                                                                                                                                                
#must be customized                                                                                                                                           
ServerLimit 20                                                                                                                                                
#must be customized                                                                                                                                           
MaxRequestsPerChild 50                                                                                                                                        
</IfModule>                                                                                                                                                   
                                                                                                                                                              
KeepAlive Off
Then enabled that configuration:

Code: Select all

a2enconf performancetune
I had another issue, however. It seems that nextcloud does not have any method to back off incoming data when clients are sending streams faster than what the storage medium can handle? The server would also crash due to too much throughput. I changed the nextcloud.conf file to contain the following:

Code: Select all

<IfModule mod_bw.c>                                                                                                                                           
BandwidthModule On                                                                                                                                            
ForceBandWidthModule On                                                                                                                                       
#Overall Limit to 4Mb                                                                                                                                         
Bandwidth all "4194304"                                                                                                                                       
MaxConnection all "400"                                                                                                                                       
#any files over 1mb will be limited to 500kb/s                                                                                                                
LargeFileLimit * 1024 512000                                                                                                                                  
BandWidthError 510                                                                                                                                            
Alias /nextcloud "/var/www/nextcloud/"                                                                                                                        
                                                                                                                                                              
<Directory /var/www/nextcloud/>                                                                                                                               
        Options +FollowSymlinks                                                                                                                               
        AllowOverride All                                                                                                                                     
                                                                                                                                                              
        <IfModule mod_dav.c>                                                                                                                                  
                Dav off                                                                                                                                       
        </IfModule>                                                                                                                                           
                                                                                                                                                              
        SetEnv HOME /var/www/nextcloud                                                                                                                        
        SetEnv HTTP_HOME /var/www/nextcloud                                                                                                                   
                                                                                                                                                              
        # Hard coding 128M OPCache size, only for /nextcloud, to suppress warning on nextcloud admin panel.                                                   
        php_admin_value opcache.memory_consumption 128                                                                                                        
                                                                                                                                                              
</Directory>                                                                                                                                                  
</IfModule>
I'm guessing that's probably not the best method of using the bandwidth module, but it seems to be working extremely well. I think my server is now behaving, these settings made a massive improvement. I will probably need to tweak them some as these values were just straight up guesses. I'll run some hardcore client syncs over the next couple days to see how it does.

If anybody has input, please share.
User avatar
MichaIng
Site Admin
Posts: 2425
Joined: Sat Nov 18, 2017 6:21 pm

Re: Nextcloud Installation

Post by MichaIng »

@twentyninehairs

May I ask which machine/Debian version you are using, as detailed tweaking as you do, significantly depend on the system. Generally everything should work just fine by using the default configuration provided by DietPi. For low RAM (SBC) devices we already tweak settings to reduce recourse requirements a way that does not risk usual tasks/behaviour.

The default webserver with DietPi is indeed Lighttpd, but you can choose another within dietpi-software before installing Nextcloud or manually choose another webserver stack for installation first, and install Nextcloud afterwards.

I am not too familiar with other Apache mpm modules, but mpm-prefork is in any known case the default together with mod-php, which should have a reason that might or might not work for you.

If you wan't a different, in cases less memory consuming, method of running PHP, try Lighttpd or Nginx with php-fpm (which is default with related DietPi webserver stacks).

So far best wishes and (still) merry Christmas and happy new year :D !
twentyninehairs
Posts: 14
Joined: Wed Feb 01, 2017 9:38 am

Re: Nextcloud Installation

Post by twentyninehairs »

Thank you for your reply! I had a very merry Christmas, I hope you did as well! :D

I'm running the default dietpi image for a Raspberry Pi 3, new installation. Uses Debian Stretch, but I suppose you know that. I selected Apache from the menu before running the nextcloud installation because I read it was more compatible in a few different non-dietpi related forums. I don't know how compatible lighhtpd is, and they probably didn't either, but I selected Apache because I thought I might have less issues.

These posted settings still cause too much I/O overhead for my installation and need more tweaking if I'm going to run it this way. These settings vastly reduced that problem, but it is still too high. I didn't fully troubleshoot yet, but I think maybe Large file transfers that span a very long period of time still appear to probably be starting too many threads. At any rate there is still high CPU I/O wait time for large long file transfers.

For a small home installation, is there any downside that you know of to using lighttpd? This is for something like 6-12 users, and I suppose I will keep the bandwidth down to around 2-4Mb/s. But some files I need to use it to transfer are maybe 60Gb in size or more. I use it for small files also, but big files are definitely transferring over a long period of time.

I've never used lighttpd previously. Do you think it would have less issues with this type of setup?

Do you know if there would be issues using the dietpi menu to switch between lighttpd and apache2 to compare the two? I'm sure I would need to rerun the nextcloud install script after switching a web server for the first time using the menu?
User avatar
MichaIng
Site Admin
Posts: 2425
Joined: Sat Nov 18, 2017 6:21 pm

Re: Nextcloud Installation

Post by MichaIng »

@twentyninehairs
Yeah, we also enjoyed Christmas here :)!

About Lighttpd:
- One thing is, that we have not yet an optimized config for ownCloud/Nextcloud for this, as someone with deeper Lighttpd knowledge would be helpful. So some incompatible modules (dav) and redirections could be missing. But in comparison to Nginx, where much adjustment is necessary, on Lighttpd ownCloud+Nextcloud do run out of the box, as far as I could test (production server is on Apache). Also never heard of an issue from others.
- As it uses php-fpm like Nginx, performance and memory consumption should be moreless the same.
- Lighttpd is not officially supported/recommended, as it is listed as not fully compatible at sabre.io, the dav backend of Nextcloud: http://sabre.io/dav/webservers/ But webdav, using windows Nextcloud client, as well as Thunderbird Lightning as caldav client worked on my tests.
- I would suggest to test it with the Nextcloud apps/features you/your users need.
- Consider also Nginx, as it developed faster the last years, supports more features/modules and overtook Lighttpd in some benchmarks in performance. But the difference should be small, if any, as said, as both use php-fpm.

About switching webserver:
- Just did it for some reasons on test systems. You can just uninstall only the webserver, using dietpi-software, choose the other one and then do "dietpi-software reinstall 114" to reinstall Nextcloud. This will on existing instances just install the other webserver specific config.
- Theoretically you should be also able to install two webservers at once and activate/inactive them: service apache2 start/stop, service php7.0-fpm start/stop + service nginx start/stop, but yeah, it's all about testing :D.
twentyninehairs
Posts: 14
Joined: Wed Feb 01, 2017 9:38 am

Re: Nextcloud Installation

Post by twentyninehairs »

@MichaIng

Sorry for the late replies. This is for my home network, and I work on it when I can. I use Zabbix network monitoring software to monitor my network performance, and understand what is happening.

I can see that the problems with Apache really are not that the Apache server is too heavy to handle the load with a small number of users 8-12 limiting bandwidth. With my testing, the Apache server appears to be able to easily handle the load using a very small fraction of system resources of the Raspberry Pi 3. It appears that the default settings allow the Apache server to far outstrip hardware capacity. I believe I can continue to tweak the Apache config making a very robust server, but I'm not opposed to using another type of web server.

Hard Drive I/O throughput might be the major bottleneck for this device. It's mostly and issue of CPU cycles wasted on disk writes. In my case, I'm using a 64GB flash disk drive for the Maria database (and dietpi user data, large to mitigate RW flash wear leveling issues) and a 4TB USB HD as the main storage for the user data. Both formatted to EXT4.

I reduced the resources given to Apache to the following contents in my performancetune.conf file:

<IfModule mpm_prefork_module>
StartServers 1
MinSpareServers 1
MaxSpareServers 2
MaxClients 50
#must be customized
ServerLimit 3
#must be customized
MaxRequestsPerChild 100
</IfModule>

KeepAlive Off

With these reduced settings, I can see that disk I/O to the MariaDB on the USB stick probably poses a minor issue with my installation. The server is very responsive with file transfers averaging about 2.4mb/sec. But not quite 1/4 CPU usage is consumed by the CPU waiting for disk I/O to the maria DB.

Also, there seems to be very brief periods that the Apache server attempts to write large chunks of data to the main data storage drive all at once. When large files syncing through the client occur, the client transfers a large chunk of the file for a while to the server for a while, then the server crashes for about 10 minutes with very high disk reads and writes to the main storage directory. Hard disk utilization spikes to levels between 15mb/sec and 13mb/sec. At this time, the client errors stating that no connection to the server is available no data is transferring to/from the server. and webapp is non-responsive. Disk I/O time spent on each operation spikes to flatline at 50ms. After about 10 minutes of this, the server recovers to do it all over again. I don't think those files ever actually transfer.

I'm not sure that the bandwidth module settings I listed previously are working properly. When I reduce the bandwidth setting to what I thought would be 1Mb: 'Bandwidth all "1048576"', and files over 1Mb to be limited to 100kb: 'LargeFileLimit * 1024 102400', I didn't see any relevant changes in the file transfer rates from clients.

Do you have issues syncing using a windows client with large files? Maybe files >2Gb?
twentyninehairs
Posts: 14
Joined: Wed Feb 01, 2017 9:38 am

Re: Nextcloud Installation

Post by twentyninehairs »

@MichaIng

Upon further analysis, I realized that when there are problems, both the read and write speed to the storage
hard drive is spiking to 15-13mb per second, the disk I/O time completely flatlines at 50ms, and the network I/O traffic pattern exactly inversely matches the Disk I/O traffic.

I believe the maximum practical throughput speed of the USB bus on the Pi should be around 30mb/sec. So that means that throughput to that storage HD is hitting the maximum throughput headroom of that USB bus. Other things are using that same bus as well likely accounting for the small percentage of IO when running at 13mb/sec. If I'm not mistaken, I believe the ethernet adapter is connected to the same USB bus the USB ports are connected to. There is other disk I/O happening at the same time as well. This would point to most all the headroom of that USB bus is probably taken by writing a single file to the storage drive. The server is probably becoming non-responsive for a long period of maybe 10 minutes because Ethernet traffic is crowded out of that bus. I could see this causing corruption of some filesystems or maybe some databases if I/O to other drives time out waiting for a response while in the middle of writing to a file.

I think the solution would need to be to setup a kind of QOS system for the kernel to manage the bottleneck of that USB bus that everything is tied to. I might use cgroups to throttle my main storage HD to 5mb/sec, for example. I'm not sure if other subsystems need limits as well.
User avatar
WarHawk
Posts: 636
Joined: Thu Jul 20, 2017 8:55 am

Re: Nextcloud Installation

Post by WarHawk »

Yeah, Nextcloud on a SBC is a bit overkill...it does take alot of oomph to run it..but if you are patient, it can perform well enough for photo backups on phones

there is also a way to pre-generate all the thumbnails...but it must be run thru a command shell (get it in the apps tab under admin)
https://github.com/rullzer/PreviewGenerator

sudo -u www-data /var/www/nextcloud//occ preview:pre-generate

It will take some time for your little processor to chop thru and make the thumbnails...then it's MUCH faster on updating the webpage

They say it should be made a cronjob every 10 min...but I have failed to put mine in crontab effectively
anyone want to help?
twentyninehairs
Posts: 14
Joined: Wed Feb 01, 2017 9:38 am

Re: Nextcloud Installation

Post by twentyninehairs »

@WarHawk If you installed nextcloud with the dietpi install script, you only need to use the command:

Code: Select all

ncc preview:pre-generate
I do not have that plugin, but I'm guessing you could do that by editing the main crontab script with the command

Code: Select all

crontab -e
Then try putting this at the end of the file, and save it.

Code: Select all

*/10 * * * * ncc preview:pre-generate
Please don't confuse the issue I'm trying to solve, however. The problem I'm having at the moment is that a windows client cannot sync large files to the server at all due to USB hard drive I/O saturating the USB bus.

It doesn't sync large files at all. I would not care if the transfers are slow, but it doesn't work at all. I would really like to solve this. When large chunks of files are written to a USB hard drive, it looks like the ethernet adapter is choked out of packets through that bus, and clients cannot access the server while those chunks are written. So a client will timeout attempting to communicate about that file being uploaded, and disconnects. Then when the server is finished writing a chunk, it becomes available again, and the client starts over attempting to send the same file. It never completes sending that file.

I'm trying to keep the USB bus from saturating by imposing I/O limits. Digging into this further today, I found that dietpi is already using cgroups for some I/O.

Does anybody have any suggestions to limit I/O to a USB disk? I do not have any experience with cgroups, so I will have some difficulty knowing what best practices would be to modify a system already in place.
twentyninehairs
Posts: 14
Joined: Wed Feb 01, 2017 9:38 am

Re: Nextcloud Installation

Post by twentyninehairs »

@WarHawk I might also recommend installing Webmin. Dietpi has it as an optional install. If you disable background data collection in the settings, Webmin doesn't take much resources to run, maybe around 40meg of memory. This utility makes managing a linux system a lot easier, and has a GUI for managing CRON jobs. Well worth it in my opionin.

@MichaIng Back to troubleshooting my large file sync issue, I mucked with the cgroup system, and I think I made some headway, but I caused my server to crash and loose all access to storage drives at one point. I think I understand what to do, but I don't know if using cgroups to set hard drive I/O to a specific throughput is a good answer, because it is slowing down performance to a set number. I wonder if ionice might be something to look into?

Anyway, I realized that this must have been happening when the server was attempting to piece together several large file chunks together. That explains why the duration was so long, and both the read and write speeds were maxed out.

I found here that the following setting can be placed in the nextcloud config.php file:

Code: Select all

'part_file_in_storage' => false,
According to that documentation, Nextcloud stores part files created during upload in the same storage location as the upload target. Setting this to false will store the part files in the root of the user folder.

My user folder and maraia database are on a 64Gb USB flash drive that is apparently not capable of saturating the USB BUS. It's 64Gb to mitigate write wear leveling issues.

This is not a particularly good fix because it doesn't eliminate the root cause of the USB bus saturation problem. It was a good enough to break up the USB bus I/O saturation issue and keep my system responsive. It now completes uploading large files. I think the UI took minor performance hit when I made this change, though.

The Apache Bandwidth module is working also, it just doesn't always hold to the numbers I think it should. There is definitely a massive change in data throughput, and things sometimes break when I shut it off, so it's definitely doing it's thing. I'm still tweaking parameters to optimize performance. It seems that the bandwidth settings can be globally loaded by themselves, so I created a file called bw.conf in the conf-available and put all my settings in that. I'll post those settings once I dial them in from test runs.

All in all, I'm very happy with Apache and Nextcloud. Apache is definitely not too heavy for the Pi, it just needs the parameters reigned in to acceptable levels. If I ever get all this straightened out, I can post a tutorial with configuration changes, if you would like.
Post Reply