nextcloud slow upload speed Topic is solved

Having issues with your DietPi installation, or, found a bug? Post it here.
Post Reply
frezeen
Posts: 12
Joined: Thu Aug 20, 2020 4:30 pm

nextcloud slow upload speed

Post by frezeen »

hi i install dietpi + nextcloud on a pi4
i move, with dietpi drive manager the data dir to ext drive

now when i up a big file (about 50gb) from webui file is up in a temp dir somewhere on sd i think and when it is complete moved to ext drive data dir.
use of this tmp dir slow all and i need doubled space, 50gb for tmp file and 50gb for storaged file. i know tmp purge when process is complete but until then 100gb are used

2 question:
1) can i disable tmp creation and up direct to data on ext drive? (best and fast solution, dont need to move file when cmpleted in tmp dir)
2) if i cant poiint 1, can i move this tmp upload to ext data where i dont have space problem?

thanks for support, really
Last edited by frezeen on Fri Aug 21, 2020 7:28 pm, edited 1 time in total.
User avatar
Joulinar
Posts: 2287
Joined: Sat Nov 16, 2019 12:49 am

Re: nextcloud tmp dir

Post by Joulinar »

Hi,

many thanks for your request. I played a little bit with Nextcloud and your assumption on the upload process is not fully correct. During an updaload, files will be uploaded in smaller chunks. The file is not uploaded as a whole. These small files are stored on

>> /mnt/dietpi_userdata/nextcloud_data/admin/uploads/

For each upload a temp folder will be created containing the files. An example from my test system:

Code: Select all

root@DietPiVM1:/mnt/dietpi_userdata/nextcloud_data/admin/uploads/web-file-upload-......# ls -la
total 61448
drwxr-xr-x 2 www-data www-data     4096 Aug 20 21:54 .
drwxr-xr-x 3 www-data www-data     4096 Aug 20 21:51 ..
-rw-r--r-- 1 www-data www-data 10485760 Aug 20 21:51 0
-rw-r--r-- 1 www-data www-data 10485760 Aug 20 21:52 10485760
-rw-r--r-- 1 www-data www-data 10485760 Aug 20 21:53 20971520
-rw-r--r-- 1 www-data www-data 10485760 Aug 20 21:53 31457280
-rw-r--r-- 1 www-data www-data 10485760 Aug 20 21:54 41943040
-rw-r--r-- 1 www-data www-data 10485760 Aug 20 21:54 52428800
As soon as the upload finished, these files will be assembled to the final file you have been uploaded and stored on correct location.

Means, if you moved dietpi_userdata to an external HDD, these temp files are already on your disk and not on the SD Card.
Pls let us know if a solution is working. This could help others if they hit by similar situation. Your DietPi Team
frezeen
Posts: 12
Joined: Thu Aug 20, 2020 4:30 pm

Re: nextcloud tmp dir

Post by frezeen »

oh, very thanks for reply and support

i forgot to write im disabling the chunk creation for speed up upload process... speed go up from 10mb/s to 60mb/s on local upload with this command. (60mb/s is my USB pendrive write speed limit)
chunk process use lots cpu rss for pack and unpacking, when upload cpu go to 100%, with this command no cpu load on upload process

ncc config:app:set files max_chunk_size --value 0

the problem is: file is uploaded somewhere on system partition and then , when completed, moved to ext usb data folder.

my goal is to up direct to ext usb without move it at end

im sure upload is going to system partiton becouse use /dev/root free space
User avatar
Joulinar
Posts: 2287
Joined: Sat Nov 16, 2019 12:49 am

Re: nextcloud tmp dir

Post by Joulinar »

ok I set as well ncc config:app:set files max_chunk_size --value 0

Code: Select all

root@DietPiVM1:~# ncc config:app:set files max_chunk_size --value 0
Config value max_chunk_size for app files set to 0
root@DietPiVM1:~# 
Files are uploaded to /var/cache/lighttpd/uploads now

Code: Select all

root@DietPiVM1:/var/cache/lighttpd/uploads# ls -la
total 37628
drwxr-x--- 2 www-data www-data   61440 Aug 21 10:46 .
drwxr-x--- 4 www-data www-data    4096 Aug 17 22:04 ..
-rw------- 1 www-data www-data 1076020 Aug 21 10:46 lighttpd-upload-2x6olV
-rw------- 1 www-data www-data 1052660 Aug 21 10:46 lighttpd-upload-46mDkB
-rw------- 1 www-data www-data 1077340 Aug 21 10:46 lighttpd-upload-4Nnpym
-rw------- 1 www-data www-data 1062880 Aug 21 10:46 lighttpd-upload-6TyJG8
-rw------- 1 www-data www-data 1071640 Aug 21 10:46 lighttpd-upload-6xeXcw
-rw------- 1 www-data www-data 1052660 Aug 21 10:46 lighttpd-upload-8KhMt1
-rw------- 1 www-data www-data 1068720 Aug 21 10:46 lighttpd-upload-af20HA
-rw------- 1 www-data www-data 1051200 Aug 21 10:46 lighttpd-upload-AniVnL
-rw------- 1 www-data www-data 1055366 Aug 21 10:46 lighttpd-upload-Ch2cgC
-rw------- 1 www-data www-data 1049740 Aug 21 10:46 lighttpd-upload-ciBG0r
-rw------- 1 www-data www-data 1074560 Aug 21 10:46 lighttpd-upload-CKlPio
-rw------- 1 www-data www-data 1056479 Aug 21 10:46 lighttpd-upload-eV4Yg3
-rw------- 1 www-data www-data  167900 Aug 21 10:46 lighttpd-upload-g2O72c
-rw------- 1 www-data www-data 1071398 Aug 21 10:46 lighttpd-upload-ICVpeO
-rw------- 1 www-data www-data 1060906 Aug 21 10:46 lighttpd-upload-iGkZnG
-rw------- 1 www-data www-data 1051200 Aug 21 10:46 lighttpd-upload-K8MWRX
-rw------- 1 www-data www-data 1073590 Aug 21 10:46 lighttpd-upload-kAY3Uj
-rw------- 1 www-data www-data 1052660 Aug 21 10:46 lighttpd-upload-KRFiJC
-rw------- 1 www-data www-data 1059476 Aug 21 10:46 lighttpd-upload-M68VpP
-rw------- 1 www-data www-data 1049740 Aug 21 10:46 lighttpd-upload-mJQ12g
-rw------- 1 www-data www-data 1051200 Aug 21 10:46 lighttpd-upload-MzOjSA
-rw------- 1 www-data www-data 1075306 Aug 21 10:46 lighttpd-upload-OdwxHV
-rw------- 1 www-data www-data 1074072 Aug 21 10:46 lighttpd-upload-QiwoVz
-rw------- 1 www-data www-data 1054120 Aug 21 10:46 lighttpd-upload-Qp9DWE
-rw------- 1 www-data www-data 1073100 Aug 21 10:46 lighttpd-upload-qRh5OV
-rw------- 1 www-data www-data 1064340 Aug 21 10:46 lighttpd-upload-S1gMal
-rw------- 1 www-data www-data 1099380 Aug 21 10:46 lighttpd-upload-SLNoNY
-rw------- 1 www-data www-data 1049740 Aug 21 10:46 lighttpd-upload-sQJX3f
-rw------- 1 www-data www-data 1068720 Aug 21 10:46 lighttpd-upload-uaLSPW
-rw------- 1 www-data www-data 1067260 Aug 21 10:46 lighttpd-upload-uEpZ0b
-rw------- 1 www-data www-data 1059030 Aug 21 10:46 lighttpd-upload-uKROWg
-rw------- 1 www-data www-data 1049740 Aug 21 10:46 lighttpd-upload-UP3Z5X
-rw------- 1 www-data www-data 1054120 Aug 21 10:46 lighttpd-upload-WGzV9f
-rw------- 1 www-data www-data 1060710 Aug 21 10:46 lighttpd-upload-wj9UBq
-rw------- 1 www-data www-data 1052660 Aug 21 10:46 lighttpd-upload-YlVzIi
-rw------- 1 www-data www-data 1057646 Aug 21 10:46 lighttpd-upload-yXzxO4
-rw------- 1 www-data www-data 1055580 Aug 21 10:46 lighttpd-upload-YyY2QH
root@DietPiVM1:/var/cache/lighttpd/uploads#
But still in chunks ;)
Pls let us know if a solution is working. This could help others if they hit by similar situation. Your DietPi Team
frezeen
Posts: 12
Joined: Thu Aug 20, 2020 4:30 pm

Re: nextcloud tmp dir

Post by frezeen »

oh.. that si funny :lol: :lol:
anyways i dont know why, but speed is so high in my case and no cpuload with occ command, do u notice this too?

ok, how can i move that dir to external usb disk?, becosue my system partion cant manage big files, its 32gb only

want to try if i move httpd cache to ext usb ill lose the speed up boost with occ command set
User avatar
Joulinar
Posts: 2287
Joined: Sat Nov 16, 2019 12:49 am

Re: nextcloud tmp dir

Post by Joulinar »

setting is done on global lighttpd config file.

Code: Select all

root@DietPiVM1:~# cat /etc/lighttpd/lighttpd.conf |grep server.upload-dirs
server.upload-dirs          = ( "/var/cache/lighttpd/uploads" )
root@DietPiVM1:~#
Pls let us know if a solution is working. This could help others if they hit by similar situation. Your DietPi Team
frezeen
Posts: 12
Joined: Thu Aug 20, 2020 4:30 pm

Re: nextcloud tmp dir

Post by frezeen »

ok testing caching on same destination disk: upload speed is fast but moving when completed is a pain... disk need to read and write same time.

sorry if i ask yet, its clear that occ command speed up the upload but this time is ligghtpd that chunk files for caching.

my question is the same: can i up a files direct to dietpi data dir without chunk big files?
i have to disable lighttpd caching big uploaded files? can i do this?
User avatar
Joulinar
Posts: 2287
Joined: Sat Nov 16, 2019 12:49 am

Re: nextcloud tmp dir

Post by Joulinar »

sorry this I don't know. Probably something to ask on lighttpd board https://redmine.lighttpd.net/projects/lighttpd/boards
Pls let us know if a solution is working. This could help others if they hit by similar situation. Your DietPi Team
frezeen
Posts: 12
Joined: Thu Aug 20, 2020 4:30 pm

Re: nextcloud tmp dir

Post by frezeen »

ok thanks for all the reply, ill ask in lighttpd forum, i think its right place for the question.

anyways for to complete info about this problem, someone need a temp solution maybe.

my temp solution:
up a server samba, add this section to /etc/samba/smb.conf

Code: Select all

[Cloud]
	path = /mnt/dietpi_userdata/nextcloud_data/admin/files
	writeable = yes
	create mode = 0775
	valid users = dietpi
	comment = Nextcloud Share
	max connections = 8
	force group = www-data
	force user = www-data
	directory mode = 0775
add this 2 lines in /var/www/nextcloud/config/config.php

Code: Select all

  'check_data_directory_permissions' => false,
  'filesystem_check_changes' => 1,
change permission of /mnt/dietpi_userdata/nextcloud_data/ and recursive to 777

exec in dietpi shell:

Code: Select all

ncc config:app:set files max_chunk_size --value 0 && reboot
login and upload files with samba protocol at 110mb/s , full my write speed of ext disk. u can found new files on web interface in real time with a refresh page
note: this have a boost speed from 10mb/s to 65mb/s web upload too in my case

thats :)
Last edited by frezeen on Wed Aug 26, 2020 12:55 pm, edited 1 time in total.
User avatar
Joulinar
Posts: 2287
Joined: Sat Nov 16, 2019 12:49 am

Re: nextcloud slow upload speed

Post by Joulinar »

thx for sharing your solution. Uploading big data via samba is indeed a good solution.
Pls let us know if a solution is working. This could help others if they hit by similar situation. Your DietPi Team
Post Reply