Cronjob "ncc files:scan --all" not working

Troubleshooting
1

Hey,
i am using Nextcloud on Dietpi to sync files, contacts and calendars between windows/android and apple. It’s working nice, but there is an issue:

As i drop my files from windows( connected via smb) into nextcloud-folders i have to re-index the files by the command ncc files:scan --all
This works perfect in the shell with user ‘root’

So i tried to setup a cronjob by adding */2 * * * * ncc files:scan --all at the end of the crontab by command crontab -e,
for checking if the file is executed i added */2 * * * * echo $(date) > /tmp/test into the file.
As i can see on the timestamp of the testfile, the job is being executed, while the file:scan seems to be withouht effect - even by introducing the command sudo infront of ncc

The output in journalctl looks like this:

Nov 05 10:26:01 DietPi dhclient[419]: No DHCPOFFERS received.
Nov 05 10:26:01 DietPi dhclient[419]: No working leases in persistent database - sleeping.
Nov 05 10:28:01 DietPi CRON[17618]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Nov 05 10:28:01 DietPi CRON[17617]: pam_unix(cron:session): session opened for user root(uid=0) by (uid=0)
Nov 05 10:28:01 DietPi CRON[17619]: (root) CMD (echo $(date) > /tmp/test)
Nov 05 10:28:01 DietPi CRON[17620]: (root) CMD (ncc files:scan --all)
Nov 05 10:28:01 DietPi CRON[17618]: (CRON) info (No MTA installed, discarding output)
Nov 05 10:28:01 DietPi CRON[17618]: pam_unix(cron:session): session closed for user root
Nov 05 10:28:01 DietPi CRON[17617]: pam_unix(cron:session): session closed for user root

I also restarted cron by service cron restart
Adding */2 * * * * ncc files:scan --all to the crontab of user www-data didn’t help either.

I feel a simple part is missing, but i can’t figure out, what it is

Edit:
Appending

*/1 * * * * php /var/www/nextcloud/occ files:scan --all

by editing sudo -u www-data crontab -e
didn’t help either

2

ncc is a custom shell function which includes sudo -u www-data php /var/www/nextcloud/occ, so shortening that command.

In a cron job or script you instead need to run the long form, at best directly as www-data user:

crontab -u www-data -e

*/2 * * * * php /var/www/nextcloud/occ files:scan --all

Are you sure that you want to scan the whole Nextcloud data dir every 2 minutes? This job will likely stack up and cause a massive constant disk load. Instead I suggest to add the Samba server as external storage (via external storages app) to Nextcloud. And of course best is to avoid Samba completely and use the native Nextcloud client on Windows.

Nextcloud is definitely not designed to add data outside of it’s native web UI or WebDAV API (inkl. clients) on a regular basis.

3

Thank you for replying.
the crontab now looks like this:

*/5 * * * * php /var/www/nextcloud/cron.php
# */2 * * * * ncc files:scan --all
# */1 * * * * php /var/www/nextcloud/occ files:scan --all --verbose 2>&1 >> /tmp/nextcloud_rescan_debug2.log
*/1 * * * * echo $(date) > /tmp/test-wwwusr
*/1 * * * * php /var/www/nextcloud/occ files:scan --all

Seems like it didnt help at all. The testfile was created.

SMB is a workaround for the Nextcloud-app since files deployed in windows were not visible to the nextcloud-apps on android/ IPadOS

In Nextcloudpi (as OS) there was an included option to set cronjobs. Unfortunately this is missing in nextcloud via DietPi

4

You mean when you upload/change files via the official Nextcloud Windows client, those changes are not visible in official Android and iOS apps? Then the issue is on a very different level which I suggest to investigate instead of applying such a workaround (which might not work for the same reason). Are uploads/changes done via Windows client visible on the server e.g. when accessing the web interface?

For the cron job, just to check whether they run now successfully:

journalctl -u cron

Do you still see (CRON) info (No MTA installed, discarding output) output for recent calls (which would indicate an error ourput)?

Bear with us that we cannot maintain the same full CLI and web UI extensions to every software title which single-purpose OS/software projects do. If Nextcloud is all you want to use, NextcloudPi is a great choice, and I’m happy to see that it finally supports Debian/Raspbian Bullseye with a new maintainer. For the cron jobs, it will do nothing more than adding them via crontab -u www-data -e or using /etc/cron.d :wink:.

5

You mean when you upload/change files via the official Nextcloud Windows client, those changes are not visible in official Android and iOS apps?<

Yes, exactly. I wasn’t able to get data visible, when deploying it via nextcloud-windowsapp. Journal of trial/error in this thread:
https://dietpi.com/forum/t/nextcloud-folder-not-visible-in-dietpi-and-aria2/14820/41

Are uploads/changes done via Windows client visible on the server e.g. when accessing the web interface?

Yes, as long, as i create folders within Nextcloud (Apps or WebUI)

For the cron job, just to check whether they run now successfully:
journalctl -u cron

The output of the last 5 minutes:

Nov 05 13:50:02 DietPi CRON[22453]: pam_unix(cron:session): session closed for user www-data
Nov 05 13:50:02 DietPi CRON[22451]: (CRON) info (No MTA installed, discarding output)
Nov 05 13:50:02 DietPi CRON[22451]: pam_unix(cron:session): session closed for user www-data
Nov 05 13:51:01 DietPi CRON[22476]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Nov 05 13:51:01 DietPi CRON[22475]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Nov 05 13:51:01 DietPi CRON[22477]: (www-data) CMD (php /var/www/nextcloud/occ files:scan --all)
Nov 05 13:51:01 DietPi CRON[22478]: (www-data) CMD (echo $(date) > /tmp/test-wwwusr)
Nov 05 13:51:01 DietPi CRON[22476]: pam_unix(cron:session): session closed for user www-data
Nov 05 13:51:02 DietPi CRON[22475]: (CRON) info (No MTA installed, discarding output)
Nov 05 13:51:02 DietPi CRON[22475]: pam_unix(cron:session): session closed for user www-data
Nov 05 13:52:01 DietPi CRON[22493]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Nov 05 13:52:01 DietPi CRON[22492]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Nov 05 13:52:01 DietPi CRON[22495]: (www-data) CMD (echo $(date) > /tmp/test-wwwusr)
Nov 05 13:52:01 DietPi CRON[22494]: (www-data) CMD (php /var/www/nextcloud/occ files:scan --all)
Nov 05 13:52:01 DietPi CRON[22493]: pam_unix(cron:session): session closed for user www-data
Nov 05 13:52:02 DietPi CRON[22492]: (CRON) info (No MTA installed, discarding output)
Nov 05 13:52:02 DietPi CRON[22492]: pam_unix(cron:session): session closed for user www-data
Nov 05 13:53:01 DietPi CRON[22708]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Nov 05 13:53:01 DietPi CRON[22707]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Nov 05 13:53:01 DietPi CRON[22709]: (www-data) CMD (php /var/www/nextcloud/occ files:scan --all)
Nov 05 13:53:01 DietPi CRON[22710]: (www-data) CMD (echo $(date) > /tmp/test-wwwusr)
Nov 05 13:53:01 DietPi CRON[22708]: pam_unix(cron:session): session closed for user www-data
Nov 05 13:53:02 DietPi CRON[22707]: (CRON) info (No MTA installed, discarding output)
Nov 05 13:53:02 DietPi CRON[22707]: pam_unix(cron:session): session closed for user www-data
Nov 05 13:54:01 DietPi CRON[22726]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Nov 05 13:54:01 DietPi CRON[22727]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Nov 05 13:54:01 DietPi CRON[22729]: (www-data) CMD (echo $(date) > /tmp/test-wwwusr)
Nov 05 13:54:01 DietPi CRON[22728]: (www-data) CMD (php /var/www/nextcloud/occ files:scan --all)
Nov 05 13:54:01 DietPi CRON[22727]: pam_unix(cron:session): session closed for user www-data
Nov 05 13:54:02 DietPi CRON[22726]: (CRON) info (No MTA installed, discarding output)
Nov 05 13:54:02 DietPi CRON[22726]: pam_unix(cron:session): session closed for user www-data
Nov 05 13:55:01 DietPi CRON[22812]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Nov 05 13:55:01 DietPi CRON[22813]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Nov 05 13:55:01 DietPi CRON[22811]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Nov 05 13:55:01 DietPi CRON[22815]: (www-data) CMD (php /var/www/nextcloud/cron.php)
Nov 05 13:55:01 DietPi CRON[22816]: (www-data) CMD (echo $(date) > /tmp/test-wwwusr)
Nov 05 13:55:01 DietPi CRON[22814]: (www-data) CMD (php /var/www/nextcloud/occ files:scan --all)
Nov 05 13:55:01 DietPi CRON[22812]: pam_unix(cron:session): session closed for user www-data
Nov 05 13:55:02 DietPi CRON[22813]: pam_unix(cron:session): session closed for user www-data
Nov 05 13:55:03 DietPi CRON[22811]: (CRON) info (No MTA installed, discarding output)
Nov 05 13:55:03 DietPi CRON[22811]: pam_unix(cron:session): session closed for user www-data
Nov 05 13:56:01 DietPi CRON[22841]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Nov 05 13:56:01 DietPi CRON[22840]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Nov 05 13:56:01 DietPi CRON[22842]: (www-data) CMD (echo $(date) > /tmp/test-wwwusr)
Nov 05 13:56:01 DietPi CRON[22843]: (www-data) CMD (php /var/www/nextcloud/occ files:scan --all)
Nov 05 13:56:01 DietPi CRON[22841]: pam_unix(cron:session): session closed for user www-data
Nov 05 13:56:02 DietPi CRON[22840]: (CRON) info (No MTA installed, discarding output)
Nov 05 13:56:02 DietPi CRON[22840]: pam_unix(cron:session): session closed for user www-data
Nov 05 13:57:01 DietPi CRON[22862]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Nov 05 13:57:01 DietPi CRON[22861]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Nov 05 13:57:01 DietPi CRON[22863]: (www-data) CMD (php /var/www/nextcloud/occ files:scan --all)
Nov 05 13:57:01 DietPi CRON[22864]: (www-data) CMD (echo $(date) > /tmp/test-wwwusr)
Nov 05 13:57:01 DietPi CRON[22862]: pam_unix(cron:session): session closed for user www-data
Nov 05 13:57:02 DietPi CRON[22861]: (CRON) info (No MTA installed, discarding output)
Nov 05 13:57:02 DietPi CRON[22861]: pam_unix(cron:session): session closed for user www-data
Nov 05 13:58:01 DietPi CRON[23077]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Nov 05 13:58:01 DietPi CRON[23076]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Nov 05 13:58:01 DietPi CRON[23078]: (www-data) CMD (php /var/www/nextcloud/occ files:scan --all)
Nov 05 13:58:01 DietPi CRON[23079]: (www-data) CMD (echo $(date) > /tmp/test-wwwusr)
Nov 05 13:58:01 DietPi CRON[23077]: pam_unix(cron:session): session closed for user www-data
Nov 05 13:58:02 DietPi CRON[23076]: (CRON) info (No MTA installed, discarding output)
Nov 05 13:58:02 DietPi CRON[23076]: pam_unix(cron:session): session closed for user www-data
Nov 05 13:59:01 DietPi CRON[23097]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Nov 05 13:59:01 DietPi CRON[23098]: pam_unix(cron:session): session opened for user www-data(uid=33) by (uid=0)
Nov 05 13:59:01 DietPi CRON[23099]: (www-data) CMD (echo $(date) > /tmp/test-wwwusr)
Nov 05 13:59:01 DietPi CRON[23100]: (www-data) CMD (php /var/www/nextcloud/occ files:scan --all)
Nov 05 13:59:01 DietPi CRON[23098]: pam_unix(cron:session): session closed for user www-data
Nov 05 13:59:02 DietPi CRON[23097]: (CRON) info (No MTA installed, discarding output)
Nov 05 13:59:02 DietPi CRON[23097]: pam_unix(cron:session): session closed for user www-data

Do you still see (CRON) info (No MTA installed, discarding output) output for recent calls (which would indicate an error ourput)?

Yes. In Nov 05 13:58:02 DietPi CRON[23076]: for example

I’m happy with DietPi: Pihole, PiVPN, Aria2 and Nextcloud running on one device is stunning!

6

For some reason it keeps updating now!
I appended the verbose command :

*/1 * * * * php /var/www/nextcloud/occ files:scan --all --verbose 2>&1 >> /tmp/nextcloud_rescan_debug2.log

to keep track of whats happening.
The output shows, that cron is running and doing its job!

7

In Nextcloud folder not visible in Dietpi and Aria2 wrote the directory nextcloud/documents/mine is visible in all apps, just not on the server, here you mean it is not visible on all apps? I think many things got mixed together with solution and workaround attempts, where most likely permissions of manually created files/dirs were wrong.

Let’s please for now concentrate on assuring that dirs/files created via official apps and web UI are synced between all instances as expected.

Please try to create a custom directory, e.g. named customName via Nextcloud Windows client. Then on the server, search for the location of this dir:

find /mnt/dietpi_userdata/nextcloud_data -type d -name 'customName'

and verify that it gets created on all clients. Only if this works, we can further look into how to download via Aria2 (which was your initial aim?), while I’d always recommend to instead upload via Windows client and skip all the permissions and file index scan hassle on the server.

EDIT: To see error output of the cron job, you need to switch STDOUT and STDERR redirects:

*/1 * * * * php /var/www/nextcloud/occ files:scan --all --verbose >> /tmp/nextcloud_rescan_debug2.log 2>&1

2>&1 needs to come last, if you want STDERR to be redirected to where STDOUT finally goes, i.e. both to be logged into the file. But otherwise great that it works now. Let’s just hope it finally isn’t required but sync works the regular ways :slightly_smiling_face:.

8

I’d like to use the Windows - App.
Reinstalled it; it synchronised (with one errormessage for a file i propably created with smb or root user) but it didn’t integrate into the windows exploerer like it did last time.

Edit:
Did a software repair and restarted. The Nextcloud-App is now visible in the Explorer

The folder ‘customName’ was created and found:

/mnt/dietpi_userdata/nextcloud_data/admin/files/customName
root@DietPi:~#

A Test with another Folder and a Document within was successful too…
Strange, that it didn’t work before, since the Windowsapp was my first try - far before using SMB.
Solved?

9

Okay, great that this works now, whatever the issue was before.

With SMB and the cron job it works as well, right?

With Aria2 when downloading via Aria2 it is indeed a little more complicated since it runs as user aria2. Once could run the service as www-data user, but then /mnt/dietpi_userdata/aria2 and /mnt/dietpi_userdata/downloads/aria2.session needs to have the user changed as well.

10

Okay, great that this works now, whatever the issue was before.

With SMB and the cron job it works as well, right?

I disconnected smb and cleaned my entries in the crontabs.
Now its running clean via apps and WebUI

TY for your info on aria2 -
It is not that important to me. Would have been nice to start a download at home, leave home and open the file later on mobile. But that is more playing around, then a often used case.
You are doing great work.

1 Like