Request: syncthing-inotify

Sciamano · 3 December 2016 22:36

Hello,
another request by me
I have noticed that using syncthing on arm devices, especially the development boards DietPi supports, can easily boost the cpu usage to 100% for long periods of time, when rescans of the shared folders take place.
I have read that syncthing-inotify sensibly reduces cpu usage by allowing longer times between rescans, simply by notifying the system of the files that have changed, instead of having it fully rescan the directories.
Could this be added to DietPi? I think it would make our boards more responsive.

More information here: https://github.com/syncthing/syncthing-inotify

Thanks!

Sciamano · 6 December 2016 14:21

In case anyone else is interested, here is how to install syncthing-inotify.

Make a directory and download latest version:

mkdir syncthing-inotify
cd syncthing-inotify
wget https://github.com/syncthing/syncthing-inotify/releases/download/v0.8.4/syncthing-inotify-linux-arm-v0.8.4.tar.gz

Untar:

tar zxvf syncthing-inotify-linux-arm-v0.8.4.tar.gz

Copy the binary to /usr/bin:

cp syncthing-inotify /usr/bin

Remove the download folder:

cd ..
rm -r syncthing-inotify

run it:

/usr/bin/syncthing-inotify

Sciamano · 6 December 2016 14:57

Unfortunately though, I don’t seem to be able to make it autostart or run as a service.
The syncthing-inotify project gives two scripts (one to run as users and one as system), and I have tried both without success. They make syncthing-inotify start but it does not work (and rescans of the directories happen as usual).
I have also tried adding it to /etc/rc.local but it happens the same.

BTW, here are the two scripts provided by the project… If anyone more capable than me could make them work I would be extremely grateful:

Run as system service: /etc/systemd/system/syncthing-inotify@.service

[Unit]
Description=Syncthing Inotify File Watcher for %I
Documentation=https://github.com/syncthing/syncthing-inotify/blob/master/README.md
After=network.target syncthing@.service
Requires=syncthing@.service

[Service]
User=%i
ExecStart=/usr/bin/syncthing-inotify -logflags=0
SuccessExitStatus=2
RestartForceExitStatus=3
Restart=on-failure
ProtectSystem=full
ProtectHome=read-only

[Install]
WantedBy=multi-user.target

Run as user: /etc/systemd/user/syncthing-inotify.service

[Unit]
Description=Syncthing Inotify File Watcher
Documentation=https://github.com/syncthing/syncthing-inotify/blob/master/README.md
After=syncthing.service
Requires=syncthing.service

[Service]
ExecStart=/usr/bin/syncthing-inotify -logflags=0
SuccessExitStatus=2
RestartForceExitStatus=3
Restart=on-failure
ProtectSystem=full
ProtectHome=read-only

[Install]
WantedBy=default.target

Thanks!!

Sciamano · 6 December 2016 15:42

Proposed solution (it seems to be working):

crontab -e

add the following:

@reboot /usr/bin/syncthing-inotify

johnvick · 6 December 2016 19:31

Have you noticed much reduction of CPU use? I have XU4/Cloudshell so am well aware of how SyncThing can tax the system. I’ll try out your solution here.

Sciamano · 6 December 2016 21:05

I can notice a drastic decrease in CPU usage, as the system won’t scan every shared directory every few minutes, but only checks and syncs those files that get changed.
It would be nice if I could get it to work as a service, though.

Sciamano · 8 December 2016 21:57

I have been discussing this issue with the guys from syncthing, as running syncthing-inotify from the crontab is not working properly.
There seems to be a problem with how syncthing is run in DietPi. Specifically, they are saying that the service file provided by DietPi is not correct and that this is causing the issue with syncthing-inotify.

This is what came out regarding the syncthing.service script in DietPi:

This file is plain wrong:

It starts syncthing is root; there is no possibility to overwrite it.
It uses some bash Foo in order to create a logfile
The type is oneshot (i guess because of the bash foo?)

I don’t understand much about these things, but I think that our development boards would benefit a lot from the introduction of syncthing-inotify on DietPi, therefore I am calling out to Fourdee again for this to be introduced or at least fixed in order for us to install syncthing-inotify successfully. Thanks!

johnvick · 9 December 2016 01:39

I was going to give this a go this weekend but I’ll hold off for now.

Before SyncThing was installable from the DietPi menu I had installed in manually but it used a lot of CPU that way also.

John

Sciamano · 9 December 2016 09:49

Yes, syncthing uses a lot of cpu during the rescan of the shared directories.
This would be avoided by switching from full rescans to partial ones by adopting the syncthing-inotify way.
The main problem seems to be that syncthing runs as root in DietPi, and this creates some kind of incompatibility with the syncthing-inotify.service script.
On the syncthing support forum they suggest to install syncthing manually on DietPi and avoid using the provided syncthing.service script, but I guess this defeats the purpose of having the logs kept in the ramdisk to avoid the frequent writes of the SD card…

johnvick · 9 December 2016 22:41

I’ve looked in the forum you refer to:

https://forum.syncthing.net/t/cant-make-syncthing-inotify-properly-autostart-at-boot/8758/24

And have got to the same point as you in that syncthing-inotify works but I cannot autostart it as a service.

I’ll keep an eye on that thread and see what develops.

John

johnvick · 9 December 2016 23:25

Got this working now:

/etc/systemd/system/syncthing.service

[Unit]
Description=Syncthing
After=network.target
Wants=syncthing-inotify@.service

[Service]
Type=oneshot
ExecStart=/bin/bash -c '/usr/bin/syncthing > /var/log/syncthing.log &'
ExecStop=/usr/bin/killall -w syncthing
RemainAfterExit=yes
User=root

[Install]
WantedBy=multi-user.target

/etc/systemd/system/syncthing-inotify@.service

[Unit]
Description=Syncthing Inotify File Watcher for %I
Documentation=https://github.com/syncthing/syncthing-inotify/blob/master/README.md
After=network.target syncthing@.service
Requires=syncthing@.service

[Service]
User=root
ExecStart=/usr/bin/syncthing-inotify -logflags=0
SuccessExitStatus=2
RestartForceExitStatus=3
Restart=on-failure
ProtectSystem=full
ProtectHome=read-only

[Install]
WantedBy=multi-user.target

Then do:

systemctl enable syncthing-inotify@service

systemctl start syncthing-inotify@service

I got the error about too many files etc so did as suggested here:

https://github.com/syncthing/syncthing-inotify#troubleshooting-for-folders-with-many-files-on-linux

I think this is how I did it but so much trial and error can’t be entirely sure. But reboot and in htop I see syncthing-inotify running.

John

Fourdee · 17 December 2016 14:14

Thanks John, great work.

I’ll get syncthing-inotify added to the installation of Syncthing by default: https://github.com/Fourdee/DietPi/issues/652

Sciamano · 18 December 2016 10:22

Thanks John for sharing your experience. I have not time to try right now, but please check that syncthing-inotify correctly signals syncthing about file changes.
You can easily do that by modifying one file on the machine with syncthing-inotify, and checking that the other machines receive the modified file immediately (while the “last rescan” time remains the same).
I’m saying this because during my tests, syncthing-inotify was correctly showing in the process list, but it didn’t actually do its job properly.

johnvick · 18 December 2016 21:43

I’m testing this now but just to clarify, if it is working it should sync the files before the syncthing predefined scan interval?

If so it doesn’t seem to be working. I’ve set the interval to 10 minutes and nothing seems to happen.

Sciamano · 18 December 2016 21:50

The purpose of syncthing-inotify is to set a long rescan interval (let’s say an hour, for example) to avoid the system to be constantly rescanning the shared folders in search of changed files. Between two rescans syncthing-inotify works by “detecting” file changes and syncing those files almost immediately.
If you modify a file and it’s not being immediately “spread” to the other nodes in the syncthing network, then syncthing-inotify isn’t working properly.

johnvick · 19 December 2016 02:09

Ah then it’s not working properly. I’ll have a look at it again when I get home from holiday.

johnvick · 19 December 2016 08:37

Decided couldn’t wait.

/etc/systemd/system/syncthing-inotify@.service

[Unit]
Description=Syncthing Inotify File Watcher for %I
Documentation=https://github.com/syncthing/syncthing-inotify/blob/master/README.md
After=network.target syncthing.service
Requires=syncthing.service

[Service]
User=root
ExecStart=/usr/bin/syncthing-inotify -logflags=0
SuccessExitStatus=2
RestartForceExitStatus=3
Restart=on-failure
ProtectSystem=full
ProtectHome=read-only

[Install]
WantedBy=multi-user.target

(Notice this is slightly different from above - now syncthing.service not syncthing@.service)

Then do:

systemctl enable syncthing-inotify@service

systemctl start syncthing-inotify@service

This seems to be working and also working after a reboot - near instantaneous transfer of files despite syncthing being set to rescan the directory every 600 seconds.

Sciamano · 19 December 2016 11:45

That’s great John! I’ll try it ASAP (which means as soon as this darn flu passes)

johnvick · 26 December 2016 01:30

Hi Fourdee,

I see this is now in v140 - can I bring your attention to the post above as I needed to make a minor change to get it working - I hope this made it into v140.

John

Fourdee · 26 December 2016 13:34

Hi John,

inotify, Yep its in v140. Installed by default with syncthing.

A simple reinstall should do it, pulls in the inotify binaries and service. Backup system 1st with dietpi-backup, just incase.
First remove the service you added:

/etc/systemd/system/syncthing-inotify@.service

Then reinstall Syncthing:

dietpi-software reinstall 50

Our service is a little different:

https://github.com/Fourdee/DietPi/blob/master/dietpi/dietpi-software#L8802-L8817