[SOLVED] Amule strange problem with NTFS usb disk and NTFS network disk Topic is solved

[RaspMyzer]

Hi all, and sorry for bad english!

I 'm looking for a solution to my problem for some time, without finding it.
I have Dietpi on a raspberry 2 in which I have installed Amuled and Transmission:
I would like to keep temporary file folders of both in a USB disk connected with a Y-shaped cable,
and I would like to merge the completed downloads of the two programs on a network disk connected via a fritzbox router accessible through user and password.
I have created in router its specific for raspberry whith rw access on the entire disc even if i could only mount the specific downloads folder.

They are both NTFS, because I need to eventually detach them and carry them around.

I created the two directories DISCO_RETE and USB in the /media folder

Code: Select all

sudo mkdir -m 777 /media/DISCO_RETE
sudo mkdir -m 777 /media/USB

I configured them on FSTAB currently in this way

Code: Select all

//192.168.0.1/fritz.nas/300GB /media/DISCO_RETE cifs username = USER, password = PASSWORD, rw, vers = 1.0 0 0
UUID = 8066F0D566F0CD42 /media/USB ntfs-3g defaults, dir_mode = 0777, file_mode = 0777, noatime 0 2

in /home/amuleusr/.aMule/amule.conf

Code: Select all

IncomingDir=/media/DISCO_RETE/DOWNLOADS
TempDir=/media/USB/aMuleTemp

In /etc/transmission-daemon/settings.json

Code: Select all

"incomplete-dir": "/media/USB/TransmissionTemp",
"download-dir": "/media/DISCO_RETE/DOWNLOADS",

With Transmission no problem, everything works perfectly.
The problem is aMule: once a file is completed, the file is copied to the network disk, BUT the download line remains black,
the download is not "marked" as complete and is not removed from the list, also because the temporary files remain in the
temporary folder, and each time the daemon or raspberry is restarted, a new file is created with the same name and (1), (2) etc.

An example of amule logs:

Code: Select all

2019-04-11 15:10:03: Sospensione invio del file: 91686CC5A503DB5187D1078F58538C6B
 [b](GOOGLE TRANSLATED  2019-04-11 15:10:03: Suspension of sending the file: 91686CC5A503DB5187D1078F58538C6B )
 2019-04-11 15:14:29: 15:14:29: Error: Impossible to set permissions for the file '/media/DISCO_RETE/DOWNLOADS/XXXXXXXXXXXXXXXXXXX' (error 1: Operation not allowed)[/b]
 2019-04-11 15:14:29: 15:14:29: Error: File '/media/USB/aMuleTemp/001.part' could not be renamed '/media/DISCO_RETE/DOWNLOADS/XXXXXXXXXXXXXXXXXXX' (error 0 : Success)
 2019-04-11 15:20:02: 15:20:02: Error: Impossible to set permissions for the file '/media/DISCO_RETE/DOWNLOADS/XXXXXXXXXXXXXXXXXXX' (error 1: Operation not allowed)
 !2019-04-11 15:20:02: Errore imprevisto durante il completamento di XXXXXXXXXXXXXXXXXXX. File messo in pausa
[b](GOOGLE TRANSLATED  ! 2019-04-11 15:20:02: Unexpected error while completing XXXXXXXXXXXXXXXXXXX. File paused)[/b]

I've tried umask = 000 and I don't know how many options including different secs (from what I understand, on the fritz it's not possible to use cifs 2 and 3, and it works only with 1) .
Cifs 1 seems to support NTFS , reading and writing seems ok, Transmission works fine too, so i think it shouldn't be a privilege issue.
The problem could be that the daemon is started with user "amuleusr", tonight i'll try to change user , setting "dietpi", perhaps used by Transmission. *EDIT* same issue with dietpiuser too...*EDIT*

When i use the cifs quiet option, some strange things happen: in a first moment, doing

Code: Select all

sudo mount -a

doesn't cause errors , but amule restart fails, so i try the reboot, and the /media/DISCO_RETE doesn't point to network disk,
but it became a folder in the internal memory of Raspberry: in that case the download ends without problems, except that i want the file in another folder!!!

After reboot sudo mount -a gives

Code: Select all

	mount error(22): Invalid argument
	Refer to the mount.cifs(8) manual page (e.g. man mount.cifs)

Removing quiet makes mount correctly again.
The directory USB went mounted correctly with quiet, it should be because the ntfs-3g allows.
The folder change happened other times with some cifs option without rising errors the first time i mount.

I didn't have this problem when using a Netgear router and i used the same disk on the network both for temporary files and for finished downloads, with guest user.

Could it be a problem when "moving" the file from an NTFS disk to another NTFS?
I have not yet tried to format the temporary disk in EXT4 and I would not do it, because I should practically leave it attached to the raspberry without then being able to use it on various windows machines

Can anyone give me an explanation or suggestion? Is there a way to see some logs?
Thanks!

[MichaIng]

First of all, did you try to set the mounts up with dietpi-drive_manager?
At least it assures correct syntax and modes. Mount the fritz.nas as Samba network drive and the NTFS usb drive should be detected fine.

As you mention ntfs-3g I guess that is fine, but did you verify, e.g. via htop, that the ntfs-3g service is running the emulates UNIX permissions? As of course NTFS does not support UNIX permissions natively.

Okay but actually the issue, based on the logs, is indeed that the CIFS/Samba mount does not support UNIX permissions. Also based on your fstab you own it to one fixed user+group. eMule now tries to change permissions but that does not work on this kind of mount. The other thing is/was the user of eMule.

How we manage this for our software/install options is to have every program run as it's own user (e.g. Transmission as transmission-daemon) but then as group dietpi, or, if the service fork itself does not support a custom group, at least we add the user to the dietpi group. The userdata/media dirs then have 775 mode, so all programs have R/W permissions since they are in dietpi group. To preserve this e.g. for new downloads as well we switched umask for at least all downloaders to 002/007.

So when you installed Transmission via dietpi-software that should be already fine, as it creates new downloads then as transmission-daemon:dietpi 664. To allow full access for eMule, add it's user to the dietpi group as well: usermod -a -G dietpi amuleusr
However running it as dietpi user directly of course works as well. It is only a security concern since dietpi is a login user with sudo permissions. Then umask should be 002 or 007 depending on how much other users might need read access. 000 of course works as well, again a security concern.
Then to the mounts, dietpi-drive_manager would do that automatically, but same here: ,uid=dietpi,gid=dietpi,file_mode=0770,dir_mode=0770 is what we add to the fstab mount options to guarantee R/W access for all installed software. In case of the local NTFS mount we do not change user/mode but add ,permissions which triggers the above mentioned ntfs-3g service to emulate UNIX permissions. Then you can chown/chmod it or sub dirs as you need.

How did you paste your fstab btw? With all the free spaces inside it looks like wrong syntax, e.g. all the options need to be in one "word" comma separated but without free spaces. Also UUID=.. without free space. But since it works I guess it's just the output format you pasted. But again, I would try to have this done automatically with dietpi-drive_manager. Do a backup of your current fstab first, just in case: G_BACKUP_FP /etc/fstab

[RaspMyzer]

First of all, thanks for the reply.
I forgot to say that I am in my first steps with Raspberry and Unix, but maybe it was understood from the things I wrote

In htop the ntfs-3g service running
/sbin/mount.ntfs-3g /dev/sda1 /media/USB -o rw

The fstab lines have actually been modified by pasting google translate , all the parameters are in a "word"

I tried the command to add the user to the dietpi group
sudo usermod -a -G dietpi amuleusr
but the problem is not solved.

I added ",permission" and get "mount error (22): Invalid argument" because cifs vers = 1.0,
then I put vers = 3.0 back but comes back the problem of mounting a DONWLOADS folder inside the memory, with the error
mount error (22): Invalid argument

So I tried adding manually
,uid=dietpi,gid=dietpi,file_mode=0770,dir_mode=0770
now I'm waiting for some downloads to finish.

I tried dietpi-drive_manager during installation but I don't remember why I couldn't configure and used the old method.
If you definitely recommend that, I'll try again tonight if id and mode don't work.
Thank you

[RaspMyzer]

I started dietpi-drive_manager, removed the DISCO_RETE share, and recreated it with the tool.
I also thought of moving the USB to /mnt.
I used DietPi-AutoStart too, and
sudo nano /var/lib/dietpi/dietpi-autostart/custom.sh
put

Code: Select all

sudo chmod 777 /mnt/DISCO_RETE/DOWNLOADS -R
sudo chmod 777 /mnt/USB/aMuleTemp -R
sudo chmod 777 /mnt/USB/TransmissionTemp -R
sudo chown -R amuleusr /home/amuleusr/.aMule

hoping could help
and

Code: Select all

sudo chmod +x /var/lib/dietpi/dietpi-autostart/custom.sh
cd /var/lib/dietpi/dietpi-autostart
sudo update-rc.d custom.sh defaults

gives
update-rc.d: error: unable to read /etc/init.d/custom.sh

Transmission continues to go well, now amule gives the error
Error: Directory xxx couldn't be created (error 13: Permission denied)

I tried
usermod -a -G dietpi amuleusr
but with
getent group
I saw the line
dietpi: x: 1000: amuleusr

but all the others lines have the user first and then the group for example
netdev: x: 108: dietpi
and
amuleusr: x: 1001:

I inverted doing usermod -a -G amuleusr dietpi
but now i get yet dietpi in group amuleusr

Does usermod create the specified group if not existing?
The error persists, tonight i'll reload the saved system image cause
sudo groupdel amuleusr
gives
groupdel: cannot remove the primary group of user 'amuleusr'
and i want restart from a more clean situation...

[RaspMyzer]

With usermod -a -G amuleusr dietpi and dietpi-drive_manager ,
was added line in fstab

Code: Select all

//192.168.0.1/fritz.nas/300GB /mnt/DISCO_RETE cifs username=USER,password=PASSWORD,iocharset=utf8,uid=dietpi,gid=dietpi,file_mode=0770,dir_mode=0770,vers=1.0,_netdev,nofail 0 0

i confirm the error

Code: Select all

Error: Directory '/mnt/DISCO_RETE/DOWNLOADS' couldn't be created (error 13: Permission denied)
ERROR: Error accessing directory. Could not create the incoming directory at '/mnt/DISCO_RETE/DOWNLOADS'.
Attempting to use default directory at location 
'/home/amuleusr/.aMule/Incoming'

Changing the mount point in /mnt/DISCO_RETE/DOWNLOADS', and 0770 in 0777 changed nothing.
with 0777 strangly i see with ls -la under /mnt
dr-xr-xr-x 1 dietpi dietpi 0 mar 25 23:54 DISCO_RETE
(no write permission to dietpi!)

With variuos attempt modifying row on fstab inserted by dietpi-drive_manager, it seems dietpi has "permission too strict" in amuled logs:
ERROR: Error accessing directory. Permissions on the incoming directory too strict!
aMule cannot proceed. To fix this, you must set read/write/exec permissions for the folder

now i changed in uid and gid from dietpi to root and , for now no error logs are shown.... Let's see at the end of download...

*EDIT*
It's ridicolous!
I added amuleusr with sudo visudo , i'm mounting the disk with uid root , and the error "Impossible to set permissions for the file" persists! : the file remains in black, it's copied again and again every reboot in the folder, and the temp file remains temp folder until i delete the download from list.
At this point i guess it's a Fritzbox security issue, or aMuled bug , but the strange thing is that by my old router and guest access i had no one of these problems!

[MichaIng]

@RaspMyzer

netdev: x: 108: dietpi

This is correct, netdev is the group here and dietpi the user.
So dietpi: x: 1000: amuleusr verifies that amuleusr is indeed inside the dietpi group.
- dietpi is a user and all users have a group with the same name as well, which we use here to share permissions across programs
- This is also the reason why you faced groupdel: cannot remove the primary group of user 'amuleusr': As long as a user exist, it's own (same name) group can never be removed.

dr-xr-xr-x 1 dietpi dietpi 0 mar 25 23:54 DISCO_RETE

Okay yeah then indeed this is an issue of the Fritz!Box NAS most likely, that the drive is mounted as read-only, even that the mount settings contain write permissions.

update-rc.d: error: unable to read /etc/init.d/custom.sh

Two things about that:
- custom.sh itself is not service that can be enabled. To enable it, use dietpi-autostart and select the related option. This will create a systemd service that starts the custom.sh script automatically. Nothing else to do.
- Do not use sudo inside the script. It is obsolete since it runs as root user anyway.
- The following as above should be removed completely, also the chmod +x is done by dietpi-autostart:

Code: Select all

sudo chmod +x /var/lib/dietpi/dietpi-autostart/custom.sh
cd /var/lib/dietpi/dietpi-autostart
sudo update-rc.d custom.sh defaults

Yeah so finally I would go through the router settings/docs about how to gain R/W access. Possibly you need to add the clients IP or address range to allow that, not sure. Let my check my own Fritz!NAS settings, although I have a branded Fritz!Box with outdated OS...

[RaspMyzer]

Thanks again for your help.
I'm not at home for some days, when i'll return i'll check and , hoping for find a solution, i'll share it


***EDIT***
Nothing to do ... Unfortunately I configured everything according to the FritxBox manual (user and NAS with relevant permissions).
I will try to format the discs from Dietpi first, then I'll try Raspbian to check further if there were any differences in the installed modules.

[RaspMyzer]

I installed Raspbian Stretch Lite 2019-04-08 and get same problems.

But since using another incoming folder on USB disk there are no problem, i could use it and INCRON ( tried it and works , moving files on network disk ) but i don't like to use a workaround and another program if i can avoid spendig resources.

Having spent too much time, i think i'll remain on this distribution , even because testing AMULED_USER="root" in this case make it work (the webserver doesn't but i bet it's some wrong configuration )

Probably should help see creation script for debian-transmission user/group for replicate it for amuleusr
( inserting amuleusr in the group haven't resolved too).
But i don't find nothing on it: now i'll use root or pi user and adjust webserver.

Sorry for not resolve it on dietpi, it remain one of the best low space and low resources distribution!
Thanks for the help!

[MichaIng]

If as well on Raspbian it only works as root user, then there seems to be still a Fritz!Box related permissions issue. Of course with root this should work on DietPi similarly, however the issue with downloaders and root is always that the media players (or whatever wants to access/sort/import those files) need root permissions then as well, which is not ideal from security aspect.

[RaspMyzer]

For sure it's a fritz bug. However like i said if i find the permission of debian-transmission i could replicate on my user.

Reinstalled dietpi, it seems to work better anyway.
As for security issues for booting with root I know it's not recommended, however my modest little home network is not a bank I'm trying to use the rasp as a small download server, and if someone will onw my raspberry, they will must use much patience if they will download my downloads with low upload of my line
unfortunately I didn't solve as I wanted but I still solved it, and this topic could be marked as solved!