Mounting Windows shares containing spaces Topic is solved

Having issues with your DietPi installation, or, found a bug? Post it here.
SNG
Posts: 9
Joined: Thu Apr 16, 2020 1:11 pm

Re: Mounting Windows shares containing spaces

Post by SNG »

OK, I've hacked it some more and got the mount detection code working.

Dietpi's file_manager uses a df command to return the files system description in human readable form and saves it in a temporary file. The code then uses MAWK to process each line of this file in turn, treating each line as a record and splitting each record into fields using space as the delimiter. Each 'record' normally returns six fields but when the share source name has spaces it results in more than six fields being returned, one extra field for each space.
The orginal code looks like this:

Code: Select all

#----------------------------------------------------------------
# PHYSICAL DRIVES
#----------------------------------------------------------------" > $fp_fstab_tmp

		# Detect mounted drives
		G_DIETPI-NOTIFY 2 'Detecting drives, please wait...'
		# - Only detect mounts with valid source path (word 1 contains "/")
		mawk '$1~/\//{print}' <<< $(df -Pha) > .df_out_tmp

		# Remove misc items from list
		# - bind: https://github.com/MichaIng/DietPi/issues/2013#issuecomment-416394374
		[[ $misc_mounts ]] && while read line
		do

			local input_mount_target=$(mawk '{print $2}' <<< $line)
			sed -i "\#[[:blank:]]$input_mount_target$#d" .df_out_tmp
			# - target + source removal via $6 required for bind mounts: https://github.com/MichaIng/DietPi/issues/2013#issuecomment-417413867
			local input_mount_source=$(mawk '{print $1}' <<< $line)
			sed -i "\#[[:blank:]]$input_mount_source$#d" .df_out_tmp
			[[ $G_DEBUG == 1 ]] && G_DIETPI-NOTIFY 0 " - Detected misc mount and removed from df scrape: $input_mount_source > $input_mount_target"

		done <<< "$misc_mounts"

		# Process final df result
		while read line
		do

			Init_New_Device

			aDRIVE_ISMOUNTED[$index]=1
			aDRIVE_MOUNT_SOURCE[$index]=$(mawk '{print $1}' <<< $line)
			aDRIVE_SIZE_TOTAL[$index]=$(mawk '{print $2}' <<< $line)
			aDRIVE_SIZE_USED[$index]=$(mawk '{print $3}' <<< $line)
			aDRIVE_SIZE_FREE[$index]=$(mawk '{print $4}' <<< $line)
			aDRIVE_SIZE_PERCENTUSED[$index]=$(mawk '{print $5}' <<< $line)
			aDRIVE_MOUNT_TARGET[$index]=$(mawk '{print $6}' <<< $line)
			# Workaround for /dev/root under RPi, force physical location
			[[ ${aDRIVE_MOUNT_TARGET[$index]} == '/' ]] && aDRIVE_MOUNT_SOURCE[$index]=$FP_ROOTFS_SOURCE

			aDRIVE_SOURCE_DEVICE[$index]=$(Return_Drive_Without_Partitions ${aDRIVE_MOUNT_SOURCE[$index]})
			[[ ${aDRIVE_MOUNT_SOURCE[$index]} == /dev/${aDRIVE_SOURCE_DEVICE[$index]} ]] || aDRIVE_ISPARTITIONTABLE[$index]=1
			aDRIVE_UUID[$index]=$(blkid ${aDRIVE_MOUNT_SOURCE[$index]} -s UUID -o value)
			(( ${aDRIVE_ISPARTITIONTABLE[$index]} )) && aDRIVE_PART_UUID[$index]=$(blkid ${aDRIVE_MOUNT_SOURCE[$index]} -s PARTUUID -o value)
and etc.

I've used the AWK built in variable $NF to get the number of fields for each line and if there are more then six I intercept it and use a loop to reconstruct the proper source share name.

My code looks like this:

Code: Select all

#----------------------------------------------------------------
# PHYSICAL DRIVES
#----------------------------------------------------------------" > $fp_fstab_tmp

		# Detect mounted drives
		G_DIETPI-NOTIFY 2 'Detecting drives, please wait...'
		# - Only detect mounts with valid source path (word 1 contains "/")
		mawk '$1~/\//{print}' <<< $(df -Pha) > .df_out_tmp

		# Remove misc items from list
		# - bind: https://github.com/MichaIng/DietPi/issues/2013#issuecomment-416394374
		[[ $misc_mounts ]] && while read line
		do

			local input_mount_target=$(mawk '{print $2}' <<< $line)
			sed -i "\#[[:blank:]]$input_mount_target$#d" .df_out_tmp
			# - target + source removal via $6 required for bind mounts: https://github.com/MichaIng/DietPi/issues/2013#issuecomment-417413867
			local input_mount_source=$(mawk '{print $1}' <<< $line)
			sed -i "\#[[:blank:]]$input_mount_source$#d" .df_out_tmp
			[[ $G_DEBUG == 1 ]] && G_DIETPI-NOTIFY 0 " - Detected misc mount and removed from df scrape: $input_mount_source > $input_mount_target"

		done <<< "$misc_mounts"
		
		# Process final df result
		while read line
		do
			Init_New_Device

			# Get the total number of fields retured for each line
			local number_fields=$(mawk '{print NF}' <<< $line)
			aDRIVE_ISMOUNTED[$index]=1

			# If there are more than 6 fields in the record then the source name has spaces!
			if [[ $number_fields -ge 7 ]]; then
				# 7 or more fields, source name has spaces
				# Get the first part of the share source name
				local source_parts=$(mawk '{print $1}' <<< $line)
				# Now add the remaining parts 
				for (( n=2; n<=($number_fields - 5); n++ ))
				do  
					source_parts="$source_parts "$(mawk "{print \$($n)}" <<< $line)
				done
				aDRIVE_MOUNT_SOURCE[$index]=$source_parts		
			else
				# Record must have 6 fields and is normal
				aDRIVE_MOUNT_SOURCE[$index]=$(mawk '{print $1}' <<< $line)
			fi

			aDRIVE_SIZE_TOTAL[$index]=$(mawk '{print $(2+NF-6)}' <<< $line)
			aDRIVE_SIZE_USED[$index]=$(mawk '{print $(3+NF-6)}' <<< $line)
			aDRIVE_SIZE_FREE[$index]=$(mawk '{print $(4+NF-6)}' <<< $line)
			aDRIVE_SIZE_PERCENTUSED[$index]=$(mawk '{print $(5+NF-6)}' <<< $line)
			aDRIVE_MOUNT_TARGET[$index]=$(mawk '{print $(6+NF-6)}' <<< $line)
			# Workaround for /dev/root under RPi, force physical location
			[[ ${aDRIVE_MOUNT_TARGET[$index]} == '/' ]] && aDRIVE_MOUNT_SOURCE[$index]=$FP_ROOTFS_SOURCE

			aDRIVE_SOURCE_DEVICE[$index]=$(Return_Drive_Without_Partitions ${aDRIVE_MOUNT_SOURCE[$index]})
			[[ ${aDRIVE_MOUNT_SOURCE[$index]} == /dev/${aDRIVE_SOURCE_DEVICE[$index]} ]] || aDRIVE_ISPARTITIONTABLE[$index]=1
			aDRIVE_UUID[$index]=$(blkid ${aDRIVE_MOUNT_SOURCE[$index]} -s UUID -o value)
			(( ${aDRIVE_ISPARTITIONTABLE[$index]} )) && aDRIVE_PART_UUID[$index]=$(blkid ${aDRIVE_MOUNT_SOURCE[$index]} -s PARTUUID -o value)
and etc.

I'm sure this can be done much more elegantly if I had a better understanding of AWK than I have garnered but, hey, it works.

My changes to the code are highlighted and explained here, changes highlighted in orange:

# Get the total number of fields returned for each line
local number_fields=$(mawk '{print NF}' <<< $line)
aDRIVE_ISMOUNTED[$index]=1

# If there are more than 6 fields in the record then the sourece name has spaces!
if [[ $number_fields -ge 7 ]]; then
# 7 or more fields, source name has spaces
# Get the first part of the share source name
local source_parts=$(mawk '{print $1}' <<< $line)
# Now add the remaining parts
for (( n=2; n<=($number_fields - 5); n++ ))
do
source_parts="$source_parts "$(mawk "{print \$($n)}" <<< $line)
done
aDRIVE_MOUNT_SOURCE[$index]=$source_parts
else
# Record must have 6 fields and is normal
aDRIVE_MOUNT_SOURCE[$index]=$(mawk '{print $1}' <<< $line)
fi
# The following change forces the allocations to be last five fields regardless
aDRIVE_SIZE_TOTAL[$index]=$(mawk '{print $(2+NF-6)}' <<< $line)
aDRIVE_SIZE_USED[$index]=$(mawk '{print $(3+NF-6)}' <<< $line)
aDRIVE_SIZE_FREE[$index]=$(mawk '{print $(4+NF-6)}' <<< $line)
aDRIVE_SIZE_PERCENTUSED[$index]=$(mawk '{print $(5+NF-6)}' <<< $line)
aDRIVE_MOUNT_TARGET[$index]=$(mawk '{print $(6+NF-6)}' <<< $line)

Regards
SNG
User avatar
Joulinar
Posts: 2059
Joined: Sat Nov 16, 2019 12:49 am

Re: Mounting Windows shares containing spaces

Post by Joulinar »

cool, thx for your time to have it further investigated. I opened an issue on GitHub to follow up on the possible improvement.

https://github.com/MichaIng/DietPi/issues/3491
Pls let us know if a solution is working. This could help others if they hit by similar situation. Your DietPi Team
User avatar
MichaIng
Site Admin
Posts: 2293
Joined: Sat Nov 18, 2017 6:21 pm

Re: Mounting Windows shares containing spaces

Post by MichaIng »

@SNG
Many thanks for reporting the issue and providing a fix. I was thinking about a way to do this more failsafe, as other fields might contain spaces as well, e.g. the mount point:
tmpfs 2020572 0 2020572 0% /mnt/test test
I was hoping these are "tab" characters, but they are spaces, one up to multiple ones. The df headers are a mixture of left and right alignments, so scraping each line based on character counts from headers is not possible from what I found.

Sadly df does not provide any alternative output format. /proc/mounts and similar do not show space usage. findmnt is an alternative, with different output modes, however based on output mode, either spaces are still an issue or things like quotes are, or "dangerous" characters are masked by hex code or similar, which would break the script at other places.

The only failsafe method I can see is to run a dedicated findmnt -no <value> <source> on each mount for each value. To get the list of sources/mounts to loop through: df -a --output=source | mawk '/\//{print}'

Did I forget/overseen anything? I'll add the above as PR and run some tests to see if it is much slower or acceptable.
SNG
Posts: 9
Joined: Thu Apr 16, 2020 1:11 pm

Re: Mounting Windows shares containing spaces

Post by SNG »

Hi, MichaIng, thanks for looking at this. I must first confess that my efforts have all been down to google! I really don't have any in depth experience with linux and had never used either df or AWK before. Sadly as I get older my memory is not what it used to be and it takes several iterations of anything to get it to stick these days :? As a result I'm finding it difficult to visualize what you are proposing.
Last edited by SNG on Thu Apr 30, 2020 4:09 pm, edited 1 time in total.
SNG
Posts: 9
Joined: Thu Apr 16, 2020 1:11 pm

Re: Mounting Windows shares containing spaces

Post by SNG »

I was only trying to 'fix' what I didn't have control over, there is a built in share on Windows Home Server which contains spaces and they are pretty common on Windows paths anyway. On the dietpi box I'm creating the mount points so have direct control over it, but to do some error catching for spaces elsewhere I thought I could check the 'percentage used' field and make sure that it is the next to last one - it seems be always a number followed by '%' for the 'records' we are looking at. Indeed if it is not where we expect it then this would imply that there are spaces in the mount point, which could then be processed appropriately. I don't see why df would introduce any other extraneous spaces in the other four 'fields' without there being an error elsewhere.

So my sequence of events would be:
Test for number of fields greater than six and if so...
Check the position of 'percentage used' , if it is not fifth field from the beginning concatenate the first fields into the share path until it would be.
Check the position of 'percentage used' and if it is not next to last field concatenate any following fields into the mount point path.
Use the position of 'percentage used' to determine the position of the other four fields and extract them.

I wouldn't have thought speed of execution in this bit of code is critical, it's not run very often and doesn't loop.

My code still leaves an error on the file_manager screen. The line above in the table still doesn't have the correct path but I cannot see where it is getting it from.

Code: Select all

│                         ●─ //192.168.99.201/BTSync ──────────────────────────────────────────────────          │
│       /mnt/sambaBTSync  : //192.168.99.201/BTSync Shares | Net | Capacity: 19T | Used: 11T (60%)               │
User avatar
MichaIng
Site Admin
Posts: 2293
Joined: Sat Nov 18, 2017 6:21 pm

Re: Mounting Windows shares containing spaces

Post by MichaIng »

PR is up to fix white space handling, including \040 conversion of spaces from Samba shares for fstab only: https://github.com/MichaIng/DietPi/pull/3497
I'll run some tests now. This is just a start, e.g. to fully support spaces, space to \040 conversion, also when scraping /etc/fstab for entries, must do done for all mount sources and targets. But this is something for a later release.
User avatar
Joulinar
Posts: 2059
Joined: Sat Nov 16, 2019 12:49 am

Re: Mounting Windows shares containing spaces

Post by Joulinar »

Ok I will do some testing once merged into dev
Pls let us know if a solution is working. This could help others if they hit by similar situation. Your DietPi Team
User avatar
MichaIng
Site Admin
Posts: 2293
Joined: Sat Nov 18, 2017 6:21 pm

Re: Mounting Windows shares containing spaces

Post by MichaIng »

Okay works fine now according to my tests. Performance has even become slightly better since bind mount exclusion needed to be done differently, which is now more efficient: https://github.com/MichaIng/DietPi/pull/3497

Only downside is that managing two mount points of the same file system is not possible now with drive manager. However adding a second mount of the same source was not possible before, and AFAIK managing those would have caused issues, at least when unmounting one it would have been lost on a new drive scan loop, etc.
Attachments
samba.png
samba.png (3.22 KiB) Viewed 864 times
User avatar
Joulinar
Posts: 2059
Joined: Sat Nov 16, 2019 12:49 am

Re: Mounting Windows shares containing spaces

Post by Joulinar »

@MichaIng
ok seems working. However I have seen this message during umount

Code: Select all

[.     ] DietPi-Drive_Manager | umount /mnt/samba
[  OK  ] DietPi-Drive_Manager | umount /mnt/samba
rmdir: failed to remove '/mnt/samba': Device or resource busy
Pls let us know if a solution is working. This could help others if they hit by similar situation. Your DietPi Team
SNG
Posts: 9
Joined: Thu Apr 16, 2020 1:11 pm

Re: Mounting Windows shares containing spaces

Post by SNG »

I've been trying to test this and do manage to mount the offending windows share. Unfortunately I'm having problems testing further since using drive_manager seems to be setting the root file system read only and as a result it cannot update fstab with any changes once this happens. To move on I need to burn a backup copy of the system to my sd card and start over. I notice that another couple of threads have been started up with similar problems, most of which seem to be this same issue.

SNG
Post Reply