Securing coturn

urostor · July 25, 2023, 12:22pm

Hello,

I have been running Nextcloud on my pi4 with dietpi successfully for some time.

The default Nextcloud Talk installation installs coturn, a STUN/TURN server. To use it one needs to open some ports and the default configuration of coturn accepts every connection.
This seems to be easy for a port scan to exploit and it may have been what happened; recently I’ve been seeing many unknown connections mediated by coturn. I have disabled the server for now. I tried commenting out no_auth in /etc/turnserver.conf, as well as uncommenting lt_cred_mech. This did not change anything.

But at least I can track the malicious IPs by running coturn in the verbose mode (another config option).

Is there a way to force only the authenticated Nextcloud accounts to be able to use the turnserver, and reject everyone else?

Jappe · July 25, 2023, 12:27pm

There is a how-to in the nextcloud forums:
https://help.nextcloud.com/t/howto-setup-nextcloud-talk-with-turn-server/30794

Joulinar · July 25, 2023, 12:30pm

created by @MichaIng

urostor · July 25, 2023, 12:35pm

Many thanks, interesting - so these options aren’t automatically set by installing Nextcloud Talk via dietpi-software?

Jappe · July 25, 2023, 12:40pm

This is what the script does:

Click me to show code

if To_Install 168 coturn # Nextcloud Talk
		then
			G_DIETPI-NOTIFY 2 'Installing Coturn TURN server'

			# Install Coturn server only, install Nextcloud Talk app after Nextcloud has been fully configured
			G_AGI coturn
			G_EXEC systemctl stop coturn
			Remove_SysV coturn 1

			# Ask user for server domain and desired TURN server port
			local invalid_text=
			local domain=$(hostname -f)
			while :
			do
				G_WHIP_DEFAULT_ITEM=$domain
				if G_WHIP_INPUTBOX "${invalid_text}Please enter your server's external domain to allow Nextcloud Talk access your TURN server:"
				then
					domain=${G_WHIP_RETURNED_VALUE#http*://}
					break
				else
					invalid_text='[ERROR] This input is required!\n\n'
				fi
			done
			invalid_text=
			local port=3478
			while :
			do
				G_WHIP_DEFAULT_ITEM=$port
				if G_WHIP_INPUTBOX "${invalid_text}Please enter the network port, that should be used for your TURN server:
\nNB: This port (UDP + TCP) needs to be forwarded by your router and/or opened in your firewall settings. Default value is: 3478" && disable_error=1 G_CHECK_VALIDINT "$G_WHIP_RETURNED_VALUE" 0
				then
					port=$G_WHIP_RETURNED_VALUE
					break
				else
					invalid_text='[ERROR] No valid entry found, value needs to be a sequence of integers. Please retry...\n\n'
				fi
			done

			# Adjust Coturn settings
			# - If /etc/turnserver.conf is not present, use default or create empty file
			if [[ ! -f '/etc/turnserver.conf' ]]
			then
				# shellcheck disable=SC2015
				[[ -f '/usr/share/doc/coturn/examples/etc/turnserver.conf.gz' ]] && gzip -cd /usr/share/doc/coturn/examples/etc/turnserver.conf.gz > /etc/turnserver.conf || > /etc/turnserver.conf
			fi
			# https://help.nextcloud.com/t/howto-setup-nextcloud-talk-with-turn-server/30794
			G_CONFIG_INJECT 'listening-port=' "listening-port=$port" /etc/turnserver.conf
			G_CONFIG_INJECT 'fingerprint' 'fingerprint' /etc/turnserver.conf
			G_CONFIG_INJECT 'use-auth-secret' 'use-auth-secret' /etc/turnserver.conf
			G_CONFIG_INJECT 'realm=' "realm=$domain" /etc/turnserver.conf
			GCI_PRESERVE=1 G_CONFIG_INJECT 'total-quota=' 'total-quota=100' /etc/turnserver.conf
			GCI_PRESERVE=1 G_CONFIG_INJECT 'bps-capacity=' 'bps-capacity=0' /etc/turnserver.conf
			G_CONFIG_INJECT 'stale-nonce' 'stale-nonce' /etc/turnserver.conf
			G_EXEC sed -i 's/^[[:blank:]]*allow-loopback-peers/#allow-loopback-peers/' /etc/turnserver.conf
			G_CONFIG_INJECT 'no-multicast-peers' 'no-multicast-peers' /etc/turnserver.conf

			# Install Nextcloud Talk app
			G_EXEC systemctl start mariadb
			G_EXEC systemctl start redis-server
			G_EXEC ncc maintenance:mode --off
			if [[ ! -d '/var/www/nextcloud/apps/spreed' ]]
			then
				# Succeed if app is already installed and on "Cannot declare class" bug: https://github.com/MichaIng/DietPi/issues/3499#issuecomment-622955490
				G_EXEC_POST_FUNC(){ [[ $exit_code != 0 && $(<"$fp_log") =~ (' already installed'$|' Cannot declare class ') ]] && exit_code=0; }
				G_EXEC ncc app:install spreed
			fi
			ncc app:enable spreed

			# Adjust Nextcloud Talk settings to use Coturn
			ncc config:app:set spreed stun_servers --value="[\"$domain:$port\"]"
			# - Generate random secret to secure TURN server access
			local secret=$(openssl rand -hex 32)
			GCI_PASSWORD=1 GCI_PRESERVE=1 G_CONFIG_INJECT 'static-auth-secret=' "static-auth-secret=$secret" /etc/turnserver.conf
			# - Scrape existing secret, in case user manually chose/edited it
			secret=$(sed -n '/^[[:blank:]]*static-auth-secret=/{s/^[^=]*=//p;q}' /etc/turnserver.conf)
			ncc config:app:set spreed turn_servers --value="[{\"server\":\"$domain:$port\",\"secret\":\"$secret\",\"protocols\":\"udp,tcp\"}]" | sed 's/"secret":".*","protocols"/"secret":"<OMITTED>","protocols"/'
			unset -v secret domain port invalid_text
		fi

So the script already cares about this stuff

urostor · July 25, 2023, 12:43pm

Don’t I need to expose port 3478 to the Internet for coturn to work? I addition to 80 and 443 for Nextcloud itself.

Jappe · July 25, 2023, 12:48pm

Yes, looks like the clients make some p2p connection to each other via this port, or something like that.

In the config is set:

use-auth-secret
static-auth-secret=<yourChosen/GeneratedSecret>

So without this secret nobody should be able to connect?

MichaIng · July 25, 2023, 12:57pm

All that can be done to secure the connection is already done and mandatory anyway. You naturally need to open the port, as the clients themselves need to talk to the TURN server, which is used as relay for the WebRTC stream. So no P2P, as long as it is not possible due to peers being behind (symmetric) NATs.

Direct P2P is always preferred/attempted first, if possible (both clients in same LAN), or with the help of the STUN server (Coturn as well), and TURN relay only as fallback. P2P does not require any open ports, as the signalling server (Nextcloud, in case with help of STUN) establishes the connection between the peers.

Naturally if you open a port, anyone can send requests on it. If you see suspicious requests, I suggest to change that port in /etc/turnserver.conf and Nextcloud settings accordingly. But note that due to the secret, no one can really use the TURN server who is not logged into your Nextcloud as well. This random 32 hex character secret should be strong enough. And if someone wants to DDoS your system, that would be much more effective on port 80/443 as the HTTP requests cause orders of magnitude more traffic, CPU time etc than the relatively small TURN requests which are dropped quickly if no correct secret is sent.

urostor · July 25, 2023, 2:15pm

Then I am really not sure what caused the high CPU usage and constant network usage of the turnserver. Indeed, now it does not appear to be happening. I re-checked again and no-auth was always commented out.

Can I later paste the list of IPs that tried to connect somewhere here? They all seemed very random.

Joulinar · July 25, 2023, 3:48pm

well I don’t see any benefit of posting IP’s
Not something we can do with them.

urostor · July 25, 2023, 9:49pm

Sorry, it keeps happening actually.

I will post what happened within just one second (with verbose coturn enabled):

Jul 24 09:51:39 DietPi turnserver[3196]: 215: : session 001000000000000001: peer usage: realm=<my.domain>, username=<>, rp=0, rb=0, sp=0, sb=0
Jul 24 09:51:39 DietPi turnserver[3196]: 215: : session 001000000000000001: closed (2nd stage), user <> realm <my.domain> origin <>, local my.router.internal.IP:3478, remote random.IP1.blah.blah:36790, reason: allocation watchdog determined stale session state
Jul 24 09:51:39 DietPi turnserver[3196]: 292: : session 003000000000000002: usage: realm=<my.domain>, username=<>, rp=2048, rb=40960, sp=2048, sb=204800
Jul 24 09:51:39 DietPi turnserver[3196]: 292: : session 003000000000000002: peer usage: realm=<my.domain>, username=<>, rp=0, rb=0, sp=0, sb=0
Jul 24 09:51:39 DietPi turnserver[3196]: 305: : session 003000000000000002: usage: realm=<my.domain>, username=<>, rp=2048, rb=40960, sp=2048, sb=204800
Jul 24 09:51:39 DietPi turnserver[3196]: 305: : session 003000000000000002: peer usage: realm=<my.domain>, username=<>, rp=0, rb=0, sp=0, sb=0
Jul 24 09:51:39 DietPi turnserver[3196]: 318: : session 001000000000000002: usage: realm=<my.domain>, username=<>, rp=2048, rb=40960, sp=2048, sb=204800
Jul 24 09:51:39 DietPi turnserver[3196]: 318: : session 001000000000000002: peer usage: realm=<my.domain>, username=<>, rp=0, rb=0, sp=0, sb=0
Jul 24 09:51:39 DietPi turnserver[3196]: 337: : session 003000000000000002: usage: realm=<my.domain>, username=<>, rp=1691, rb=33820, sp=1691, sb=169100
Jul 24 09:51:39 DietPi turnserver[3196]: 337: : session 003000000000000002: peer usage: realm=<my.domain>, username=<>, rp=0, rb=0, sp=0, sb=0
Jul 24 09:51:39 DietPi turnserver[3196]: 337: : session 003000000000000002: closed (2nd stage), user <> realm <my.domain> origin <>, local my.router.internal.IP:3478, remote random.IP2.blah.blah:44450, reason: allocation watchdog determined stale session state
Jul 24 09:51:39 DietPi turnserver[3196]: 361: : session 001000000000000002: usage: realm=<my.domain>, username=<>, rp=1641, rb=32820, sp=1641, sb=164100
Jul 24 09:51:39 DietPi turnserver[3196]: 361: : session 001000000000000002: peer usage: realm=<my.domain>, username=<>, rp=0, rb=0, sp=0, sb=0
Jul 24 09:51:39 DietPi turnserver[3196]: 361: : session 001000000000000002: closed (2nd stage), user <> realm <my.domain> origin <>, local my.router.internal.IP:3478, remote random.IP3.blah.blah:31080, reason: allocation watchdog determined stale session state
Jul 24 09:51:39 DietPi turnserver[3196]: 427: : session 002000000000000006: usage: realm=<my.domain>, username=<>, rp=2048, rb=40960, sp=2048, sb=204800
Jul 24 09:51:39 DietPi turnserver[3196]: 427: : session 002000000000000006: peer usage: realm=<my.domain>, username=<>, rp=0, rb=0, sp=0, sb=0
Jul 24 09:51:39 DietPi turnserver[3196]: 433: : session 000000000000000001: usage: realm=<my.domain>, username=<>, rp=2048, rb=40960, sp=2048, sb=204800
Jul 24 09:51:39 DietPi turnserver[3196]: 433: : session 000000000000000001: peer usage: realm=<my.domain>, username=<>, rp=0, rb=0, sp=0, sb=0
Jul 24 09:51:39 DietPi turnserver[3196]: 440: : session 002000000000000006: usage: realm=<my.domain>, username=<>, rp=2048, rb=40960, sp=2048, sb=204800
Jul 24 09:51:39 DietPi turnserver[3196]: 440: : session 002000000000000006: peer usage: realm=<my.domain>, username=<>, rp=0, rb=0, sp=0, sb=0
Jul 24 09:51:39 DietPi turnserver[3196]: 447: : session 000000000000000001: usage: realm=<my.domain>, username=<>, rp=2048, rb=40960, sp=2048, sb=204800
Jul 24 09:51:39 DietPi turnserver[3196]: 447: : session 000000000000000001: peer usage: realm=<my.domain>, username=<>, rp=0, rb=0, sp=0, sb=0
Jul 24 09:51:39 DietPi turnserver[3196]: 460: : session 001000000000000003: usage: realm=<my.domain>, username=<>, rp=116, rb=2320, sp=116, sb=11600
Jul 24 09:51:39 DietPi turnserver[3196]: 460: : session 001000000000000003: peer usage: realm=<my.domain>, username=<>, rp=0, rb=0, sp=0, sb=0
Jul 24 09:51:39 DietPi turnserver[3196]: 460: : session 001000000000000003: closed (2nd stage), user <> realm <my.domain> origin <>, local my.router.internal.IP:3478, remote random.IP4.blah.blah:37256, reason: allocation watchdog determined stale session state
Jul 24 09:51:39 DietPi turnserver[3196]: 460: : session 000000000000000001: usage: realm=<my.domain>, username=<>, rp=2048, rb=40960, sp=2048, sb=204800
Jul 24 09:51:39 DietPi turnserver[3196]: 460: : session 000000000000000001: peer usage: realm=<my.domain>, username=<>, rp=0, rb=0, sp=0, sb=0
Jul 24 09:51:39 DietPi turnserver[3196]: 471: : session 000000000000000001: usage: realm=<my.domain>, username=<>, rp=2048, rb=40960, sp=2048, sb=204800
Jul 24 09:51:39 DietPi turnserver[3196]: 471: : session 000000000000000001: peer usage: realm=<my.domain>, username=<>, rp=0, rb=0, sp=0, sb=0
Jul 24 09:51:39 DietPi turnserver[3196]: 474: : session 002000000000000006: usage: realm=<my.domain>, username=<>, rp=623, rb=12460, sp=623, sb=62300
Jul 24 09:51:39 DietPi turnserver[3196]: 474: : session 002000000000000006: peer usage: realm=<my.domain>, username=<>, rp=0, rb=0, sp=0, sb=0
Jul 24 09:51:39 DietPi turnserver[3196]: 475: : session 001000000000000004: usage: realm=<my.domain>, username=<>, rp=52, rb=1040, sp=52, sb=5200
Jul 24 09:51:39 DietPi turnserver[3196]: 474: : session 002000000000000006: closed (2nd stage), user <> realm <my.domain> origin <>, local my.router.internal.IP:3478, remote random.IP5.blah.blah:30007, reason: allocation watchdog determined stale session state
Jul 24 09:51:39 DietPi turnserver[3196]: 475: : session 001000000000000004: peer usage: realm=<my.domain>, username=<>, rp=0, rb=0, sp=0, sb=0
Jul 24 09:51:39 DietPi turnserver[3196]: 475: : session 001000000000000004: closed (2nd stage), user <> realm <my.domain> origin <>, local my.router.internal.IP:3478, remote random.IP6.blah.blah:36790, reason: allocation watchdog determined stale session state

To me it seems that there are many attempted connections to the server but they appropriately get rejected. This causes >10% of CPU use and ceases after some time, happens periodically. About 40 KB/s upload usage, 20 down. Is this normal then? Just bots looking for servers to exploit?

MichaIng · July 29, 2023, 12:16pm

Indeed this looks like bots. Actually the IPs can be used to verify this as you can see which country they are originating from with some geoip tool: https://www.iplocation.net/

I often see such attempts on public SSH servers on port 22, mostly from China.

I would definitely change the port to mitigate this. Probably we should change our default as well if the default TURN server port became a common bot target in the meantime.

Joulinar · July 29, 2023, 1:15pm

@MichaIng we could check on our own turn server how this is going? If we see similar?

urostor · July 29, 2023, 1:17pm

Yeah, they mostly come from Asia and South America - at least that’s what my select whois lookups told me.

@Joulinar can I suggest enabling the verbose mode in /etc/turnserver.conf?

MichaIng · July 29, 2023, 1:59pm

Indeed, last bunch until last night 01:00. The errors look different in our case: UDP package sent errors, but indeed also from Spain and USA, so countries which cannot be us. I’ll switch the port in our case as well and see whether it helps.
EDIT: Done.

Not by default. The logs are very verbose already, and the error messages you see tell you everything you need to know already. But of course for deeper investigating, feel free to change it temporarily.

urostor · July 29, 2023, 2:25pm

I was not suggesting enabling this by default, just to check if this kind of thing was happening to you…

MichaIng · July 29, 2023, 2:30pm

~~Ah, did you get your logs only with verbose logs enabled?~~ EDIT: Ah, you wrote this already . I changed the port already, so if the UDP errors do not re-appear, then I guess it really was the same.

And this means we should switch the default port (we offer to change it via input box) on our Nextcloud Talk/Coturn install option. Any suggestions? 3478 => 3578?

It has no entry in the well-known list Wikipedia at lest: List of TCP and UDP port numbers - Wikipedia
Of course, like almost any 4 digits port, there are applications which use it: Service Name and Transport Protocol Port Number Registry

urostor · August 11, 2023, 10:19am

I think this is a good idea but I’m not sure how easy it will be to reconfigure all the software - wouldn’t all Talk clients need to be reconfigured?

And what do you recommend to do for now?

MichaIng · August 11, 2023, 4:28pm

Just Nextcloud, respectively the Talk app, and of course in case forwarded ports in router.

However, I do not plan to apply this on existing systems but instead add a changelog entry about the changed default and its reasons + a hint where to change the port manually, or do via dietpi-software reinstall 168.