No response traffic on application after port forward

S10 · 4 September 2020 11:51

Hi everyone,

I’m having a hard time troubleshooting some strange behaviour im experiencing on my dietpi install.
Hope somebody here can help me solve this.

I recently acquired a domainname and i’m trying to enable a https secure (with lets encrypt) connection with multiple applications like
nginx / domoticz / Emby server etc.
i’ve enabled a port forwarding rule on my router to the static ip adres of my dietpi installation (192.168.1.20)
But when i check the domainname or the public ip adres with sites like https://www.canyouseeme.org, the port seems to be closed. Which is strange because the port forwarding rule is no different than any other working port forwards.

So i started troubleshooting the problem by following the traffic from https://www.canyouseeme.org

First up the traffic from the internet to my router, the traffic arrives at the router and is forwarded to the correct ip 192.168.1.20

admin@UnifiSecurityGateway:~$ sudo tcpdump -n -i eth1 port 443 and host 192.168.1.20
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes
13:43:57.905462 IP 52.202.215.126.54981 > 192.168.1.20.443: Flags [S], seq 343550789, win 26883, options [mss 1460,sackOK,TS val 4070849679 ecr 0,nop,wscale 7], length 0
13:43:58.902144 IP 52.202.215.126.54981 > 192.168.1.20.443: Flags [S], seq 343550789, win 26883, options [mss 1460,sackOK,TS val 4070849929 ecr 0,nop,wscale 7], length 0
13:44:00.906108 IP 52.202.215.126.54981 > 192.168.1.20.443: Flags [S], seq 343550789, win 26883, options [mss 1460,sackOK,TS val 4070850430 ecr 0,nop,wscale 7], length 0

Next up the traffic from the router to my dietpi install 192.168.1.20. And here lies the problem.

root@DietPi:~# tcpdump -n -i eth0 port 443
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
13:46:40.094570 IP 52.202.215.126.55275 > 192.168.1.20.443: Flags [S], seq 253625873, win 26883, options [mss 1460,sackOK,TS val 4070890220 ecr 0,nop,wscale 7], length 0
13:46:41.094047 IP 52.202.215.126.55275 > 192.168.1.20.443: Flags [S], seq 253625873, win 26883, options [mss 1460,sackOK,TS val 4070890470 ecr 0,nop,wscale 7], length 0
13:46:43.097854 IP 52.202.215.126.55275 > 192.168.1.20.443: Flags [S], seq 253625873, win 26883, options [mss 1460,sackOK,TS val 4070890971 ecr 0,nop,wscale 7], length 0

My dietpi install seems to never reply to the incoming traffic with the correct response. So the port fowarding rule is working but there is no outbound traffic. So i thought that this must be local firewall issue. So i opened my iptables.

root@DietPi:~# iptables --list
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

There are no firewall rules active which could cause this behaviour and now i’m out of options.
What could be issue i’m not seeing?

Joulinar · 4 September 2020 12:03

Hi,

did you check if an application is LISTEN on port 443?

lsof -i -P -n | grep LISTEN

might be lsof is not installed on your system. Just use apt install lsof if it’s missing

S10 · 4 September 2020 12:43

LSOF was not installed, so i installed it and run the command:

root@DietPi:~# lsof -i -P -n | grep LISTEN
mono       427            www-data    5u  IPv4  20957      0t0  TCP *:8084 (LISTEN)
sshd       734                root    3u  IPv4  21940      0t0  TCP *:22 (LISTEN)
sshd       734                root    4u  IPv6  21942      0t0  TCP *:22 (LISTEN)
smbd       804                root   28u  IPv6  21178      0t0  TCP *:445 (LISTEN)
smbd       804                root   29u  IPv6  21179      0t0  TCP *:139 (LISTEN)
smbd       804                root   30u  IPv4  21180      0t0  TCP *:445 (LISTEN)
smbd       804                root   31u  IPv4  21181      0t0  TCP *:139 (LISTEN)
nginx      828                root    5u  IPv4  21229      0t0  TCP *:80 (LISTEN)
nginx      828                root    6u  IPv6  21230      0t0  TCP *:80 (LISTEN)
nginx      828                root    7u  IPv6  21231      0t0  TCP *:443 (LISTEN)
nginx      828                root    8u  IPv4  21232      0t0  TCP *:443 (LISTEN)
nginx      829            www-data    5u  IPv4  21229      0t0  TCP *:80 (LISTEN)
nginx      829            www-data    6u  IPv6  21230      0t0  TCP *:80 (LISTEN)
nginx      829            www-data    7u  IPv6  21231      0t0  TCP *:443 (LISTEN)
nginx      829            www-data    8u  IPv4  21232      0t0  TCP *:443 (LISTEN)
nginx      830            www-data    5u  IPv4  21229      0t0  TCP *:80 (LISTEN)
nginx      830            www-data    6u  IPv6  21230      0t0  TCP *:80 (LISTEN)
nginx      830            www-data    7u  IPv6  21231      0t0  TCP *:443 (LISTEN)
nginx      830            www-data    8u  IPv4  21232      0t0  TCP *:443 (LISTEN)
nginx      831            www-data    5u  IPv4  21229      0t0  TCP *:80 (LISTEN)
nginx      831            www-data    6u  IPv6  21230      0t0  TCP *:80 (LISTEN)
nginx      831            www-data    7u  IPv6  21231      0t0  TCP *:443 (LISTEN)
nginx      831            www-data    8u  IPv4  21232      0t0  TCP *:443 (LISTEN)
nginx      832            www-data    5u  IPv4  21229      0t0  TCP *:80 (LISTEN)
nginx      832            www-data    6u  IPv6  21230      0t0  TCP *:80 (LISTEN)
nginx      832            www-data    7u  IPv6  21231      0t0  TCP *:443 (LISTEN)
nginx      832            www-data    8u  IPv4  21232      0t0  TCP *:443 (LISTEN)
nginx      833            www-data    5u  IPv4  21229      0t0  TCP *:80 (LISTEN)
nginx      833            www-data    6u  IPv6  21230      0t0  TCP *:80 (LISTEN)
nginx      833            www-data    7u  IPv6  21231      0t0  TCP *:443 (LISTEN)
nginx      833            www-data    8u  IPv4  21232      0t0  TCP *:443 (LISTEN)
nginx      834            www-data    5u  IPv4  21229      0t0  TCP *:80 (LISTEN)
nginx      834            www-data    6u  IPv6  21230      0t0  TCP *:80 (LISTEN)
nginx      834            www-data    7u  IPv6  21231      0t0  TCP *:443 (LISTEN)
nginx      834            www-data    8u  IPv4  21232      0t0  TCP *:443 (LISTEN)
nginx      835            www-data    5u  IPv4  21229      0t0  TCP *:80 (LISTEN)
nginx      835            www-data    6u  IPv6  21230      0t0  TCP *:80 (LISTEN)
nginx      835            www-data    7u  IPv6  21231      0t0  TCP *:443 (LISTEN)
nginx      835            www-data    8u  IPv4  21232      0t0  TCP *:443 (LISTEN)
nginx      836            www-data    5u  IPv4  21229      0t0  TCP *:80 (LISTEN)
nginx      836            www-data    6u  IPv6  21230      0t0  TCP *:80 (LISTEN)
nginx      836            www-data    7u  IPv6  21231      0t0  TCP *:443 (LISTEN)
nginx      836            www-data    8u  IPv4  21232      0t0  TCP *:443 (LISTEN)
transmiss  838 debian-transmission   15u  IPv4  21256      0t0  TCP *:9091 (LISTEN)
transmiss  838 debian-transmission   16u  IPv4  21257      0t0  TCP *:30197 (LISTEN)
transmiss  838 debian-transmission   17u  IPv6  21259      0t0  TCP *:30197 (LISTEN)
nzbget     856              nzbget    6u  IPv4  21272      0t0  TCP *:6789 (LISTEN)
EmbyServe  865                emby  196u  IPv6  21312      0t0  TCP *:8096 (LISTEN)
EmbyServe  865                emby  201u  IPv6  21313      0t0  TCP *:8920 (LISTEN)
mono       869              sonarr   10u  IPv4  21380      0t0  TCP *:8989 (LISTEN)
mono       872              radarr    7u  IPv4  25134      0t0  TCP *:7878 (LISTEN)
jackett   8473             jackett  199u  IPv6  34158      0t0  TCP *:9117 (LISTEN)

Joulinar · 4 September 2020 12:48

are you able to connect to port 443 within your local network?

S10 · 4 September 2020 12:48

Important note is that this behaviour is only occuring when i try to connect to my ngix webserver from outside the LAN.
When i re-run the same testscript from within the LAN i get a response:

Below an example from within the LAN on which you see, that the dietpi install is responding to the incoming traffic

14:47:22.342124 IP 192.168.1.1.62003 > 192.168.1.20.443: Flags [S], seq 722020548, win 64240, options [mss 1460,nop,wscale 8,nop,nop,sackOK], length 0
14:47:22.342404 IP 192.168.1.20.443 > 192.168.1.1.62003: Flags [S.], seq 1773059636, ack 722020549, win 29200, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
14:47:22.346445 IP 192.168.1.1.62003 > 192.168.1.20.443: Flags [.], ack 1, win 513, length 0
14:47:22.349960 IP 192.168.1.1.62003 > 192.168.1.20.443: Flags [P.], seq 1:518, ack 1, win 513, length 517
14:47:22.350045 IP 192.168.1.20.443 > 192.168.1.1.62003: Flags [.], ack 518, win 251, length 0

Joulinar · 4 September 2020 12:59

probably it doesn’t make a difference, but did you try to reboot your system?

S10 · 4 September 2020 13:21

Amazing!!
It’s alive, ALIVE!!

So many hours of breaking my head around this. And installing some simple package fixed it.
I dont know what LSOF actually does, but its great

Joulinar · 4 September 2020 13:24

Well lsof isn’t anything special. It’s a diagnostic tool only. https://packages.debian.org/en/buster/lsof

But I guess you did a reboot?

S10 · 4 September 2020 13:43

I did do a reboot, but have been doing that multiple times the past week, with no change in the issue.
I dont know what is changed, apart from installing lsof but suddenly there is response traffic.

root@DietPi:~# tcpdump -n -i eth0 port 443
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
15:42:38.918280 IP 109.36.143.39.10587 > 192.168.1.20.443: Flags [S], seq 689541103, win 65535, options [mss 1300,nop,wscale 7,nop,nop,TS val 456470160 ecr 0,sackOK,eol], length 0
15:42:38.918491 IP 192.168.1.20.443 > 109.36.143.39.10587: Flags [S.], seq 2815750703, ack 689541104, win 28960, options [mss 1460,sackOK,TS val 4057371431 ecr 456470160,nop,wscale 7], length 0
15:42:38.968107 IP 109.36.143.39.10587 > 192.168.1.20.443: Flags [.], ack 1, win 19531, options [nop,nop,TS val 456470208 ecr 4057371431], length 0
15:42:38.979583 IP 109.36.143.39.10587 > 192.168.1.20.443: Flags [P.], seq 1:518, ack 1, win 19531, options [nop,nop,TS val 456470209 ecr 4057371431], length 517
15:42:38.979792 IP 192.168.1.20.443 > 109.36.143.39.10587: Flags [.], ack 518, win 235, options [nop,nop,TS val 4057371492 ecr 456470209], length 0
15:42:39.049548 IP 192.168.1.20.443 > 109.36.143.39.10587: Flags [.], seq 1:2577, ack 518, win 235, options [nop,nop,TS val 4057371562 ecr 456470209], length 2576
15:42:39.049601 IP 192.168.1.20.443 > 109.36.143.39.10587: Flags [P.], seq 2577:3496, ack 518, win 235, options [nop,nop,TS val 4057371562 ecr 456470209], length 919
15:42:39.072031 IP 109.36.143.39.10587 > 192.168.1.20.443: Flags [.], ack 1289, win 19531, options [nop,nop,TS val 456470209 ecr 4057371562], length 0
15:42:39.072281 IP 109.36.143.39.10587 > 192.168.1.20.443: Flags [.], ack 3496, win 19531, options [nop,nop,TS val 456470209 ecr 4057371562], length 0
15:42:39.128819 IP 109.36.143.39.10587 > 192.168.1.20.443: Flags [P.], seq 518:603, ack 3496, win 19531, options [nop,nop,TS val 456470371 ecr 4057371562], length 85
15:42:39.128892 IP 192.168.1.20.443 > 109.36.143.39.10587: Flags [.], ack 603, win 235, options [nop,nop,TS val 4057371642 ecr 456470371], length 0
15:42:39.129819 IP 192.168.1.20.443 > 109.36.143.39.10587: Flags [P.], seq 3496:3539, ack 603, win 235, options [nop,nop,TS val 4057371643 ecr 456470371], length 43
15:42:39.177808 IP 109.36.143.39.10587 > 192.168.1.20.443: Flags [.], ack 3539, win 19531, options [nop,nop,TS val 456470417 ecr 4057371643], length 0
15:42:39.186197 IP 109.36.143.39.10587 > 192.168.1.20.443: Flags [P.], seq 603:972, ack 3539, win 19531, options [nop,nop,TS val 456470421 ecr 4057371643], length 369
15:42:39.186500 IP 192.168.1.20.443 > 109.36.143.39.10587: Flags [P.], seq 3539:4418, ack 972, win 243, options [nop,nop,TS val 4057371699 ecr 456470421], length 879
15:42:39.228228 IP 109.36.143.39.10587 > 192.168.1.20.443: Flags [.], ack 4418, win 19531, options [nop,nop,TS val 456470474 ecr 4057371699], length 0
15:42:40.286119 IP 109.36.143.39.10587 > 192.168.1.20.443: Flags [P.], seq 972:995, ack 4418, win 19531, options [nop,nop,TS val 456471516 ecr 4057371699], length 23
15:42:40.286584 IP 109.36.143.39.10587 > 192.168.1.20.443: Flags [F.], seq 995, ack 4418, win 19531, options [nop,nop,TS val 456471516 ecr 4057371699], length 0
15:42:40.287091 IP 192.168.1.20.443 > 109.36.143.39.10587: Flags [F.], seq 4418, ack 996, win 243, options [nop,nop,TS val 4057372800 ecr 456471516], length 0
15:42:40.305098 IP 109.36.143.39.10587 > 192.168.1.20.443: Flags [.], ack 4419, win 19531, options [nop,nop,TS val 456471516 ecr 4057372800], length 0

Joulinar · 4 September 2020 13:58

yeah that’s strange somehow. Usually lsof should nave such an impact. If you like, you could remove lsof, do another reboot and check how it behave.

S10 · 4 September 2020 15:57

I did a reboot without removing lsof and i’m back at square 1

root@DietPi:~# tcpdump -n -i eth0 port 443
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
17:52:12.001598 IP 109.36.143.39.1344 > 192.168.1.20.443: Flags [S], seq 1941187099, win 65535, options [mss 1300,nop,wscale 7,nop,nop,TS val 458575080 ecr 0,sackOK,eol], length 0
17:52:12.360788 IP 109.36.143.39.1344 > 192.168.1.20.443: Flags [S], seq 1941187099, win 65535, options [mss 1300,nop,wscale 7,nop,nop,TS val 458575447 ecr 0,sackOK,eol], length 0
17:52:12.508828 IP 109.36.143.39.1344 > 192.168.1.20.443: Flags [S], seq 1941187099, win 65535, options [mss 1300,nop,wscale 7,nop,nop,TS val 458575583 ecr 0,sackOK,eol], length 0
17:52:12.648418 IP 109.36.143.39.1344 > 192.168.1.20.443: Flags [S], seq 1941187099, win 65535, options [mss 1300,nop,wscale 7,nop,nop,TS val 458575719 ecr 0,sackOK,eol], length 0
17:52:13.480744 IP 109.36.143.39.1344 > 192.168.1.20.443: Flags [S], seq 1941187099, win 65535, options [mss 1300,nop,wscale 7,nop,nop,TS val 458576557 ecr 0,sackOK,eol], length 0
17:52:24.672036 IP 109.36.143.39.1344 > 192.168.1.20.443: Flags [S], seq 1941187099, win 65535, options [mss 1300,sackOK,eol], length 0

Something changed in between the reboot. i didnt change anything but installed lsof. Maybe the iptables?

root@DietPi:~# iptables --list
Chain INPUT (policy ACCEPT)
target     prot opt source               destination

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination

S10 · 4 September 2020 18:03

Solved it!

I totally forgot that on the background i was running a vpn connection which conflicts with my portforward rules. Becasuse the traffic is rerouted through the vpn tunnel and so the reply traffic is tunneled through a totally different public ip than it came in.

Disabling the vpn, and the port forward works.

Joulinar · 4 September 2020 18:31

uh yeah that’s a good reason. Good that you found it.