Wireguard connection is not working on Odroid Bullseye

dr-rox · 12 January 2022 22:29

I have freshly reflashed my old buster WIreguard VPN server to bullseye and Wireguard is not working anymore. It worked well on buster though. I did some digging using tcpdump on the board and found the traffic from the wireguard client - my phone is actually forwarded to eth0 and out, but when the response gets back from the internet to eth0, it’s not forwarded back to wg0. This is kinda strange as all iptables rules are the same on my bullseye. So I can’t figure out what’s the problem here. Maybe you did encounter the same issue?

Bug report reference code: d4a5c44f-cbf9-4eb9-b34b-d564702d8e19

Device model : Odroid XU3/XU4/MC1/HC1/HC2 (armv7l)

# iptables -L -v
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
  149  8940 ACCEPT     all  --  wg0    any     anywhere             anywhere

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

# iptables -L -v -t nat
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
   66  4141 MASQUERADE  all  --  any    eth0    anywhere             anywhere

# sysctl -a | grep forward
net.ipv4.conf.all.forwarding = 0
net.ipv4.conf.all.mc_forwarding = 0
net.ipv4.conf.default.forwarding = 0
net.ipv4.conf.default.mc_forwarding = 0
net.ipv4.conf.eth0.forwarding = 1
net.ipv4.conf.eth0.mc_forwarding = 0
net.ipv4.conf.lo.forwarding = 0
net.ipv4.conf.lo.mc_forwarding = 0
net.ipv4.conf.wg0.forwarding = 1
net.ipv4.conf.wg0.mc_forwarding = 0
net.ipv4.ip_forward = 0
net.ipv4.ip_forward_use_pmtu = 0
net.ipv6.conf.all.forwarding = 0
net.ipv6.conf.all.mc_forwarding = 0
net.ipv6.conf.default.forwarding = 0
net.ipv6.conf.default.mc_forwarding = 0
net.ipv6.conf.eth0.forwarding = 1
net.ipv6.conf.eth0.mc_forwarding = 0
net.ipv6.conf.lo.forwarding = 0
net.ipv6.conf.lo.mc_forwarding = 0
net.ipv6.conf.wg0.forwarding = 1
net.ipv6.conf.wg0.mc_forwarding = 0

# lsmod
Module                  Size  Used by
binfmt_misc            20480  1
sctp_diag              16384  0
sctp                  270336  5 sctp_diag
tcp_diag               16384  0
udp_diag               16384  0
inet_diag              20480  3 tcp_diag,sctp_diag,udp_diag
unix_diag              16384  0
nft_chain_nat_ipv6     16384  1
nf_conntrack_ipv6      20480  1
nf_defrag_ipv6         20480  1 nf_conntrack_ipv6
nf_nat_ipv6            16384  1 nft_chain_nat_ipv6
ip6t_MASQUERADE        16384  1
nf_nat_masquerade_ipv6    16384  1 ip6t_MASQUERADE
nf_tables_ipv6         16384  3
nft_chain_nat_ipv4     16384  1
nf_conntrack_ipv4      16384  2
nf_defrag_ipv4         16384  1 nf_conntrack_ipv4
nf_nat_ipv4            16384  1 nft_chain_nat_ipv4
ipt_MASQUERADE         16384  1
nf_nat_masquerade_ipv4    16384  1 ipt_MASQUERADE
nf_nat                 24576  4 nf_nat_masquerade_ipv6,nf_nat_ipv6,nf_nat_masquerade_ipv4,nf_nat_ipv4
nf_conntrack          106496  8 nf_conntrack_ipv6,nf_conntrack_ipv4,ipt_MASQUERADE,nf_nat_masquerade_ipv6,nf_nat_ipv6,nf_nat_masquerade_ipv4,nf_nat_ipv4,nf_nat
libcrc32c              16384  3 nf_conntrack,nf_nat,sctp
nft_compat             20480  2
nft_counter            16384  4
nft_meta               16384  4
nf_tables_ipv4         16384  3
nf_tables              73728  13 nft_chain_nat_ipv6,nft_chain_nat_ipv4,nft_compat,nf_tables_ipv6,nf_tables_ipv4,nft_meta,nft_counter
nfnetlink              16384  2 nft_compat,nf_tables
wireguard             139264  0
ip6_udp_tunnel         16384  1 wireguard
udp_tunnel             16384  1 wireguard
sg                     32768  0
w1_gpio                16384  0
evdev                  24576  0
wire                   24576  1 w1_gpio
uio_pdrv_genirq        16384  0
uio                    20480  1 uio_pdrv_genirq
exynos_gpiomem         16384  0
fuse                   86016  1
ip_tables              24576  0
ipv6                  376832  58 nf_conntrack_ipv6,nf_nat_masquerade_ipv6,udp_diag,wireguard,nf_nat_ipv6,sctp
sd_mod                 49152  0
uas                    20480  0
usb_storage            49152  1 uas
scsi_mod              135168  4 sd_mod,usb_storage,uas,sg
spidev                 20480  0
spi_s3c64xx            20480  0
gpio_keys              20480  0
extcon_usb_gpio        16384  0

Joulinar · 13 January 2022 08:40

Could you share the tcpdump command you used to track WireGuard? And did you install WireGuard using dietpi-software?

dr-rox · 13 January 2022 13:27

Yes, I’ve installed Wireguard from the dietpi-software. Set up as a server and transferred the generated client config to my phone.

I’m using Wireshark ssh remote packet capture, wireshark connects to my odroid, launches tcpdump -U -i eth0 -w - and streams the output to show on wireshark GUI.

The tunnel itself is working so I can ping my phone from my odroid using vpn IP:

# ping 10.9.0.2
PING 10.9.0.2 (10.9.0.2) 56(84) bytes of data.
64 bytes from 10.9.0.2: icmp_seq=1 ttl=64 time=37.9 ms
64 bytes from 10.9.0.2: icmp_seq=2 ttl=64 time=14.8 ms
64 bytes from 10.9.0.2: icmp_seq=3 ttl=64 time=134 ms

But I can’t get my web browsing pushed through the vpn. Interesting part is that traffic from my vpn goes out from my eth0 to the internet and back from the internet to eth0. Here I pinged google from my phone connected to the vpn, request pushed through the vpn is going to 172.217.16.14 from my Odroid eth0 192.168.0.201, then response back to eth0 from google:

But on return it doesn’t reach wg0 for some reason. This is how it looks when capturing wg0 interface:

Nothing gets back.

trendy · 14 January 2022 10:05

What is the output of iptables-save -c ?

dr-rox · 14 January 2022 11:12

iptables-save -c

# iptables-save -c
# Generated by iptables-save v1.8.7 on Fri Jan 14 13:11:08 2022
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
[0:0] -A FORWARD -i wg0 -j ACCEPT
COMMIT
# Completed on Fri Jan 14 13:11:08 2022
# Generated by iptables-save v1.8.7 on Fri Jan 14 13:11:08 2022
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
[19:1305] -A POSTROUTING -o eth0 -j MASQUERADE
COMMIT
# Completed on Fri Jan 14 13:11:08 2022

trendy · 14 January 2022 13:39

echo “1” > /proc/sys/net/ipv4/ip_forward
After you verify that it works, fix it in /etc/sysctl.conf .

dr-rox · 14 January 2022 14:06

Doesn’t help, my forwards are already enabled:

# sysctl -a | grep forward
net.ipv4.conf.all.forwarding = 1
net.ipv4.conf.all.mc_forwarding = 0
net.ipv4.conf.default.forwarding = 1
net.ipv4.conf.default.mc_forwarding = 0
net.ipv4.conf.eth0.forwarding = 1
net.ipv4.conf.eth0.mc_forwarding = 0
net.ipv4.conf.lo.forwarding = 1
net.ipv4.conf.lo.mc_forwarding = 0
net.ipv4.conf.wg0.forwarding = 1
net.ipv4.conf.wg0.mc_forwarding = 0
net.ipv4.ip_forward = 1
net.ipv4.ip_forward_use_pmtu = 0
net.ipv6.conf.all.forwarding = 0
net.ipv6.conf.all.mc_forwarding = 0
net.ipv6.conf.default.forwarding = 0
net.ipv6.conf.default.mc_forwarding = 0
net.ipv6.conf.eth0.forwarding = 1
net.ipv6.conf.eth0.mc_forwarding = 0
net.ipv6.conf.lo.forwarding = 0
net.ipv6.conf.lo.mc_forwarding = 0
net.ipv6.conf.wg0.forwarding = 1
net.ipv6.conf.wg0.mc_forwarding = 0

Joulinar · 14 January 2022 14:20

usually forwarding should be set by wg0.conf

Can you share your config files? Remove personal key data and/or ddns entries. I don’t have a Ordoid but on RPi Bullseye it’s working fine.

Can you try to capture all interfaces using tcpdump? Just to see all in one picture. I used following command to see any communication

ssh root@192.168.x.x "/usr/bin/tcpdump -i any -U -w - 'not (host 192.168.x.x and port 22)'" | "C:\Program Files\Wireshark\Wireshark.exe" -i - -k

dr-rox · 14 January 2022 16:54

wg0.conf

[Interface]
Address = 10.9.0.1/24
PrivateKey = ***
ListenPort = 30000

PostUp = sysctl net.ipv4.conf.%i.forwarding=1 net.ipv4.conf.$(ip r l 0/0 | mawk '{print $5;exit}').forwarding=1
PostUp = sysctl net.ipv6.conf.$(ip r l 0/0 | mawk '{print $5;exit}').accept_ra=2
PostUp = sysctl net.ipv6.conf.%i.forwarding=1 net.ipv6.conf.$(ip r l 0/0 | mawk '{print $5;exit}').forwarding=1
PostUp = iptables -A FORWARD -i %i -j ACCEPT; iptables -t nat -A POSTROUTING -o $(ip r l 0/0 | mawk '{print $5;exit}') -j MASQUERADE
PostUp = ip6tables -A FORWARD -i %i -j ACCEPT; ip6tables -t nat -A POSTROUTING -o $(ip r l 0/0 | mawk '{print $5;exit}') -j MASQUERADE
PostDown = iptables -D FORWARD -i %i -j ACCEPT; iptables -t nat -D POSTROUTING -o $(ip r l 0/0 | mawk '{print $5;exit}') -j MASQUERADE
PostDown = ip6tables -D FORWARD -i %i -j ACCEPT; ip6tables -t nat -D POSTROUTING -o $(ip r l 0/0 | mawk '{print $5;exit}') -j MASQUERADE

# Client 1
[Peer]
PublicKey = ***
AllowedIPs = 10.9.0.2/32

Client config

[Interface]
Address = 10.9.0.2/24
PrivateKey = ***
[Peer]
PublicKey = ***
AllowedIPs = 0.0.0.0/0, ::/0
Endpoint = my-ip-address:30000
PersistentKeepalive = 25

And this is WIreshark capture on any interfaces, pinging google:

trendy · 14 January 2022 23:46

It seems as if the connection tracking is not working. Or the timeout for conntrack is too low. Did you change anything like that?

dr-rox · 15 January 2022 11:50

No, it’s a fresh install. What’s a good way to check the conntrack?

trendy · 15 January 2022 22:11

You’d need to install the conntrack userspace utility.

dr-rox · 17 January 2022 13:53

For timeouts they look like this:

# sysctl net.netfilter | grep timeout
net.netfilter.nf_conntrack_dccp_timeout_closereq = 64
net.netfilter.nf_conntrack_dccp_timeout_closing = 64
net.netfilter.nf_conntrack_dccp_timeout_open = 43200
net.netfilter.nf_conntrack_dccp_timeout_partopen = 480
net.netfilter.nf_conntrack_dccp_timeout_request = 240
net.netfilter.nf_conntrack_dccp_timeout_respond = 480
net.netfilter.nf_conntrack_dccp_timeout_timewait = 240
net.netfilter.nf_conntrack_frag6_timeout = 60
net.netfilter.nf_conntrack_generic_timeout = 600
net.netfilter.nf_conntrack_icmp_timeout = 30
net.netfilter.nf_conntrack_icmpv6_timeout = 30
net.netfilter.nf_conntrack_sctp_timeout_closed = 10
net.netfilter.nf_conntrack_sctp_timeout_cookie_echoed = 3
net.netfilter.nf_conntrack_sctp_timeout_cookie_wait = 3
net.netfilter.nf_conntrack_sctp_timeout_established = 432000
net.netfilter.nf_conntrack_sctp_timeout_heartbeat_acked = 210
net.netfilter.nf_conntrack_sctp_timeout_heartbeat_sent = 30
net.netfilter.nf_conntrack_sctp_timeout_shutdown_ack_sent = 3
net.netfilter.nf_conntrack_sctp_timeout_shutdown_recd = 0
net.netfilter.nf_conntrack_sctp_timeout_shutdown_sent = 0
net.netfilter.nf_conntrack_tcp_timeout_close = 10
net.netfilter.nf_conntrack_tcp_timeout_close_wait = 60
net.netfilter.nf_conntrack_tcp_timeout_established = 432000
net.netfilter.nf_conntrack_tcp_timeout_fin_wait = 120
net.netfilter.nf_conntrack_tcp_timeout_last_ack = 30
net.netfilter.nf_conntrack_tcp_timeout_max_retrans = 300
net.netfilter.nf_conntrack_tcp_timeout_syn_recv = 60
net.netfilter.nf_conntrack_tcp_timeout_syn_sent = 120
net.netfilter.nf_conntrack_tcp_timeout_time_wait = 120
net.netfilter.nf_conntrack_tcp_timeout_unacknowledged = 300
net.netfilter.nf_conntrack_udp_timeout = 30
net.netfilter.nf_conntrack_udp_timeout_stream = 180

trendy · 18 January 2022 08:27

These look good to me.
Try to ping from the dietpi itself using the wg IP: ping -I 10.9.0.1 8.8.4.4

dr-rox · 18 January 2022 10:57

Ping doesn’t work when running from wg0 with that command.

Running wireshark on any interface looks like this:

trendy · 18 January 2022 12:31

There is something wrong with the masquerade on your dietpi.
I’d say it might be faster to setup a new sdcard with a very basic installation and only add the wg vpn service to try again, rather than troubleshoot further on this one for now.

dr-rox · 18 January 2022 13:03

Did a clean flash with Wireguard alone and still doesn’t work… Oh well, maybe will use older version for the time…

Joulinar · 18 January 2022 21:02

in generell it is working. Personally I’m using it on daily basis on my RPi4B hosting PiHole + WireGuard server.

MichaIng
Any idea why WireGuard server is not working on Odroid? If I’m not mistaken our implementation is same on all device.

MichaIng · 19 January 2022 20:15

Yes it should be the same, but comes pre-compiled with the kernel on RPi while it is compiled on install on Odroids. I’m getting an Odroid XU4 soon and can check whether I can replicate.

One idea, can you show:

ls -l /etc/alternatives/iptables

Probably nftables is used by default while not being supported well with that kernel yet. Solution would be:

update-alternatives --set iptables /usr/sbin/iptables-legacy
update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
systemctl restart wg-quick@wg0

dr-rox · 26 January 2022 14:57

MichaIng
This fixed it!! Thanks!

The command listed nft as iptables

# ls -l /etc/alternatives/iptables
lrwxrwxrwx 1 root root 22 Jan 26 14:43 /etc/alternatives/iptables -> /usr/sbin/iptables-nft

Running the update-alternatives fixed my VPN

update-alternatives --set iptables /usr/sbin/iptables-legacy
update-alternatives --set ip6tables /usr/sbin/ip6tables-legacy
systemctl restart wg-quick@wg0