Here are the relevant parts of the OpenVPN 2.4 server config that got me 900+ Mbps iperf3 on GbE LAN. The tunnel was between two PCs with high single-core performance, a Xeon 2450v2 and an i7-3770. OpenVPN uses 50% of a CPU core on the client & server when the tunnel is busy. For reference, I tried running the OpenVPN server on my WiFi router, it peaked out at 60 Mbps.
# Use TCP, I couldn't get good perf out of UDP.
proto tcp
# tun or tap, roughly same perf
dev tun
# Use AES-256-GCM:
# - more secure than 128 bit
# - GCM has built-in authentication, see https://en.wikipedia.org/wiki/Galois/Counter_Mode
# - AES-NI accelerated, the raw crypto runs at GB/s speeds per core.
cipher AES-256-GCM
# Don't split the jumbo packets traversing the tunnel.
# This is useful when tun-mtu is different from 1500.
# With default value, my tunnel runs at 630 Mbps, with mssfix 0 it goes to 930 Mbps.
mssfix 0
# Use jumbo frames over the tunnel.
# This reduces the number of packets sent, which reduces CPU load.
# On the other hand, now you need 6-7 MTU 1500 packets to send one tunnel packet.
# If one of those gets lost, it delays the entire jumbo packet.
# Digression:
# Testing between two VBox VMs on a i7-7700HQ laptop, MTU 9000 pegs the vCPUs to 100% and the tunnel runs at 1 Gbps.
# A non-tunneled iperf3 runs at 3 Gbps between the VMs.
# Upping this to 65k got me 2 Gbps on the tunnel and half the CPU use.
tun-mtu 9000
# Send packets right away instead of bundling them into bigger packets.
# Improves latency over the tunnel.
tcp-nodelay
# Increase the transmission queue length.
# Keeps the TUN busy to get higher throughput.
# Without QoS, you should get worse latency though.
txqueuelen 15000
# Increase the TCP queue size in OpenVPN.
# When OpenVPN overflows the TCP queue, it drops the overflow packets.
# Which kills your bandwidth unless you're using a fancy TCP congestion algo.
# Increase the queue limit to reduce packet loss and TCP throttling.
tcp-queue-limit 256
And here is the client config, pretty much the same except that we only need to set tcp-nodelay on the server:
proto tcp
cipher AES-256-GCM
mssfix 0
tun-mtu 9000
txqueuelen 15000
tcp-queue-limit 256
To test, run iperf3 -s
on the server and connect to it over the tunnel from the client: iperf3 -c 10.8.0.1
. For more interesting tests, run the iperf server on a different host on the endpoint LAN, or try to access network shares.
I'm still tuning this (and learning about the networking stack) to get a Good Enough connection between the two sites, let me know if you got any tips or corrections.
P.S. Here's the iperf3 output.
$ iperf3 -c 10.8.0.1
Connecting to host 10.8.0.1, port 5201
[ 4] local 10.8.2.10 port 39590 connected to 10.8.0.1 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 112 MBytes 942 Mbits/sec 0 3.01 MBytes
[ 4] 1.00-2.00 sec 110 MBytes 923 Mbits/sec 0 3.01 MBytes
[ 4] 2.00-3.00 sec 111 MBytes 933 Mbits/sec 0 3.01 MBytes
[ 4] 3.00-4.00 sec 110 MBytes 923 Mbits/sec 0 3.01 MBytes
[ 4] 4.00-5.00 sec 110 MBytes 923 Mbits/sec 0 3.01 MBytes
[ 4] 5.00-6.00 sec 111 MBytes 933 Mbits/sec 0 3.01 MBytes
[ 4] 6.00-7.00 sec 110 MBytes 923 Mbits/sec 0 3.01 MBytes
[ 4] 7.00-8.00 sec 110 MBytes 923 Mbits/sec 0 3.01 MBytes
[ 4] 8.00-9.00 sec 111 MBytes 933 Mbits/sec 0 3.01 MBytes
[ 4] 9.00-10.00 sec 110 MBytes 923 Mbits/sec 0 3.01 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 1.08 GBytes 928 Mbits/sec 0 sender
[ 4] 0.00-10.00 sec 1.08 GBytes 927 Mbits/sec receiver
iperf Done.
No comments:
Post a Comment