IPsec VPN Tunnel to AWS VPC using PFsense – Packet loss and connectivity issues.
Like most trying to accomplish this I found the post here.
I was having issues with packet loss when testing with ICMP packet over an IPSEC tunnel. Hardware was a Dell R610 with on-board Broadcom 4 port interface cards. I configured each tunnel and they would both come online if you clicked the button to enable the tunnel. What they don’t tell you is this will cause routing issues. You can’t have both tunnels up at the same time. You might make the same mistake if you click the PLAY button next to both tunnels on the IPSEC status page.
There are a few key settings to get the VPC to work properly and have a fast transfer rate. This configuration also had little to no packet loss from the multiple networks that used the VPC.
1) Critical – make sure only one IPSEC tunnel is started first. Use the tunnel they have defined as 1st in the list. If the second was enable AND in the upstate I would get packet loss and slow performance. You can have it enabled just don’t click the button to force it up. If you are having packet loss try stopping the 2nd tunnel.
2) MMS clamping on IPSEC tunnel. This is under the advanced settings of the IPSEC config. Make sure this is set to 1387. Set TCP to be the max size as specified in config you downloaded from Amazon and set this on the WAN interface.
3)Make sure that you summarize your networks as much as possible, I.E. 192.168.0.0/16 10.128.0.0/16 vs /24
Other things that need to be configured:
-If this is a one armed router, then make sure that the setting under NAT/ Networking is allowing traffic to go in/out on the same interface.
-Make sure Ipsec rules allow the correct traffic.
-Modify the settings recommended here for BCE interfaces from the Pfsense Wiki. I performed this since I was using the on-board Broadcom network card in a Dell server. In /boot/loader.conf.local – Add the following (or create the file if it does not exist):
That will increase the amount of network memory buffers, disable TSO directly, and disable msix.
If this is going to be part of a failover cluster. You can then go and configure your CARP settings according to this post.
If you have issues with the tunnel coming up when you fail over the secondary unit you have to disconnect/reconnect the IPSec service located here. I had to sometimes stop/start it 3 or 4 times before traffic would flow over the tunnel after a failover. This was only happening when I failed back and forth between testing. If I just a single failover it worked properly and only dropped a packet or 2.
You should then configure iperf on a VPC and a local machine to test performance.
Start the server with
On a client within your network run:
iperf -c 10.128.128.100 -fM -m -i5 -t25
Other options I configure that were no-standard:
Enabled secure shell.
Firewall Optimization Options Normal
Bypass firewall rules for traffic on the same interface
Enable MSS clamping on VPN traffic 1387
WAN interface MTU 1436