The other day I had weird problems with masqueraded connections. Certain HTTP connections were simply going deaf after a random amount of time. The behaviour seemed random at first, but then I discovered that only specific webservers like www.nmap.org or www.speedtest.net are affected.
The server, doing the masquerading was based on Centos 7. It was also using the new FirewallD system, so I thought the problem might be there. Normally, I wouldn't bother, but this one somehow didn't let me sleep.
Following the leads, I discovered that certain TCP packets are not coming through. How come, if all the PING replies are there? I could confirm that the really big packets are being lost. This brought me to the TCP fragmentation.
enp1s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500 ppp0: flags=4305<UP,POINTOPOINT,RUNNING,NOARP,MULTICAST> mtu 1492
See the MTU (maximum transmission unit) value? I guessed certain packets are not being properly fragmented, thus not passing the route. Doing some reading, I found this:
When a user requests a web site, a client/server negotiation occurs between the PC and the web server that hosts the web site. During the negotiation, a maximum MTU size is negotiated. Since the PC negotiates and its default MTU size is 1500 bytes (Windows 3x, 9x, NT, ME, and so forth), the web server negotiates an MTU size of 1500 bytes. Therefore, regardless of the MTU size you configure on the router, the web server still sends packets up to 1500 bytes in size.
Luckily the guys at Netfilter wrote an extension that solves the problem. In the words of the author.
This patch by Marc Boucher <firstname.lastname@example.org> adds a new target that allows you to examine and alter the MSS value of TCP SYN packets, to control the maximum size for that connection. As explained by Marc himself, THIS IS A HACK, used to overcome criminally brain-dead ISPs or servers which block ICMP Fragmentation Needed packets.
Typical usage would be :
# iptables -A FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
So, applying this to the new firewalld system solved the problem.
firewall-cmd --permanent --direct --add-passthrough ipv4 -I FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu
A little bit of explanation here.