Understanding TCP MSS, IP MTU, TCP MSS Clamping and SSL/TCP Proxy Behavior

—
One of the most misunderstood networking topics is the relationship between IP MTU, TCP MSS, TCP MSS Clamping, and SSL/TCP proxy devices such as Load Balancers, Firewalls, Reverse Proxies, and Application Delivery Controllers — ADCs.
Many engineers assume that if a TCP endpoint advertises an MSS of:
1380 bytes
every TCP segment transmitted must contain exactly:
1380 bytes
of TCP payload.
That assumption is incorrect.
MSS defines a maximum TCP payload size for the receiving direction. It is not a required segment size.
Similarly, it is sometimes assumed that a Load Balancer simply forwards the original TCP packets between the client and server.
That is not necessarily true.
When a Load Balancer or ADC operates as a Full TCP Proxy, it terminates the client-side TCP connection and establishes another TCP connection toward the backend server.
The two TCP connections can therefore have different:

• MSS values

• Window sizes

• TCP options

• Sequence-number spaces

• Congestion-control state

• RTT characteristics

• TLS sessions

• Network MTUs

Understanding these relationships is essential when troubleshooting:

• Slow applications

• SSL/TLS offloading

• IP fragmentation

• Path MTU problems

• TCP MSS mismatches

• Blocked ICMP messages

• VPN tunnels

• GRE tunnels

• IPsec overhead

• SD-WAN

• Cloud networks

• Data-center Load Balancers

Understanding IP MTU and TCP MSS

What Is IP MTU?

MTU — Maximum Transmission Unit defines the maximum size of an IP packet that can be transmitted over a particular link/interface without requiring IP-layer fragmentation.
For a normal TCP/IPv4 packet without IP or TCP options, the IP packet consists conceptually of:

• IPv4 Header

• TCP Header

• TCP Payload

Common MTU Examples

What Is TCP MSS?

TCP MSS — Maximum Segment Size is the maximum TCP payload size that an endpoint advertises it is willing to receive in a TCP segment.
MSS refers to the TCP data portion.
It does not include the IP or TCP header itself.
For a basic TCP/IPv4 packet without additional header options:
MSS = MTU - IPv4 Header - TCP Header
Using the common minimum header sizes:
MSS = 1500 - 20 - 20
Therefore:
MSS = 1460 bytes
For IPv6 with its 40-byte base header and a 20-byte TCP header:
MSS = 1500 - 40 - 20
Therefore:
MSS = 1440 bytes
Additional headers or encapsulation can change the practical calculation.

MTU vs MSS

TCP Three-Way Handshake and MSS Negotiation

The MSS option is normally exchanged in TCP packets containing the SYN flag.
For example:

MSS Is Directional

This point is extremely important.
Suppose:

Therefore, it is misleading to describe a TCP connection as simply having “an MSS of 1380.”
Each endpoint advertises its own receive MSS.

MSS Is a Maximum, Not a Required Size

If an endpoint advertises:
MSS = 1380
the peer can send:

• 1380 bytes

• 1200 bytes

• 800 bytes

• 500 bytes

• 100 bytes

of TCP payload in individual segments.
It does not have to send exactly 1380 bytes every time.

TCP MSS Clamping and Path MTU

Network paths frequently contain additional encapsulation.
Examples include:

• VPN

• GRE

• IPsec

• PPPoE

• SD-WAN

• Cloud tunnels

• MPLS-related service designs

• Other overlay networks

Additional encapsulation consumes bytes.
A path that cannot safely carry the original packet size may therefore experience:

• Fragmentation

• Packet drops

• PMTUD dependency

• Retransmissions

• Application stalls

What Is TCP MSS Clamping?

TCP MSS Clamping modifies the MSS option advertised in TCP SYN packets.
Consider the same example:
Original Client MSS
1380
The gateway firewall changes it to:
1300
The downstream TCP peer therefore sees:
MSS = 1300
rather than:
MSS = 1380

Why MSS Clamping Exists

MSS clamping is commonly deployed where a known reduced-MTU path exists and TCP packet sizing needs to account for that path.
It can be especially useful across:

• VPN tunnels

• GRE

• IPsec

• PPPoE

• Overlay networks

Path MTU Discovery — PMTUD

Path MTU Discovery attempts to determine the largest IP packet that can traverse the complete path without requiring fragmentation.
For IPv4, the classic PMTUD process involves packets that cannot be fragmented and ICMP feedback from a router that encounters a smaller outgoing MTU.
Conceptually:

1. Sender transmits a packet.
2. A router encounters a smaller outgoing MTU.
3. The packet cannot be forwarded at its current size.
4. The router drops it.
5. The router returns an appropriate ICMP message.
6. The sender reduces the packet size.
7. Communication continues.

For IPv6, routers do not fragment transit packets. The source is expected to respond appropriately to ICMPv6 Packet Too Big information.

PMTU Black-Hole Problems

If the required ICMP messages are filtered, a sender may not receive the information needed to adjust packet size.
Symptoms can include:

• TCP retransmissions

• Connections that establish but stall

• Small requests working while large transfers fail

• TLS handshake or application-data stalls

• VPN-specific connectivity problems

SSL/TCP Proxy Behaviour and Independent TCP Sessions

Modern Load Balancers and Application Delivery Controllers may operate as Full TCP Proxies.
Examples include products or software from:

• A10

• F5

• Citrix

• NGINX

• HAProxy

• Envoy

The exact behavior depends on the product and configuration.
In a Full Proxy architecture, the device does not simply forward the original TCP connection unchanged from client to server.
Instead, it can create two independent TCP connections.

Client-Side Connection

Client ↔ Load Balancer

Server-Side Connection

Load Balancer ↔ Backend Server

Example Proxy Behaviour

The two sessions can independently negotiate or maintain:

• MSS

• Window scaling

• SACK

• Timestamps

• Sequence numbers

• Flow-control state

• Congestion-control state

TLS can also be independent when TLS termination or re-encryption is involved.

SSL/TLS Offloading Example

A common architecture is:

The backend connection can therefore use:

• Different MSS

• Different network MTU

• Different TCP window

• Different TCP options

• Different TLS session if re-encryption is used

Complete Real-World Packet Flow Example

We will now use the original packet-flow example throughout this section.
The topology is:
Client → Gateway Firewall → Load Balancer → Backend Server
The important values are:

• Client originally advertises MSS 1380

• Gateway Firewall clamps the MSS to 1300

• Load Balancer terminates the client-side TCP connection

• Load Balancer creates a new backend TCP connection

• Load Balancer advertises MSS 1380 toward the server

• Backend Server advertises MSS 1400

Step 1 — Client Sends TCP SYN

The client starts the TCP connection.

Conceptually:

Step 2 — Gateway Firewall Performs MSS Clamping

The firewall inspects the TCP SYN.
It changes:
MSS 1380 → MSS 1300

The firewall then forwards the modified SYN toward the Load Balancer.

The downstream peer therefore sees the client’s advertised receive MSS as:
1300 bytes
rather than the original:
1380 bytes

Step 3 — Load Balancer Receives and Terminates the Connection

The Load Balancer receives:
SYN MSS = 1300
The client-side TCP handshake completes.
The resulting connection is:
TCP Session #1
Client ↔ Gateway Firewall ↔ Load Balancer
The Load Balancer terminates this TCP session.
If SSL offloading is enabled, it also terminates the client-side TLS session.

Step 4 — Load Balancer Creates the Backend TCP Session

The Load Balancer now creates a new connection toward the backend server.
This is:
TCP Session #2
Load Balancer ↔ Backend Server
The backend handshake uses the original example:

Backend MSS Advertisement

Directional Effective Payload Limits

Because MSS represents what the receiving endpoint advertises:

End-to-End Session Summary

What Did the Firewall Actually Change?

The firewall changed the MSS option in the TCP SYN:
1380 → 1300
It did not automatically change the MSS of the independent backend connection created later by the Load Balancer.

What Did the Load Balancer Do?

The Load Balancer:

1. Accepted the client-side TCP connection.
2. Completed the client-side TCP handshake.
3. Terminated the TCP connection locally.
4. Terminated TLS if SSL offloading was configured.
5. Processed the application traffic.
6. Created an independent backend TCP connection.
7. Negotiated backend TCP parameters separately.

TCP Segment Size, PMTU Problems and Troubleshooting

Can a Load Balancer Send a 100-Byte TCP Segment?

Yes.
Suppose the peer has advertised:
MSS = 1380
The Load Balancer could send:

TCP Payload = 100 Bytes
TCP Payload = 100 Bytes
TCP Payload = 100 Bytes

This is valid.
MSS specifies the maximum, not the minimum.
Smaller TCP segments can occur because of:

• Small application writes

• Proxy behavior

• Buffering

• Streaming

• Flow control

• Congestion-control behavior

• TCP pacing

• Latency-sensitive application behavior

• Interaction with Nagle’s algorithm and delayed ACK behavior

The receiving TCP stack reassembles the byte stream for the application.

What If an Application Writes 2000 Bytes?

Suppose the applicable peer-advertised MSS is:
1380
The application writes:
2000 bytes
to the TCP socket.
This does not normally mean that TCP transmits one 2000-byte TCP payload.
The TCP stack can segment the byte stream into multiple TCP segments.
A simplified conceptual example is:

Important Wireshark Consideration — TCP Offload

Packet captures taken directly on a transmitting or receiving host can sometimes display unexpectedly large TCP segments because of operating-system and NIC offload features such as:

• TSO

• GSO

• GRO

• LRO

Therefore, a host capture may appear to show packets larger than the expected MSS even though the packets transmitted on the physical network are segmented differently.

IPv4 Fragmentation vs IPv6

IPv4 and IPv6 fragmentation behavior differs.
For IPv4, fragmentation may occur when permitted, although modern networks generally try to avoid relying on fragmentation.
For IPv6, routers do not fragment transit packets. The source is expected to react to ICMPv6 Packet Too Big information and adjust appropriately.

Common MTU/MSS Symptoms

Possible symptoms include:

• TCP handshake succeeds but data stalls

• Small requests work but larger responses fail

• TLS ClientHello or early packets succeed but later traffic stalls

• Repeated retransmissions

• VPN-specific application failures

• Large uploads fail

• Large downloads stall

These symptoms are useful indicators, but none individually proves an MTU problem.

Useful Linux Tests

Check interface MTU:

ip link show

Inspect TCP SYN MSS:

tcpdump -ni any 'tcp[tcpflags] & tcp-syn != 0'

Capture traffic for later analysis:

tcpdump -ni any -s 0 -w mtu-mss-test.pcap

Useful Wireshark Filters

Display TCP SYN packets:

tcp.flags.syn == 1

Display TCP MSS options:

tcp.options.mss_val

Display retransmission analysis:

tcp.analysis.retransmission

Recommended Multi-Point Capture

For the example in this article, useful capture locations are:

1. Client side before MSS clamping
2. Firewall egress after MSS clamping
3. Load Balancer client-side interface
4. Load Balancer server-side interface
5. Backend server

This allows you to prove:

• Original client MSS = 1380

• Firewall-clamped MSS = 1300

• Client-side TCP termination

• Independent backend SYN

• Load Balancer backend MSS = 1380

• Server backend MSS = 1400

MTU and MSS Troubleshooting Workflow

Best Practices and Key Takeaways

MTU Best Practices

Use an MTU appropriate for the actual network design.
A physical Ethernet MTU of:
1500
is common, but tunnels and overlays require sufficient underlay headroom or adjusted inner MTU.
Do not force 1500 everywhere if the actual path cannot support it.

Allow Required ICMP

Do not indiscriminately block all ICMP.
PMTUD relies on important ICMP and ICMPv6 messages.
Blocking them can contribute to difficult-to-diagnose connectivity failures.

Use MSS Clamping Where Appropriate

MSS clamping can be useful across known reduced-MTU paths such as:

• VPN

• GRE

• IPsec

• PPPoE

• SD-WAN

• Overlay networks

Calculate the value according to the actual path rather than selecting an arbitrary number.

Avoid Unnecessary Fragmentation

Fragmentation introduces additional processing and can complicate:

• Security inspection

• Troubleshooting

• Packet loss behavior

• Performance

Where possible, design the network so endpoints send appropriately sized packets.

Understand Proxy Boundaries

A Full TCP Proxy creates a clear transport boundary.
For this article’s example:

Final Key Takeaways — MTU, MSS and PMTUD

1. MTU defines the maximum IP packet size supported by a particular link or interface without IP fragmentation.
2. Path MTU represents the limiting MTU across an end-to-end network path.
3. MSS represents the maximum TCP payload an endpoint advertises it is willing to receive.
4. MSS does not include the IP or TCP header.
5. MSS is directional.
6. MSS is a maximum, not a required TCP segment size.
7. A TCP sender can transmit segments smaller than the peer’s advertised MSS.
8. Applications write a TCP byte stream; the TCP stack handles segmentation.
9. MSS Clamping modifies the MSS option carried in TCP SYN packets.
10. MSS clamping can help TCP operate across reduced-MTU tunnel paths.
11. MSS clamping is not a complete replacement for correct MTU and PMTUD design.
12. Required ICMP and ICMPv6 messages are important for Path MTU Discovery.
13. Blocking required PMTUD feedback can contribute to PMTU black-hole behavior.
14. IPv4 and IPv6 fragmentation behavior is different.

Final Key Takeaways — Full Proxy and Troubleshooting

1. A Full TCP Proxy terminates one TCP connection and creates another.
2. Client-side and server-side proxy connections negotiate TCP parameters independently.
3. SSL offloading can terminate the client TLS session independently of the backend connection.
4. An MSS clamped from 1380 to 1300 on the client side does not automatically force the backend connection to use 1300.
5. In the backend example, the Load Balancer advertises 1380 and the server advertises 1400.
6. Host packet captures can be misleading because of TSO, GSO, GRO, and LRO.
7. Multi-point packet captures are one of the best ways to prove where MSS changes occur.
8. TCP retransmissions alone do not prove an MTU problem.

Useful Links and References

SanchitGurukul Static Resources

Official External References