A Complete Guide with Real Packet Flow Examples
One of the most misunderstood networking topics is the relationship between IP MTU, TCP MSS, TCP MSS Clamping, and SSL/TCP Proxy devices such as Load Balancers, Firewalls, Reverse Proxies, and ADCs.
Many engineers assume that if a client advertises an MSS of 1380 bytes, every packet transmitted must be exactly 1380 bytes.
That assumption is incorrect.
Similarly, many people believe a Load Balancer simply forwards TCP packets. Modern Layer-4/Layer-7 load balancers usually do not. When SSL Offloading or Full Proxy mode is enabled, the Load Balancer terminates one TCP session and creates another completely independent TCP session toward the backend server.
Understanding this behaviour is essential for troubleshooting:
- Slow applications
- SSL offloading
- Fragmentation
- PMTU issues
- MSS mismatch
- Firewalls dropping ICMP
- VPN tunnels
- GRE/IPsec overhead
- Data center load balancers
This article explains everything from first principles.
What is IP MTU?
MTU (Maximum Transmission Unit) is the maximum IP packet size that can be transmitted over a network interface without fragmentation.
MTU includes:
- IP Header
- TCP/UDP Header
- Payload
Example:
Common MTU Values
| Technology | Typical MTU |
| Ethernet | 1500 |
| Jumbo Frame | 9000 |
| PPPoE | 1492 |
| GRE Tunnel | 1476 |
| IPsec VPN | 1400–1460 |
| VXLAN | 1450 |
| WireGuard | ~1420 |
What is TCP MSS?
TCP MSS (Maximum Segment Size) is the largest TCP payload a sender is willing to receive in a single TCP segment.
It does not include:
- Ethernet Header
- IP Header
- TCP Header
It includes only:
TCP Data
Formula:
MTU vs MSS
| MTU | MSS |
| IP packet size | TCP payload size |
| Layer 3 | Layer 4 |
| Includes headers | Payload only |
| Interface property | TCP connection property |
TCP Three-Way Handshake and MSS Negotiation
The advertised MSS is the maximum payload each endpoint is willing to receive.
The effective payload size in each direction is determined independently:
- Client → Server: limited by the server’s advertised receive MSS.
- Server → Client: limited by the client’s advertised receive MSS.
A sender may transmit smaller segments than the negotiated MSS at any time, but should not exceed the peer’s advertised receive MSS.
Relationship Between MTU and MSS
TCP MSS Clamping
Sometimes the network path contains smaller MTUs due to:
- VPN
- GRE Tunnel
- MPLS
- PPPoE
- IPSec
- Cloud Interconnect
- SD-WAN
Instead of allowing fragmentation, a firewall or router can rewrite the MSS value in SYN packets.
Example
The server therefore limits its TCP payloads sent to the client to 1300 bytes, reducing the chance of fragmentation.
Why MSS Clamping Exists
SSL/TCP Proxy Behaviour
Modern Load Balancers such as:
- A10
- F5 BIG-IP
- Citrix ADC
- NGINX
- HAProxy
- Envoy
often operate as full TCP proxies.
They terminate the client’s TCP session and establish a separate TCP session to the backend.
| Step | Network Operation | MSS |
|---|---|---|
| 1 | Client sends SYN | 1380 |
| 2 | TCP Session #1 established (Client ↔ Load Balancer) | 1380 |
| 3 | Load Balancer terminates the client connection | — |
| 4 | Load Balancer creates a new TCP connection | 1400 |
| 5 | TCP Session #2 established (Load Balancer ↔ Server) | 1400 |
| 6 | Server communicates with Load Balancer using the new session | 1400 |
These sessions are independent.
Each side negotiates:
- MSS
- Window Size
- SACK
- Timestamps
- Congestion Control
- TLS Parameters (if applicable)
independently.
SSL Offloading Example
| Step | Traffic Flow | Protocol |
|---|---|---|
| 1 | Client | HTTPS |
| 2 | Client → Load Balancer | Encrypted SSL/TLS |
| 3 | Load Balancer | Terminates SSL/TLS Session |
| 4 | Decrypts the Client Traffic | SSL Offloading |
| 5 | Applies Security Policies (Optional) | WAF / Authentication / Inspection |
| 6 | Creates a New Backend Connection | Independent TCP Session |
| 7 | Load Balancer → Server | HTTP or HTTPS |
| 8 | Server | Processes the Application Request |
The backend may use:
- HTTP
- HTTPS
- Different MSS
- Different MTU
- Different Window Size
without affecting the client-side negotiation.
Complete Real-World Packet Flow
Step 1 – Client Sends TCP SYN
| Packet | Value |
|---|---|
| Source | Client |
| Destination | Server (through network) |
| TCP Flag | SYN |
| Advertised MSS | 1380 Bytes |
Step 2 – Gateway Firewall Performs MSS Clamping
| Before Firewall | After Firewall |
|---|---|
| MSS = 1380 | MSS = 1300 |
Firewall Action
| Operation | Description |
|---|---|
| Packet Inspection | Reads the TCP SYN packet |
| MSS Rewrite | Changes MSS from 1380 → 1300 |
| Purpose | Prevents fragmentation on smaller MTU paths |
Server Perspective
| Server Learns |
|---|
| Client Maximum Receive MSS = 1300 Bytes |
Step 3 – Load Balancer Receives the Connection
| Client Side (TCP Session #1) | |
|---|---|
| Client MSS | 1300 (after firewall clamping) |
| TCP Handshake | Completed |
| SSL | TLS Session Established |
| Load Balancer | Terminates Client TCP Session |
| SSL Offloading | Decrypts Client Traffic |
| Next Action | Creates a new backend TCP connection |
Step 4 – Backend TCP Session
Backend TCP Handshake
| Packet | Advertised MSS |
|---|---|
| Load Balancer → Server | SYN (MSS = 1380) |
| Server → Load Balancer | SYN-ACK (MSS = 1400) |
| Load Balancer → Server | ACK |
Backend Negotiation
| Device | Advertised MSS |
|---|---|
| Load Balancer | 1380 Bytes |
| Server | 1400 Bytes |
Effective MSS
| Direction | Maximum TCP Payload |
|---|---|
| Server → Load Balancer | 1380 Bytes |
| Load Balancer → Server | 1400 Bytes (limited by the server’s receive capability, which is 1400 in this example) |
Note: In each direction, the sender must not exceed the receiver’s advertised MSS. This means the server sending to the load balancer is limited by the load balancer’s advertised MSS (1380 bytes), while the load balancer sending to the server is limited by the server’s advertised MSS (1400 bytes).
End-to-End Session Summary
| TCP Session | Endpoints | MSS | Description |
|---|---|---|---|
| TCP Session #1 | Client ↔ Gateway Firewall ↔ Load Balancer | 1300 (after MSS clamping) | Client-side TCP connection with SSL/TLS |
| TCP Session #2 | Load Balancer ↔ Server | 1380 / 1400 | Independent backend TCP connection |
Key Takeaways
| ✔ | Description |
|---|---|
| ✔ | The firewall modifies only the MSS option in the TCP SYN packet. |
| ✔ | The Load Balancer terminates the client TCP session. |
| ✔ | SSL is decrypted during SSL Offloading (if enabled). |
| ✔ | The Load Balancer creates a new, independent TCP session to the backend server. |
| ✔ | Each TCP session negotiates its own MSS, Window Size, TCP Options, and TLS parameters independently. |
What Happens if the Client Sends More Than Its Advertised MSS?
Suppose the client advertised:
1380 MSS
Then later sends:
2000 Bytes of TCP payload
In normal TCP implementations, this should not happen because the sender’s TCP stack segments application data into TCP segments that respect the peer’s advertised MSS.
If an application writes 2000 bytes, the TCP stack typically splits it into multiple segments (for example, 1380 bytes and 620 bytes, subject to other factors).
If an oversized IP packet somehow appears (for example, due to encapsulation, misconfiguration, or non-TCP traffic), then:
- If the IP packet exceeds the path MTU and fragmentation is allowed, routers or the sender may fragment it (IPv4).
- If the DF (Don’t Fragment) bit is set and the packet exceeds the path MTU, the packet is dropped and an ICMP “Fragmentation Needed” message should be returned (Path MTU Discovery).
- If ICMP is filtered, Path MTU Discovery may fail, causing retransmissions, timeouts, or “black hole” connections.
Can a Load Balancer Send 100-Byte TCP Segments?
Yes.
Assume negotiated MSS:
1380
The Load Balancer sends:
100 Bytes
100 Bytes
100 Bytes
This is perfectly valid.
MSS is a maximum, not a minimum.
TCP may send smaller segments for many reasons, including:
- Small application writes
- Nagle algorithm behavior
- Delayed ACK interactions
- TCP pacing
- Congestion control
- Proxy buffering or streaming
- Flow control
- Low-latency delivery
The receiver reassembles these segments into the original byte stream before delivering the data to the application.
Path MTU Discovery (PMTUD)
Instead of fragmenting packets, modern networks rely on Path MTU Discovery (PMTUD) to learn the smallest MTU along the path.
Basic process:
- Sender transmits packets with the Don’t Fragment (DF) bit set (IPv4).
- If a router cannot forward the packet because it exceeds the outgoing interface MTU, it drops the packet.
- The router sends an ICMP “Fragmentation Needed” message indicating the supported MTU.
- The sender reduces its packet size and continues transmitting.
If these ICMP messages are blocked by firewalls, communication can stall. MSS clamping is commonly used to reduce the likelihood of such PMTU issues, especially across tunnels and VPNs.
Best Practices
✔ Use MTU 1500 whenever possible.
✔ Enable PMTU Discovery.
✔ Allow required ICMP “Fragmentation Needed” messages through firewalls.
✔ Configure MSS Clamping on VPN and tunnel interfaces where appropriate.
✔ Avoid unnecessary IP fragmentation.
✔ Monitor TCP retransmissions and out-of-order segments during troubleshooting.
✔ Remember that proxy devices create independent TCP sessions and negotiate TCP options separately.
RFC References
- RFC 791 – Internet Protocol (IPv4)
- RFC 793 – Transmission Control Protocol (updated by RFC 9293)
- RFC 879 – TCP Maximum Segment Size and Related Topics
- RFC 1191 – Path MTU Discovery (IPv4)
- RFC 1981 – Path MTU Discovery for IPv6 (updated by RFC 8201)
- RFC 8201 – Path MTU Discovery for IPv6
- RFC 9293 – Transmission Control Protocol (TCP) Specification
Key Takeaways
- MTU defines the largest IP packet that can traverse a link without fragmentation.
- MSS defines the largest TCP payload a receiver is willing to accept in a segment.
- MSS Clamping rewrites the MSS value in TCP SYN packets to prevent fragmentation across smaller-MTU paths.
- SSL/TCP proxies terminate one TCP connection and create another, negotiating TCP parameters independently on each side.
- MSS is a maximum, not a required packet size—a sender may legally transmit much smaller segments.
- Applications do not send “1380-byte packets” directly; they write a byte stream, and the TCP stack segments it according to the negotiated MSS, congestion control, and other TCP algorithms.
Useful References
| Resource | Description |
| RFC 791 | Internet Protocol (IPv4) |
| RFC 793 / RFC 9293 | TCP Specification |
| RFC 879 | Maximum Segment Size (MSS) |
| RFC 1191 | Path MTU Discovery (IPv4) |
| RFC 8201 | Path MTU Discovery (IPv6) |
| Wireshark | Analyze MSS negotiation, PMTUD, TCP segmentation, and retransmissions |
| tcpdump | Capture TCP SYN packets and verify MSS values |
| hping3 | Generate custom TCP packets for lab testing |
| OpenSSL | Test TLS handshakes with SSL offloading devices |
Useful Links
| Resource | Type | Link |
|---|---|---|
| IETF RFC1191 | External reference | Open |
| IETF RFC791 | External reference | Open |
| IETF RFC9293 | External reference | Open |
| SanchitGurukul Home | SanchitGurukul | Open |
| Learn | SanchitGurukul | Open |
| Tools | SanchitGurukul | Open |
| Resources | SanchitGurukul | Open |
| Quizzes | SanchitGurukul | Open |
Your feedback matters
Was this post helpful?
Discussion
0 approved comments
No approved comments yet. You can start the discussion below.
Leave a Comment
No login is required. Name and email are used for moderation/security. Your email is never displayed publicly. All comments require administrator approval.
Discover more from SanchitGurukul
Subscribe to get the latest posts sent to your email.