Understanding the TCP 3-Way Handshake with Wireshark

1) What you’ll learn (TL;DR)

• How the TCP handshake works: SYN → SYN/ACK → ACK
• How to capture only what matters (capture vs display filters)
• How to read sequence/ack numbers, and TCP options (MSS, Window Scale, SACK, Timestamps)
• How to measure handshake RTT and spot issues (drops, RSTs, retransmits)
• How to troubleshoot firewall blocks, half-opens, SYN floods, and odd behaviors (TFO, SYN cookies)

2) Lab Topology (SG)

[ SG-Client Laptop ]

Hostname: sg-client
   OS: Ubuntu/Windows/macOS
   IP: 192.168.50.10

[ Home/Office Router / NAT ]

Firewall as a Internet gateway.

[ SG-Web VM ]

   Hostname: web.sanchitgurukul.xyz
   Public IP: 203.0.113.27
   Services: HTTP:80 / HTTPS:443

©sanchitgurukul

You can swap the public IP with your real VM IP; I used the documentation block 203.0.113.0/24 for safety.

3) Prerequisites

On SG-Web (server):

• Linux VM (Ubuntu 22.04+)
• NGINX or Apache installed

      sudo apt update
    

      sudo apt install -y nginx
    

      sudo systemctl enable --now nginx
    

      sudo ufw allow 80/tcp
    

# if you’ll test TLS too

      sudo ufw allow 443/tcp
    

On SG-Client (your laptop/desktop):

• Wireshark installed
• Optional CLI tools: curl, nc (netcat), tcpdump, tshark

4) Quick theory refresher (what to look for)

1. SYN (client → server)
   • Client proposes Initial Sequence Number (ISN C)
   • Sends TCP options: MSS, SACK Permitted, Window Scale, Timestamps (common)
2. SYN/ACK (server → client)
   • Server replies with SYN+ACK
   • Proposes its own ISN (ISN S) and ACKs ISN C + 1
3. ACK (client → server)
   • Client ACKs ISN S + 1 → connection ESTABLISHED

Where in Wireshark?

• Expand Transmission Control Protocol in the packet details:
  • Flags: SYN, ACK, etc.
  • Sequence number / Acknowledgment number
  • Options: MSS, WS, SACK, TSval/TSecr

5) Start capturing (the right way)

Choose one good capture filter (reduces noise)

• Just the server: host 203.0.113.27
• Only HTTP to the server: tcp port 80 and host 203.0.113.27
• Only handshake packets (approx) — better as display filter, see next section.

Tip: Use capture filters to record less, display filters to view less.

Helpful display filters (use in Wireshark’s filter bar)

• Show only handshakes:

      tcp.flags.syn == 1 || tcp.flags.ack == 1 or tcp.flags.syn == 1 || (tcp.flags.ack == 1 && tcp.seq == 1 && tcp.ack > 1)
    

• Classic handshake trio for a specific server:

      tcp && ip.addr == 203.0.113.27 && (tcp.flags.syn == 1 || tcp.flags.ack == 1)
    

• Retransmissions / timeouts:

      tcp.analysis.retransmission || tcp.analysis.lost_segment || tcp.analysis.timeout
    

• Resets:

      tcp.flags.reset == 1
    

6) Generate traffic (live)

On SG-Client:

HTTP (plain):

      curl -I http://sanchitgurukul.xyz
    

# or to a specific IP:

      curl -I http://203.0.113.27

    

HTTPS (TLS):

      curl -I https://sanchitgurukul.xyz --connect-timeout 5
    

Netcat (connection only):

      nc -vz 203.0.113.27 80
    

      nc -vz 203.0.113.27 443
    

Windows (PowerShell):

      Test-NetConnection sanchitgurukul.xyz -Port 80 -InformationLevel Detailed
    

output:
PS C:\Users\sg> Test-NetConnection sanchitgurukul.xyz -Port 80 -InformationLevel Detailed                                                                                                                                                                                                                                                                        ComputerName            : sanchitgurukul.xyz
RemoteAddress           : 203.0.113.27
RemotePort              : 80
NameResolutionResults   : 203.0.113.27
                          
MatchingIPsecRules      :
NetworkIsolationContext : Internet
InterfaceAlias          : Wi-Fi
SourceAddress           : 192.168.68.50
NetRoute (NextHop)      : 192.168.68.1
TcpTestSucceeded        : True

Now switch to Wireshark and watch for SYN → SYN/ACK → ACK.

7) Read a handshake like a pro

Pick the first SYN:

• Note Seq = X (client ISN).
• Check Options in [TCP options]:
  • Maximum segment size (MSS) often 1460 on Ethernet
  • Window scale (WS) e.g., 7 → scale factor 2^7=128
  • SACK permitted
  • Timestamps: TSval, TSecr

Select the SYN/ACK:

• Flags: SYN, ACK
• Seq = Y (server ISN), Ack = X + 1
• Check server’s options (may differ)

Select the final ACK:

• Flags: ACK
• Seq = X + 1, Ack = Y + 1
• After this, data can flow (HTTP GET, TLS ClientHello, etc.)

8) Measure handshake latency (RTT-ish)

Method A (fast & visual):

1. Right-click the SYN, choose Set/Unset Time Reference (or press Ctrl+T).
2. Look at the Time column for SYN/ACK → that delta ≈ server path + server processing.
3. Delta to the final ACK shows client return timing.

Method B (per-flow graph):
Statistics → Flow Graph → TCP flow to see ordering and timing.

Method C (tshark, offline):

# Export one flow as capture.pcapng, then:

      tshark -r capture.pcapng -Y "tcp.flags.syn==1 && tcp.flags.ack==0" -T fields -e frame.number -e frame.time_relative -e ip.src -e tcp.stream > syn.txt
    

      tshark -r capture.pcapng -Y "tcp.flags.syn==1 && tcp.flags.ack==1" -T fields -e frame.number -e frame.time_relative -e ip.src -e tcp.stream > synack.txt
    

# Correlate by tcp.stream to compute deltas (SYN→SYN/ACK) with a quick script/spreadsheet.

9) Advanced: decoding TCP options (what they tell you)

• MSS advertises the largest TCP payload you can accept per segment (impacts fragmentation).
• Window Scale extends receive window beyond 65,535 (look for “WS: N”).
• SACK Permitted enables selective ACKs (crucial for loss recovery).
• Timestamps help with RTT measurement and PAWS (Protection Against Wrapped Sequence numbers).

If you see no options in SYN/ACK (just MSS) from a server under load, it might be using SYN cookies (defense against SYN floods). Cookies can restrict options.

10) Common handshake failure patterns (and how to spot them)

Useful host counters (Linux):

# On server

      sudo ss -s
    

      My Name is Sanchit Agrawal. This is my website. Managed and designed by me.
    

# look at TCP states (SYN-RECV etc.)

      netstat -s | egrep -i 'listen|SYN|reset'
    

# Kernel tunables for SYN backlog

      sysctl net.ipv4.tcp_max_syn_backlog
    

      sysctl net.ipv4.tcp_syncookies
    

11) Bonus: Look at the first app data right after handshake

• HTTP: You’ll see GET / (or HTTP/1.1 200 OK) after the 3rd packet.
• HTTPS: You’ll see ClientHello immediately after the 3rd packet. Filter:

      tls.handshake.type == 1 # ClientHello
    

12) Repro tricks (to learn quickly)

• Force a handshake without much data:

      nc -vz 203.0.113.27 80
    

• Simulate a drop (server drops SYN/ACK) on Linux server:

      sudo iptables -I INPUT -p tcp --sport 80 --tcp-flags SYN,ACK SYN,ACK -j DROP 
    

• Now the client’s SYN will retransmit—watch it in Wireshark—then remove the rule:

      sudo iptables -D INPUT 1
    

13) Handy Wireshark filter snippets (copy/paste)

• Only the 3 packets of a single TCP stream (select a packet → Right-click → Apply as Filter → Selected → A→B then add flags):

      tcp.stream == N && (tcp.flags.syn == 1 || tcp.flags.ack == 1)
    

• Show only TCP options:

      tcp.options
    

• Show SYN packets with Window Scale:

      tcp.flags.syn == 1 && tcp.options.wscale.shift
    

14) Best practices

• Use capture filters to keep pcap sizes small.
• Set the Time Display Format to “Seconds Since Previous Displayed Packet” for quick deltas.
• Capture both ends (client + server) when possible to isolate path issues.
• On high-speed links, consider tcpdump with ring buffers:

      sudo tcpdump -i eth0 -w handshakes.pcapng -C 100 -W 10 'tcp[tcpflags] & (tcp-syn|tcp-ack) != 0 and host 203.0.113.27'
    

• Document the exact time of your test; compare with server logs for correlation.

15) Full step-by-step “live lab” checklist

1. Prepare SG-Web
   • Install/start NGINX, open ports 80/443
   • Verify with curl -I http://localhost on the VM
     • Make sure you’re running this command inside the terminal.
     • Check that the VM is up and the network configuration allows access.
2. Resolve & reachability
   • From SG-Client: nslookup sanchitgurukul.xyz
   • ping 203.0.113.27 (ping may be blocked; not required by TCP)
3. Start Wireshark with a capture filter (e.g., host 203.0.113.27)
4. Trigger the handshake
   • curl -I http://sanchitgurukul.xyz
5. Apply display filter
   • tcp.flags.syn == 1 || tcp.flags.ack == 1
6. Analyze
   • Confirm SYN → SYN/ACK → ACK
   • Read Seq/Ack, MSS, WS, SACK, TS
7. Measure latency
   • Time-reference the SYN; note deltas to SYN/ACK, then ACK
8. Validate app layer
   • HTTP headers or TLS ClientHello follows the 3rd packet
9. Troubleshoot if broken
   • Look for retransmits, RSTs; check firewall/NAT/server service
10. Export evidence
    • File → Export Specified Packets… (pcapng and PNG screenshots)

16) Quick “What good looks like”

• Handshake completes with no retransmissions
• MSS aligns with path MTU (e.g., 1460 over Ethernet)
• WS & SACK present (modern stacks)
• SYN→SYN/ACK delta is reasonable for geography (e.g., <20–50 ms metro, 100–250 ms intercontinental)
• App data immediately follows (GET/ClientHello)

17) Summary

The TCP 3-way handshake is small but mighty: it negotiates capabilities and verifies both sides are ready. With the right capture and display filters, you can quickly validate the handshake, measure latency, and pinpoint failures—whether it’s a sleepy service, a strict firewall, or a congested path.

18) Useful Links

https://www.wireshark.org/docs/relnotes

https://sanchitgurukul.com/basic-networking

https://sanchitgurukul.com/network-security