Understanding the TCP 3-Way Handshake with Wireshark

—
The TCP 3-way handshake is one of the most important concepts in computer networking.
Before two devices can exchange application data using TCP, they normally need to establish a TCP connection.
This happens through three basic messages:
SYN → SYN/ACK → ACK
For example, when a client connects to a web server using HTTPS, the TCP connection normally needs to be established before the TLS handshake begins.
Understanding these three packets helps troubleshoot many common problems, including:

• Server not responding

• Firewall blocking a connection

• Port not listening

• TCP resets

• High connection-establishment latency

• SYN retransmissions

• Routing problems

• NAT problems

• Asymmetric paths

• SYN floods

In this article, we will use Wireshark to capture a real TCP handshake, understand sequence and acknowledgment numbers, inspect TCP options, measure connection-establishment timing, and troubleshoot common failures.

—

Understanding the TCP 3-Way Handshake

The TCP handshake allows both endpoints to establish connection state before exchanging application data.
The three steps are:

Step 1 — Client Sends SYN

The client starts the connection by sending a TCP packet with the SYN flag set.
Conceptually:
Client → Server: SYN
The SYN communicates:
“I want to establish a TCP connection.”
The packet also contains an Initial Sequence Number, or ISN.
Suppose the client selects:

The client may also advertise TCP capabilities such as:

• Maximum Segment Size

• Window Scale

• SACK Permitted

• TCP Timestamps

Step 2 — Server Sends SYN-ACK

If the server accepts the connection attempt, it replies with SYN and ACK flags set.
Conceptually:
Server → Client: SYN-ACK
Suppose the client sequence number was:

Why 1001?
Because a SYN consumes one sequence number.
The server also chooses its own Initial Sequence Number.
For example:

The SYN-ACK therefore does two things:

• Acknowledges the client’s SYN

• Sends the server’s own SYN

Step 3 — Client Sends ACK

The client acknowledges the server’s SYN.

After this exchange, the TCP connection is established.

Why Wireshark May Show Seq = 0

When you capture a real handshake, Wireshark may show:

SYN

Seq = 0

SYN-ACK

Seq = 0
Ack = 1

ACK

Seq = 1
Ack = 1

This does not mean the real TCP Initial Sequence Number is zero.
Wireshark normally uses relative TCP sequence numbers to make packet analysis easier.

—

Build the SG TCP Handshake Lab

We can now capture a real handshake.

Lab Topology

Use one client and one web server.

Example Lab Addresses

The 203.0.113.0/24 network is reserved for documentation and examples.
Replace the example server address with your actual lab server address.

SG-Web Requirements

You can use a Linux VM running NGINX or Apache.
Check NGINX:

sudo systemctl status nginx

Start it if necessary:

sudo systemctl start nginx

Check listening TCP ports:

sudo ss -lntp

A web server listening on TCP 80 may appear similar to:

LISTEN 0 511 0.0.0.0:80 0.0.0.0:*

SG-Client Requirements

Install Wireshark.
Optional troubleshooting tools include:

• curl

• netcat

• tcpdump

• tshark

—

Capture and Analyze the Handshake

Wireshark provides two different filter types.

Capture Filters

Capture filters determine which packets Wireshark records.

Display Filters

Display filters control which packets from an existing capture Wireshark displays.
These use different syntax.

Capture Traffic to One Server

Use this capture filter:

host 203.0.113.27

Capture HTTP

tcp port 80 and host 203.0.113.27

Capture HTTPS

tcp port 443 and host 203.0.113.27

Generate HTTP Traffic

From SG-Client:

curl -I http://203.0.113.27/

Or use your actual hostname:

curl -I http://example.com/

Generate HTTPS Traffic

curl -I https://example.com/

Generate Only a TCP Connection With Netcat

For HTTP:

nc -vz 203.0.113.27 80

For HTTPS:

nc -vz 203.0.113.27 443

Windows PowerShell Test

Test-NetConnection 203.0.113.27 -Port 443

Example successful result:

ComputerName     : 203.0.113.27
RemotePort       : 443
TcpTestSucceeded : True

Display Initial SYN Packets

tcp.flags.syn == 1 && tcp.flags.ack == 0

Display SYN and SYN-ACK Packets

tcp.flags.syn == 1

Because both the initial SYN and SYN-ACK contain the SYN flag, this filter shows both.

Find a Specific TCP Connection

The easiest approach is usually to identify one packet from the connection and inspect its TCP stream.
Wireshark assigns a stream number such as:

tcp.stream = 7

Filter that connection with:

tcp.stream == 7

Replace 7 with the actual stream number from your capture.

What a Healthy Handshake Looks Like

A simplified packet list may appear as:

Client → Server    SYN
Server → Client    SYN, ACK
Client → Server    ACK
Client → Server    TLS ClientHello

For plain HTTP, the next packet may instead contain an HTTP request.

Inspect the SYN Packet

Select the SYN.
Expand:
Transmission Control Protocol
Look at:

• Source port

• Destination port

• Sequence number

• Flags

• Window

• TCP options

Inspect SYN-ACK

Select the server response.
Verify:

• SYN = set

• ACK = set

• Server sequence number

• Client acknowledgment

• Server TCP options

Inspect Final ACK

Select the third packet.
Verify:

• ACK = set

• SYN = not set

• Client acknowledges the server’s SYN

The TCP connection is now established.
—

Understanding TCP Options and Handshake Timing

The handshake does more than confirm connectivity.
It also allows endpoints to advertise important TCP capabilities.

Maximum Segment Size — MSS

MSS tells the remote endpoint the largest TCP payload the sender is prepared to receive in one TCP segment for that connection.
A common value on a standard IPv4 Ethernet network is:

MSS = 1460

This commonly corresponds to:
1500-byte Ethernet IP MTU
minus
20-byte IPv4 header
minus
20-byte TCP header
However, MSS is not always 1460.
It may differ because of:

• IPv6

• VPNs

• Tunnels

• PPPoE

• Interface MTU

• TCP options and implementation behavior

• MSS adjustment by network devices

Window Scale

The TCP receive-window field is limited in size.
Window Scale allows TCP to represent a much larger receive window.
For example:

Window scaling is useful on high-bandwidth or high-latency connections where a larger receive window can improve throughput.

SACK Permitted

SACK means Selective Acknowledgment.
If packet loss occurs, SACK can help the receiver tell the sender which portions of data have already arrived.
This can improve loss recovery.

TCP Timestamps

TCP timestamps may include:

• TSval

• TSecr

They can support TCP timing mechanisms and PAWS protection.

Compare Client and Server Options

The options in SYN and SYN-ACK do not have to be identical.
For example:

Different advertised values are not automatically evidence of a problem.

Measuring SYN to SYN-ACK Time

One useful measurement is the time from:
Client SYN
to
Server SYN-ACK observed at the client
This is a useful approximation of connection-establishment response time from that observation point.
It contains network propagation plus any relevant server-side response delay.
It should not automatically be interpreted as pure network RTT.

Method 1 — Time Reference

Select the SYN.
Right-click and choose:
Set/Unset Time Reference
Then inspect the time of the SYN-ACK.

Method 2 — Time Display Format

Wireshark can display time relative to the previous displayed packet.
This is useful for quickly examining packet-to-packet delays.

Method 3 — Flow Graph

Open:
Statistics → Flow Graph
Use the graph to examine packet ordering and timing.

—

Troubleshooting TCP Handshake Problems

The TCP handshake is extremely useful for determining where a connection fails.

Scenario 1 — SYN Repeats but No SYN-ACK Arrives

A client sends:

SYN

SYN Retransmission

SYN Retransmission

SYN Retransmission

No server response is visible.
Possible causes include:

• Server unreachable

• Firewall silently dropping packets

• Security group blocking the port

• Routing problem

• NAT problem

• Service path problem

• Server response returning through another path

Scenario 2 — SYN Followed by RST

You may see:

Client → Server    SYN
Server → Client    RST, ACK

A common cause is that the destination host is reachable but no service is listening on the requested TCP port.
Check the server:

sudo ss -lntp

For TCP 443:

sudo ss -lntp | grep ':443'

Scenario 3 — Handshake Completes and Then RST Appears

You may see:

SYN

SYN-ACK

ACK

RST

This means the initial TCP handshake completed.
Do not immediately blame MSS, MTU, or Window Scale.
Possible explanations include:

• Application closes the connection

• Client aborts

• Server aborts

• Firewall or proxy injects a reset

• Protocol mismatch

• Policy rejection

Determine which endpoint actually sent the RST.
Check:

• Source IP

• Destination IP

• TTL or Hop Limit where useful

• Sequence numbers

• Capture location

• Timing

• Application behavior

Scenario 4 — SYN-ACK Is Slow

A large SYN-to-SYN-ACK delay does not automatically prove server overload.
Possible contributors include:

• Geographic distance

• Congestion

• Packet loss

• Slow server response

• Firewall processing

• NAT

• Network path changes

• Virtualized infrastructure

Compare against a known-good baseline.

Scenario 5 — SYN-ACK Repeats

You may see:

Client → Server    SYN
Server → Client    SYN-ACK
Server → Client    SYN-ACK Retransmission
Server → Client    SYN-ACK Retransmission

A likely interpretation is that the server has not observed the expected final ACK.
Possible causes include:

• Final ACK lost

• Client-side firewall

• Return-path problem

• Asymmetric routing

• NAT issue

• Packet capture taken at only one point

Scenario 6 — SYN Flood or Many Half-Open Connections

A server receiving many SYN packets without completing handshakes may accumulate connections in SYN-RECV state.
Check Linux TCP state:

ss -ant state syn-recv

Count them:

ss -ant state syn-recv | wc -l

Check TCP statistics:

netstat -s | grep -i syn

Large numbers of incomplete handshakes can result from:

• Traffic spikes

• Scanning

• Network loss

• Misconfiguration

• SYN-flood attacks

Do not classify traffic as an attack based only on SYN packets.
Investigate source distribution, rates, completion ratios, baselines, and server state.

SYN Cookies

Operating systems may use SYN cookies to help protect the SYN backlog during overload or SYN-flood conditions.
SYN cookies can affect how TCP options are represented or retained depending on implementation.
Do not use the absence of one TCP option as proof that SYN cookies are active.
Verify server counters and operating-system behavior.
Linux example:

sysctl net.ipv4.tcp_syncookies

Complete TCP Handshake Troubleshooting Workflow

Troubleshooting Matrix

—

Practical Labs and Best Practices

The following exercises help turn the handshake theory into practical troubleshooting skills.

Lab 1 — Capture a Healthy HTTP Handshake

Start Wireshark with:

tcp port 80 and host 203.0.113.27

Start the capture.
Generate traffic:

curl -I http://203.0.113.27/

Find the connection and inspect:

SYN

SYN-ACK

ACK

HTTP Request

Lab 2 — Capture HTTPS

Use:

tcp port 443 and host 203.0.113.27

Generate:

curl -I https://example.com/

A simplified sequence may look like:

SYN

SYN-ACK

ACK

TLS ClientHello

This clearly separates:
TCP connection establishment
from:
TLS connection establishment

Lab 3 — Test a Closed Port

Choose a port that you know is closed on your controlled lab server.
For example:

nc -vz 203.0.113.27 65000

Capture the result.
A reachable host with a closed TCP port may return a reset.
Compare that behavior with a silently filtered port.

Lab 4 — Simulate a Silent Drop

Use only on your own controlled Linux lab server.
Temporarily drop incoming TCP SYN packets for a dedicated test port:

sudo iptables -I INPUT -p tcp --dport 8080 --syn -j DROP

From SG-Client:

nc -vz 203.0.113.27 8080

Observe the SYN retransmissions.
After the test, remove the rule:

sudo iptables -D INPUT -p tcp --dport 8080 --syn -j DROP

Lab 5 — Compare Client and Server Captures

Start a capture on SG-Client.
At the same time, capture on SG-Web:

sudo tcpdump -ni any tcp port 443

Generate one connection.
Compare both captures.

Useful Wireshark Filters

Initial SYN only:

tcp.flags.syn == 1 && tcp.flags.ack == 0

SYN and SYN-ACK:

tcp.flags.syn == 1

RST packets:

tcp.flags.reset == 1

Wireshark-detected retransmissions:

tcp.analysis.retransmission

Fast retransmissions:

tcp.analysis.fast_retransmission

Duplicate ACK analysis:

tcp.analysis.duplicate_ack

One TCP stream:

tcp.stream == 7

SYN packets containing MSS:

tcp.flags.syn == 1 && tcp.options.mss_val

SYN packets containing Window Scale:

tcp.flags.syn == 1 && tcp.options.wscale.shift

Best Practices

For reliable TCP handshake troubleshooting:

1. Start the capture before generating the connection.
2. Use a narrow capture filter where practical.
3. Identify one TCP stream before detailed analysis.
4. Inspect both directions of the connection.
5. Check sequence and acknowledgment numbers.
6. Inspect TCP options in SYN and SYN-ACK.
7. Measure timing instead of guessing.
8. Identify the actual sender of any RST.
9. Do not diagnose packet loss from a single capture point when multiple capture points are available.
10. Correlate packet timestamps with firewall, proxy, load-balancer, and server logs.

—

Summary

The TCP 3-way handshake establishes a TCP connection using three basic steps:
SYN → SYN-ACK → ACK
The client sends SYN.
The server responds with SYN-ACK.
The client completes the handshake with ACK.
After that, application communication can begin.
For example:
HTTP
TCP handshake completes, then the HTTP request can begin.
HTTPS over TCP
TCP handshake completes, then the TLS handshake can begin.
Wireshark allows you to inspect:

• SYN

• SYN-ACK

• ACK

• Sequence numbers

• Acknowledgment numbers

• MSS

• Window Scale

• SACK

• TCP timestamps

• Retransmissions

• Resets

• Handshake timing

The handshake is also an excellent troubleshooting tool.
Repeated SYN packets suggest that the expected response has not been observed by the client.
Repeated SYN-ACK packets suggest that the server has not observed the expected final ACK.
A reset indicates an endpoint or intermediate device deliberately terminated or rejected the TCP connection.
A completed handshake followed by an application failure means you should continue troubleshooting beyond basic TCP establishment.

—

Useful Links and References

SanchitGurukul Static Resources

Related SanchitGurukul Articles

Official External References