IPsec Explained: ESP, Tunnel Mode and IKEv2
IPsec is an IP-layer security architecture, not a single handshake. Policy selects traffic for protection, Security Associations define how it is processed, and IKEv2 commonly negotiates keys and parameters.
Updated · Editorial approach and simulation limits
IPsec applies policy to IP traffic
RFC 4301 describes IPsec as a security architecture for IPv4 and IPv6. For each packet, policy determines whether it should be protected, discarded or allowed to bypass IPsec. This makes IPsec more than a tunnel switch: the result depends on selectors, local configuration, endpoints and the security services negotiated for the flow.
The Security Policy Database, or SPD, contains ordered policy entries that select traffic using information such as addresses, next-layer protocol and ports. A matching rule can require IPsec protection, permit the packet without protection, or discard it. When troubleshooting, verify the selectors and rule order on both endpoints before assuming that encryption itself failed.
A policy decision is made for outbound and inbound traffic. A mismatch can let one direction work while replies are dropped or processed under a different rule.
Security Associations are directional state
A Security Association (SA) is a set of parameters for applying a security protocol to traffic in one direction. It is identified in the receiving path using values such as the Security Parameters Index, or SPI, along with the destination address and IPsec protocol. The peer generally needs a corresponding SA for the reverse direction.
The Security Association Database stores the state used to process protected packets, including the selected algorithms, keys, sequence state and lifetime information. Two peers can have matching concepts but still fail if the installed SAs disagree about an SPI, traffic selector, algorithm or expiration.
An SA is not the same thing as an entire VPN tunnel, and it is not a single TCP-style connection. A tunnel deployment may contain several pairs of directional SAs and can periodically create replacements before old state expires.
Transport mode and tunnel mode protect different packet views
In transport mode, IPsec protects the payload carried after the original IP header while retaining that header as the outer routing header. This is commonly used for host-to-host protection, subject to policy and implementation requirements. Observers can still see the outer source and destination addresses.
In tunnel mode, the original IP packet is carried inside a new outer IP packet. Gateways can protect traffic between networks while routing on the outer addresses. The original, inner addresses are part of the encapsulated packet and are not used by the outer network to route the packet.
A capture outside a tunnel may show gateway addresses and an ESP packet while hiding the protected inner TCP, UDP or ICMP exchange. A capture on an endpoint after decapsulation may show the original traffic. Always state where the capture was taken.
ESP, AH and the visible fields in a capture
IPsec defines Encapsulating Security Payload (ESP) and Authentication Header (AH) as security protocols. ESP can provide confidentiality, integrity and data-origin authentication according to the selected transform and mode. AH provides integrity and authentication services but does not encrypt the payload; its coverage also interacts differently with fields that can change in transit.
The visible IP protocol number for native ESP is 50; AH uses protocol number 51. The ESP header exposes values needed to identify the SA and sequence position, while the encrypted payload can conceal the inner transport header and application data. These fields are useful for grouping packets but are not enough to recover plaintext.
A visible ESP packet does not tell you which application protocol was protected. Conversely, an IPsec policy can allow some traffic to bypass protection, so finding an unencrypted packet is not automatically evidence that the VPN is broken.
IKEv2 negotiates keys and child SAs
IKEv2, specified in RFC 7296, is commonly used to authenticate peers, negotiate cryptographic parameters and establish IPsec SAs. Its IKE SA protects subsequent IKE messages. Child SAs describe the IPsec data protection used for selected traffic and can be created or rekeyed during the lifetime of the IKE relationship.
The IKE exchange and the protected data path are different conversations. A successful IKE handshake proves that peers established the required IKE state; it does not by itself prove that the right traffic selectors were installed, that an ESP packet traversed the path, or that the application behind the tunnel is healthy.
When reading IKE logs, distinguish negotiation failures from data-plane failures. Authentication, proposal, identity or traffic-selector mismatches appear in control-plane exchanges; an established IKE SA with no matching data traffic points toward routing, policy, selectors or firewall handling.
NAT traversal can place ESP inside UDP
Native ESP is not TCP or UDP and has no transport port field for a NAT device to track. When the peers detect and negotiate NAT traversal, RFC 3948 defines encapsulating ESP packets inside UDP so they can cross Network Address Translators. A trace can then show UDP transport around the ESP payload, commonly on UDP port 4500 in IKE/IPsec deployments.
Seeing UDP/4500 is a clue, not proof that the enclosed traffic is an ordinary UDP application. The payload may be encapsulated ESP or IKE-related traffic, so inspect the packet format and connection state before classifying it.
NAT keepalives exist to keep a mapping alive; RFC 3948 says they should not be used by themselves to decide whether a peer is live. If a tunnel drops after an idle period, correlate keepalive behavior with NAT state and peer logs.
Troubleshoot policy, negotiation and data separately
If no IKE exchange appears, check peer addresses, reachability, firewall rules and the configured initiation behavior. If IKE begins but does not establish, inspect authentication, proposal compatibility, identities and negotiated parameters in the peer logs.
If IKE succeeds but no data passes, compare the installed directional SAs, SPIs, traffic selectors, SPD decisions and route to the remote subnet. Check whether the trace is taken before encapsulation, between gateways or after decapsulation. A single capture point can hide the part of the path that explains the failure.
If ESP packets arrive but the receiver rejects them, investigate SA state, sequence/replay checks, integrity or decryption errors, key lifetime and NAT traversal. Do not publish keys, pre-shared secrets, decrypted payloads or subscriber identifiers in an issue report.
What the IPsec lesson models
Protocol Explorer presents a simplified IPsec sequence to show the relationship between negotiation and protected traffic. Real deployments vary in algorithms, authentication, IKE version, policy, address families, rekey timing and NAT handling. The site does not establish a VPN or inspect visitor traffic.
Use RFC 4301 for the architecture, RFC 7296 for IKEv2 and RFC 3948 for UDP encapsulation of ESP. For a real incident, pair those standards with the endpoint configuration and an authorized capture from a known observation point.
Explore IPsec interactively
Open the matching lesson to follow the exchange packet by packet. This is a scripted learning model with illustrative timing, not a capture of your network. The overview is free; the full lesson is included with the one-time Pro pass.
Standards and primary sources
Use these specifications for authoritative protocol requirements. The lesson summarizes selected behaviors and does not replace the source documents.
Related protocol lessons
Browse all security and access protocols ·