What is transport mode and tunnel mode explain about the scope of AH and ESP in these modes?

When using VPN as a client, there are two distinct modes that can be available. These are usually transparent to the end user, but are important to understand. Making an assumption of which option is in use can lead to delayed issue resolution as well as frustration by the engineer and client.

For the purpose of this post, we're primarily focusing on transport mode and tunnel mode in terms of the IPSEC protocol. It’s important to note that Request For Comments (RFCs) were published in the mid 1990s and superseded by new ones a few years later. This is a suite of protocols that has held up over time for IPv4 and refreshed for IPv6.

What is a VPN?

VPN stands for Virtual Private Network. Traditionally, they used to secure network traffic from one site or endpoint to another. By secure, we mean encrypt such that the contents cannot be viewed except for by the intended parties/sites. Most of the time this is used to encrypt those connections over the Internet. This is not the only use case though. Some organizations require encryption over private circuits such as MPLS, VPLS or Point to Point. While they are private connections, they are not secured nor encrypted and a VPN helps with that.

What Is Transport Mode?

Transport mode is an available option for creating a tunnel. This is primarily reserved for Point-to-Point tunnels. Those are where endpoints only need to communicate directly with each other. In today's world of SD-WAN where full mesh VPN tunnels are at the push of a few buttons this may seem crazy but you do have to remember when these types of tunnels started out. In the mid 1990's, encryption was not everywhere as it is today.

A key differentiator in transport mode is that the IP headers are not encrypted. It is only the payload of the data. Because the data is not really encapsulated like tunnel mode, the headers need to be unencrypted so routers can look at the source and destination and determine how to route them. For this reason, the original source and destination are viewable to everyone. This may not be an issue but it is the main distinction point aside from tunneling sites versus specific endpoints.

Transport Mode: Use Case 1

Prior to IPSEC, tunneling protocols existed. They served the purpose of connecting sites together over the internet. These protocols did not use much if any encryption. Security was fairly relaxed in the early days of the internet. At the time protocols like L2TP (Layer Two Tunneling Protocol) or Cisco's GRE (Generic Routing Encapsulation) existed. While they provided great tunneling capabilities, their encryption was limited to nonexistent.

It was easy to just configure a transport mode tunnel between sites. It served the purpose of encrypting these already existing tunnels. It allowed organizations to ease into IPSEC without having to rebuild their tunnels in transport mode. If they used something like GRE, the tunnel worked, they just wanted to layer on encryption to make it more secure. Their knowledge of GRE configuration and troubleshooting wouldn't go to waste.

Transport Mode: Use Case 2

In some cases, entire sites do not need to be encrypted, just two endpoints on the internet. In that case, transport mode could be used, particularly if you are not concerned about encrypting the IP headers and obscuring the original sources and destinations. This can work well for two endpoints that communicate directly over the internet, particularly in cases where the protocol is more or less plain text/raw. This is compared to modern applications and protocols that may use modern encryption techniques like TLS.

Transport Mode: Use Case 3

In some cases, point- to-site (P2S) connections may use a transport tunnel to encrypt connectivity between a client and a VPN concentrator. They may do this because they then use a proprietary tunneling protocol as mentioned in Use Case 1. This is fairly rare today though as they typically use IPSEC Tunnel Mode or a form of TLS/SSL/HTTPS tunnels.

Expounding on it a bit though, a Transport Mode tunnel is very similar to a TLS or HTTPS connection in that the IP header information is viewable but the payload is encrypted and protected. It is a modern day example of a Transport Mode tunnel.

What Is Tunnel Mode?

Tunnel mode is the most popular today between the two IPSEC modes. It takes roughly the same amount of configuration to setup but allows access to an entire site. It’s usually used For site-to-site (S2S) tunnels or P2S tunnels. While entire sites are usually in scope though, Firewall ACLs can further restrict access to actual endpoints.

Unlike transport mode though, the entire original packet is encapsulated, encrypted and then appended to a new packet. The outer packet appears to communicate between the public endpoints. Someone sniffing the packets cannot view the internal packets which show the original source, destination or payloads. This not only provides great protection (encryption) but also provides some obscurity too.

Tunnel Mode: Use Case 1

In the case of S2S tunnels, an entire site is tunneled to another site privately. This works very much like GRE as we discussed earlier except that tunnel mode provides the GRE functionality and the encryption all in the same technology. It is a singular configuration, instead of configuring two separate protocols and tunnels. As engineers become more familiar with this, it just made sense as IPSEC is a standardized protocol so it interoperated with many different vendors. In its early days while the RFCs were being revised it was a little rough but it has gone through quite a bit to become universally accepted and used between nearly all vendors.

Tunnel Mode: Use Case 2

For a P2S tunnel, this is typically your end-user VPN client connection where you have the end client (Point) tunneling to an entire site. Many times this acts just like a normal S2S tunnel except one side (client side) is a /32 or singular IP address in the tunnel endpoint.

This is the use case that most people are familiar with for corporate workers. Particularly those with older and legacy thick clients that require direct access to back end servers to communicate.

Tunnel Mode Hesitancy

In some cases, there may be IP overlaps that prevent tunnel mode. There are tricks to work around this such as smaller subnets or doing network address translation (NAT) to remap the overlap. Sometimes these can be tricky and therefore Transport Mode may be more acceptable in some of those edge cases.

In IPSEC's early days, there was a little hesitancy and skepticism so the tried and true tunneling protocols were used. Those environments typically required homogeneous environments as many of the tunnel protocols were vendor specific. In today's world though, it is very convenient to have a protocol mode that both tunnels and encrypts, unifying the technology and that works across most if not all vendors on the market.

Which Mode To Use?

Generally, tunnel mode is what you're going to run across most often as an IT pro. The topic of transport mode versus tunnel mode does not come up and it’s just assumed to be tunnel mode. Unless you have a very legacy environment using older protocols, tunnel mode is your choice. In other cases you may have an extreme edge case and in those scenarios you usually know you need transport mode.

What is transport mode and tunnel mode explain about the scope of AH and ESP in these modes?

Download

Tunnel Mode is a method of sending data over the Internet where the data is encrypted and the original IP address information is also encrypted. 

The Encapsulating Security Payload (ESP) operates in Transport Mode or Tunnel Mode. In Tunnel Mode, ESP encrypts the data and  the IP header information. 

The Internet Security (IPsec) protocol uses ESP and Authentication Header (AH) to secure data as it travels over the Internet in packets. ESP handles data encryption and some authentication of data. AH only provides authentication. Both protocols may be used independently or they may be grouped as IPsec. IPsec is used in virtual private networks (VPNs).

Example:

“VPN connections are intended to conceal the source of the information being transmitted within the network and among its users. That’s why information carried this way uses ESP tunnel mode so the information itself and the IP header info are not visible.”

Differentiate between the transport mode and tunnel mode of IP Sec and explain how authentication and confidentiality are achieved using IP Sec.

What is transport mode and tunnel mode explain about the scope of AH and ESP in these modes?

IPSec’s protocol objective is to provide security services for IP packets such as encrypting sensitive data, authentication, protection against replay and data confidentiality.

As outlined in our IPSec protocol article, Encapsulating Security Payload (ESP) and Authentication Header (AH) are the two IPSec security protocols used to provide these security services.  Analysing  the ESP and AH protocols is out of this article’s scope, however you can turn to our IPSec article where you’ll find an in-depth analysis and packet diagrams to help make the concept clear.

 

Understanding IPSec Modes –Tunnel Mode & Transport Mode

IPSec can be configured to operate in two different modes, Tunnel and Transport mode. Use of each mode depends on the requirements and implementation of IPSec.

 

IPSec Tunnel Mode

IPSec tunnel mode is the default mode. With tunnel mode, the entire original IP packet is protected by IPSec. This means IPSec wraps the original packet, encrypts it, adds a new IP header and sends it to the other side of the VPN tunnel (IPSec peer).

Tunnel mode is most commonly used between gateways (Cisco routers or ASA firewalls), or at an end-station to a gateway, the gateway acting as a proxy for the hosts behind it.

Tunnel mode is used to encrypt traffic between secure IPSec Gateways, for example two Cisco routers connected over the Internet via IPSec VPN. Configuration and setup of this topology is extensively covered in our Site-to-Site IPSec VPN article. In this example, each router acts as an IPSec Gateway for their LAN, providing secure connectivity to the remote network:

What is transport mode and tunnel mode explain about the scope of AH and ESP in these modes?


Another example of tunnel mode is an IPSec tunnel between a Cisco VPN Client and an IPSec Gateway (e.g ASA5510 or PIX Firewall). The client connects to the IPSec Gateway. Traffic from the client is encrypted, encapsulated inside a new IP packet and sent to the other end. Once decrypted by the firewall appliance, the client’s original IP packet is sent to the local network.

In tunnel mode, an IPSec header (AH or ESP header) is inserted between the IP header and the upper layer protocol. Between AH and ESP,  ESP is most commonly used in IPSec VPN Tunnel configuration.

The packet diagram below illustrates IPSec Tunnel mode with ESP header:

What is transport mode and tunnel mode explain about the scope of AH and ESP in these modes?

 ESP is identified in the New IP header with an IP protocol ID of 50.

The packet diagram below illustrates IPSec Tunnel mode with AH header:

What is transport mode and tunnel mode explain about the scope of AH and ESP in these modes?

The AH can be applied alone or together with the ESP, when IPSec is in tunnel mode. AH’s job is to protect the entire packet. The AH does not protect all of the fields in the New IP Header because some change in transit, and the sender cannot predict how they might change. The AH protects everything that does not change in transit. AH is identified in the New IP header with an IP protocol ID of 51.

 

IPSec Transport Mode

IPSec Transport mode is used for end-to-end communications, for example, for communication between a client and a server or between a workstation and a gateway (if the gateway is being treated as a host).  A good example would be an encrypted Telnet or Remote Desktop session from a workstation to a server.

What is transport mode and tunnel mode explain about the scope of AH and ESP in these modes?

Transport mode provides the protection of our data, also known as IP Payload, and consists of TCP/UDP header + Data, through an AH or ESP header. The payload is encapsulated by the IPSec headers and trailers. The original IP headers remain intact, except that the IP protocol field is changed to ESP (50) or AH (51), and the original protocol value is saved in the IPsec trailer to be restored when the packet is decrypted.

IPSec transport mode is usually used when another tunneling protocol (like GRE) is used to first encapsulate the IP data packet, then IPSec is used to protect the GRE tunnel packets. IPSec protects the GRE tunnel traffic in transport mode.


The packet diagram below illustrates IPSec Transport mode with ESP header:

What is transport mode and tunnel mode explain about the scope of AH and ESP in these modes?

Notice that the original IP Header is moved to the front. Placing the sender’s IP header at the front (with minor changes to the protocol ID), proves that transport mode does not provide protection or encryption to the original IP header and ESP is identified in the New IP header with an IP protocol ID of 50.

The packet diagram below illustrates IPSec Transport mode with AH header:

What is transport mode and tunnel mode explain about the scope of AH and ESP in these modes?

The AH can be applied alone or together with the ESP when IPSec is in transport mode. AH’s job is to protect the entire packet, however, IPSec in transport mode does not create a new IP header in front of the packet but places a copy of the original with some minor changes to the protocol ID therefore not providing essential protection to the details contained in the IP header (Source IP, destination IP etc). AH is identified in the New IP header with an IP protocol ID of 51.

In both ESP and AH cases with IPSec Transport mode, the IP header is exposed.

Back to Network Protocols Section