IP Security (IPSec) is defined and standardized in RFC 2401. Following is an excerpt from that RFC, which defines the IPSec framework:

IPSec provides security services at the IP layer by enabling a system to select required security protocols, determine the algorithm(s) to use for the service(s), and put in place any cryptographic keys required to provide the requested services. IPSec can be used to protect one or more “paths” between a pair of hosts, between a pair of security gateways, or between a security gateway and a host. The term “security gateway” is used throughout the IPSec documents to refer to an intermediate system that implements IPSec protocols. For example, a router or a firewall implementing IPSec is a security gateway.

We cover IPSec in detail in many of our security courses including the CompTIA Security+ certification course with labs and exams.

The IPSec framework is a set of open standards developed by the Internet Engineering Task Force (IETF). IPSec is implemented by a set of cryptographic protocols for securing IP datagrams. The IPSec framework secures IP traffic operating at the Network Layer of the OSI Model, thus securing all network applications and communications that use the IP network.

While IPSec is defined in a number of RFCs, it is not a mandatory requirement for IP version 4. However, IPSec is a mandatory requirement for IP version 6. Using a combination of hashing, symmetric key, and asymmetric key cryptographic algorithms, the IPSec framework offers the following security services:

• Peer authentication
• Data confidentiality
• Data integrity
• Data origin authentication
• Replay detection
• Access control
• Traffic flow confidentiality

NOTE: A hash, also referred to as a message digest, is a unique number from a sequence of text that is generated by applying a mathematical formula. The basic function of a hash is to take a random-length block of data and return a fixed-size bit string, which is then referred to as the hash value or message digest. Hashing is used to provide data integrity. Symmetric key cryptography uses a single key for the encryption and decryption process and is commonly referred to as secret key or pre-shared key cryptography. Asymmetric key cryptography uses two keys for the encryption and decryption process (i.e., one key for encryption and another key for decryption). Asymmetric key cryptography is also referred to as public-key cryptography.

IPSec uses Security Association (SA) to provide security to a given IP connection. IPSec SAs are unidirectional, which means that two SAs are required for bidirectional data communications sessions. Internet Security Association and Key Management Protocol (ISAKMP) describes the framework for key management and defines the procedure and packet format necessary to establish, negotiate, modify, and delete Security Association. ISAKMP offers the identification of the peers only; it does not offer a key exchange mechanism. ISAKAMP is documented in RFC 2408 but was made obsolete by RFC 4306, which standardized Internet Key Exchange (IKE) version 2.

Internet Key Exchange is a hybrid protocol. It is basically a combination of ISAKMP, Oakley key exchange, and the SKEME protocol. Oakley key exchange describes a series of key exchanges, called modes, and details the services provided by each. These services include perfect forward secrecy for keys, identity protection, and authentication. The SKEME protocol describes a versatile key exchange technique that provides anonymity, non-repudiation, and quick key refreshment.

Internet Key Exchange defines the mechanism for exchanging keys. IKE derives authenticated keying material and negotiates SAs that are used for the Encapsulating Security Payload (ESP) and Authentication Header (AH) protocols. IKE uses UDP port 500 and is documented in RFC 2409, and updated in RFC 4306, which is IKEv2 (the current IKE standard).

NOTE: The terms ISAKMP and IKE are often used interchangeably in many texts because the protocols are almost the same. Therefore, while the remainder of this chapter will include references to IKE, you should be aware of this fact in case you encounter a question using either one of these terms on the ROUTE certification exam.

IKE is a two-phase, multimode protocol that offers the following three methods of authenticating a remote peer:

• Public key signature
• Pre-shared key
• Public key encryption

The public key signature method also referred to as the RSA-Signature method, is the most secure method and requires Public Key Infrastructure (PKI). The pre-shared key method uses statically defined keys and is the most common method because of its ease of deployment; however, this method is not scalable or secure. Finally, public key encryption, commonly referred to as the RSA-Nonce, is similar to public-key signature but requires prior knowledge of the peer’s public key. However, it is important to know that this method has limited support and is not as widely implemented or used as the other two methods.

Internet Key Exchange phase 1 verifies the identity of a remote peer and the two peers establish a secure authentication communications channel. This phase is primarily concerned with the protection suite for IKE messages. IKE phase 1 also protects the negotiation of phase 2 communication. IKE phase 2 is used to protect data and establish SA for IPSec. IKE phase 2 is used to negotiate the protection suite (i.e., ESP and AH), the algorithms that will be used within the protection suite (e.g., DES, 3DES, AES, and SHA-1), the networks or traffic that is being encrypted (which is sometimes referred to as proxy identities or phase 2 identities), and any optional keying material for negotiated protocols.

NOTE: ESP and AH are IPSec headers that are used to protect data. These are described later in this section. DES, 3DES, AES, and SHA-1 are symmetric key cryptography standards that are used to encrypt the contents of a message in order to provide data confidentiality. Because you are not required to demonstrate any advanced security knowledge, this guide will not be going into any further detail on these standards in this chapter.

IPSec Modes

IPSec can use two different modes to secure data communications using one of two modes. The two modes that are used to secure a given IP connection are as follows:

• Transport mode
• Tunnel mode

Transport mode protects the payload (data) of the original IP datagram. This mode is typically used for host-to-host and end-to-end sessions. Tunnel mode, on the other hand, is used to protect data in network-to-network (i.e., between networks). This is the mode used to secure data traversing between two routers in site-to-site VPN implementations in Cisco IOS software.

In transport mode, the original IP header is retained while the IPSec header is inserted between the original IP header and the payload as illustrated in Figure 1 below:

Fig. 1. IPSec Transport Mode

Unlike transport mode, tunnel mode allows IPSec to encapsulate the entire IP packet (i.e., the header as well as the payload) in a new IP packet. Tunnel mode is illustrated in Figure 2 below:

Fig. 2. IPSec Tunnel Mode

In both transport and tunnel mode, IPSec adds a new header to all the IP packets to provide the required information for securing the data within the original IP datagram. This header may be either an Encapsulating Security Payload (ESP) header or an Authentication Header (AH) header. ESP is an IP-based protocol that uses IP port 50 for communication between IPSec peers. ESP is documented in RFC 4303 and is used to provide confidentiality, integrity, and authenticity of the data. Additionally, ESP also provides anti-replay protection services. ESP does not provide any kind of protection for the outer IP header.

Authentication Header (AH) is an IP-based protocol that uses IP port 51 for communication between IPSec peers. AH is used to protect the integrity and authenticity of data, and also offers anti-replay protection; however, AH does not provide confidentiality protection. Unlike ESP, AH provides protection to the IP header; however, not all fields in the IP header are encrypted. For example, the TTL field is not encrypted. AH is described in RFC 2402.