Intrusion Detection and Prevention

The rapid growth of the Internet has resulted in the vast proliferation of viruses, worms, Trojans, and other malicious codes and programs. This makes modern-day networks easy targets because infection can spread across the network rapidly. Due to this growing threat, networks need to be designed and equipped with solid intelligence to diagnose and mitigate threats in real-time. Intrusion Detection Systems and Intrusion Prevention Systems protect networks from unauthorized access by detecting and, if capable, preventing unauthorized access to network resources or hosts.

We cover IDS vs IPS in our free CompTIA Network+ book.

This post is broken up into the following sections:

• Intrusion Detection and Prevention Systems
• IDS and IPS Signatures
• IDS and IPS System Types
• Cisco Network Intrusion Prevention Systems
• Cisco Host Intrusion Prevention Systems
• Cisco IOS IPS Overview and Implementation
• Anomaly Detection and Mitigation

It was originally written for the CCNA Security (IINS) exam which has now retired. You can study the new CyberOps Associate course here.

Intrusion Detection and Prevention Systems

The terms Intrusion Detection System (IDS) and Intrusion Prevention System (IPS) are commonly, but incorrectly, used interchangeably. While IDS and IPS devices perform the same basic functions, there are significant differences in their capabilities, functionality, and implementation, as we will learn through this chapter. It is imperative that you are able to differentiate between an IDS device and an IPS device, as well as their operation, functionality, and implementation within networks and on endpoints.

Intrusion Detection Systems

An Intrusion Detection System is used to detect malicious behavior that can compromise the security and trust of a computer system or network. This includes, but is not limited to, network attacks against vulnerable services, data-driven attacks on applications, host-based attacks such as privilege escalation, unauthorized logins, and access to sensitive files, as well as malware.

Although there are different types of IDS implementations, as we will learn later in this chapter, from a networking standpoint, the term IDS is typically limited to sensors (which refers to the software that runs on the physical appliance and is used to detect threats) that use only promiscuous monitoring based on an out-of-packet stream. Promiscuous monitoring refers to the monitoring of data from anywhere within the network, while the term out-of-packet implies that the IDS is not directly in the physical path of the traffic that it is monitoring. These concepts are illustrated in the following network diagram:

The diagram illustrated above shows the placement of a Network-based IDS device. In step 1, an attack is initiated by the attacker to Host 1. Although the IDS is not directly in the physical traffic path between the attacker and Host 1, by using promiscuous monitoring, the Intrusion Detection System is able to see this attack because a copy of all network traffic between the host and the attacker is sent to it via the port mirroring configuration on the network switch it is connected to, as illustrated in step 2.

NOTE: In Cisco Catalyst switches, administrators can configure the Switched Port Analyzer feature, which allows administrators to mirror traffic received from a port, a group of ports, a VLAN, or a group of VLANs to a destination port where this information can be viewed by the device connected to the port, such as an IDS, for example. Although SPAN configuration is beyond the scope of the IINS course requirements, the following example illustrates how a Cisco IOS Catalyst switch can be configured to mirror all transmit and receive traffic on port FastEthernet1/1 to port FastEthernet5/5, which is connected to an Intrusion Prevention System:

Sw1(config)#monitor session 1 source interface fastethernet 1/1 Sw1(config)#monitor session 1 destination interface fastethernet 5/5

Additionally, SPAN can be used to mirror traffic received from an entire VLAN. The following configuration example illustrates how to mirror all traffic in VLAN 55 to port FastEthernet12/1, which is connected to an Intrusion Prevention System:

Sw1(config)#monitor session 1 source vlan 55 Sw1(config)#monitor session 1 destination interface fastethernet 12/1

Continuing with the IDS example, upon viewing this attack, the IDS will log the information and signal an alert, as illustrated in step 3. It is important to keep in mind that an IDS system cannot prevent the attack from happening on its own; however, it can instruct the firewall to block (shun) this traffic via dynamic ACL configuration, as illustrated in step 4.

Intrusion Prevention Systems

An IPS monitors network and system activities for malicious or unauthorized behavior and reacts in real-time (i.e. is reactive) to block or prevent those activities. Unlike IDS devices, IPS devices typically resides inline, which means that user and network traffic actually traverses and flows through the IPS device. This allows an IPS to drop malicious packets or traffic flows while still allowing all other legitimate traffic flows and packets to pass through. The implementation of a Network-based IPS device is illustrated in the following diagram:

 

As illustrated in the diagram above, the Network-based IPS is in the actual physical path of traffic flow between Host 1 and the Internet, and vice-versa. By residing inline, i.e. in the actual physical traffic path, IPS devices have the capability to respond to suspicious activity by resetting connections themselves. Additionally, IPS devices can reprogram other infrastructure devices (e.g. routers and firewalls) to block network traffic (which is referred to as shunning) from the suspected malicious source. These actions can be performed automatically by the IPS itself or manually by the network security administrator.

While an IPS is typically implemented inline, it is important to remember that it can also be implemented in promiscuous mode, i.e. in a manner similar to an IDS. Additionally, an IPS can also be implemented in both an inline and a promiscuous mode, depending on the number of physical interfaces the device has, as illustrated in the following diagram:

The diagram above illustrates two separate networks within the Company X corporate network: the internal network and the core business unit network. For the purposes of this example, assume that these two networks are separate physical entities. An IPS has been deployed inline within the internal network, using the Gi0 and Gi1 interfaces. This allows the IPS to monitor and block all traffic from internal network hosts to the Internet, and vice-versa.

In addition to this, the IPS is also deployed in a promiscuous manner using the Gi2 interface. This allows the IPS to monitor all traffic from the core business unit to the partner network. However, because this interface is not inline (i.e. is operating in promiscuous mode), the IPS cannot block (shun) traffic between these two networks by itself, but it can instruct the core business unit firewall to shun the traffic using dynamic ACLs and filtering.

IDS and IPS System Architecture

IDS and IPS devices typically use protocol analyzers to decode Application Layer protocols, such as HTTP or FTP, which allows them to evaluate different parts of the protocol for anomalous behavior or exploits, such as embedding techniques used by attackers.

At a very basic level, IDS and IPS appliances are composed of three core components. These components are a console, an engine, and a sensor. The console is used to monitor generated events and alerts, and is also used to control and configure the sensors.

The engine uses a system of rules to generate alerts from received security events and records events logged by the sensor into a database.

The sensor provides packet capturing and analyzing capabilities when monitoring the traffic. This is achieved via the use of multiple micro-engines that the sensor will use to analyze traffic to see if it matches a particular signature. A signature is simply a pattern that the IDS and IPS devices look for in traffic and match against. IDS and IPS signatures are described in detail in the following section.

Although going into specific detail on IDS and IPS system architecture is beyond the scope of the IINS certification requirements, the following diagram illustrates the basic design and architecture of the Cisco IPS Sensor Software system used in Cisco IPS appliances:

The following section provides a brief description of the components illustrated in the diagram:

• The MainApp is the core engine of the sensor operating system. It is responsible for all major functions, including, but not limited to, managing the system processes, configuring the system, starting or stopping other applications, and performing routine maintenance.

• The Event Store is the placeholder for storing all the sensor events, which include system messages, alerts, and errors.

• The SensorApp, which is also referred to as the Analysis Engine, provides packet capturing and analyzing capabilities when monitoring the traffic.

• The CLI is the Command Line Interface that allows network security administrators to manage and configure the sensor. The CLI can be accessed using various methods, which include a direct sensor console connection, Telnet, and Secure Shell (SSH). This is also referred to as the command and control interface.

• IDAPI stands for Integrated Database Application Program Interface or Independent Database Application Program Interface. The sensor operating system applications communicate with each other using IDAPI.

• The Ethernet port is used to connect to the monitored network. This can be placed inline or used to view traffic in a promiscuous operating mode. It is important to know that at a minimum, IPS systems must have two interfaces so that they can be placed inline. IDS devices, on the other hand, can have only a single interface.

The Cisco IPS Sensor OS runs on a Linux platform, which is secured and hardened by removing unnecessary software packages from the OS, disabling unused services, restricting network access, and removing access to the shell.

IDS and IPS Terminology

Before we move on to specific IDS and IPS topics, it is important to understand the terminology that is used when talking about Intrusion Detection Systems and Intrusion Prevention Systems. The following section lists and describes terminology relating to these two technologies that you should be familiar with and are expected to know. These terms are as follows:

• Alert or Alarm
• True attack stimulus
• False attack stimulus
• False positive
• False negative
• Noise
• Site policy
• Site policy awareness
• Confidence value
• Alarm filtering

An alert or alarm is a signal that is sent by an IDS or IPS suggesting a system has been or is being attacked. IDS and IPS devices can send alerts or alarms via numerous communication protocols, all of which will be described in detail later in this chapter.

A true attack stimulus is an IDS or IPS event that triggers the system to produce an alarm and react as though a real attack were in progress. In other words, the system believes an attack is happening, based on configured thresholds or detected signatures in monitored traffic.

A false attack stimulus is an IDS or IPS event that signals the system to produce an alarm when no attack has taken place. This typically occurs when a flow is mistakenly identified as an attack when it is not, in fact, an actual attack.

A false positive is an IDS or IPS alert or alarm that is triggered when no actual attack has taken place. A false positive is basically an incorrect positive alert or alarm. The IDS or IPS incorrectly believes that an attack is occurring and triggers a false alarm.

A false negative is defined as a failure of the system to detect an actual attack. In other words, the IDS or IPS fails to identify an actual attack as a true, legitimate attack, allowing the attacker to access the network or network hosts as though he or she were a legitimate user.

Noise is the data or interference that can cause the IDS or IPS device to trigger a false positive. It is important to keep in mind that noise is based on the interpretation of the vendor manufacturing the IDS or IPS device. For example, the IDS or IPS of one vendor might view a certain application stream as an attack, based on the pattern, while that of another might not.

A site policy provides the guidelines within an organization that control the rules and configurations of the system.

Site policy awareness is the ability the IDS or IPS device has to change its rules and configurations dynamically in response to changing environmental activity.

The confidence value is a value an organization places on itself based on past performance and analysis to help determine its ability to effectively identify an attack.

And, finally, alarm filtering is the process of categorizing attack alerts produced by the system in order to distinguish false positives from actual attacks.

IDS and IPS Detection Methods

Now that we have an understanding of the differences between an IDS and an IPS, as well as the terminology used when discussing IDS and IPS technologies, the following section describes the methods used by an IDS or IPS system to detect attacks. The following four methods are used by IDS and IPS devices to detect attacks:

• Policy-based Detection
• Anomaly-based Detection
• Honey Pot Detection
• Signature-based Detection

The policy-based detection method is dependent on an administrator’s definition of normal and acceptable behavior of the network. For example, the administrator could define normal or acceptable communication between specific types of hosts, such as a web server and its clients, and the IDS or IPS device could then identify any traffic that is considered out-of-profile and generate an alarm or notification to report that activity.

An anomaly-based detection method is a system for detecting computer intrusions and misuse by monitoring system or network activity and classifying it as either normal or abnormal. This classification method is based on rules and will detect any type of misuse that falls out of what is considered normal system or network operation. Anomaly-based detection may be implemented using either statistical or non-statistical anomaly-based detection methods.

• A statistical anomaly-based system establishes a performance baseline based on what the IDS or IPS considers as normal network traffic. It will then sample current network traffic activity to this baseline in order to detect whether it is within baseline parameters. If the sampled traffic is outside baseline parameters, an alarm will be triggered.

For example, assume that there is usually an average of 100 connections to a web server every hour, and all of a sudden the server receives 5000 connections within an hour, the system will consider this an anomaly and generate or trigger an alarm. However, it is important to know that this could actually be legitimate traffic, which means the alarm would be a false positive.

• A non-statistical anomaly-based system allows network security administrators to define what traffic patterns are supposed to look like, and is somewhat similar to a policy-based detection method. As is the case with statistical and policy-based detection methods, a non-statistical anomaly-based method is prone to a significant amount of false positives.

As an example, assume that a company with several thousand employees is using Trend Micro anti-virus software, and the license has expired and needs to be renewed. Based on this, thousands of clients begin to renew and download their licenses and updates, respectively, at the same time. If the IDS or IPS system is using non-statistical anomaly-based detection, it would easily think that this was some kind of attack and generate or trigger an alarm. However, given that this is actually legitimate user traffic, even though it is out-of-profile from what the network security administrators have defined, this would be considered a false positive.

A honey pot is a trap set to lure attackers away from legitimate targets. The system that is designated as the honey pot acts as an attractive attack target, and attackers attack it, instead of other legitimate network servers and devices, because it appears to be more valuable. Honey pot detection monitors the honey pot and allows network security administrators to gather valuable information and knowledge, such as what the attacker is typing, what programs he or she is running, etc., which can then be used to protect legitimate systems.

Signature-based detection examines network traffic for preconfigured and predetermined attack patterns, which are known as signatures. Many attacks today have distinct signatures. Once a known signature is identified by the system, an alarm will be triggered or the traffic will be blocked. Because attackers are constantly changing and updating methods of attack to avoid detection, it is important to remember that a signature-based system must have its signature database constantly updated to ensure that it can mitigate emerging threats. Signature-based detection is the primary method used by IDS and IPS devices.

IDS and IPS Signatures

As stated in the previous section, signature-based detection is the primary method used by IDS and IPS devices. Signatures are at the heart of signature-based detection solutions. A good set of signatures should be flexible enough to detect known attacks and, in some cases, newer attacks that are being developed by attackers. Signatures can be grouped into two general classes: attack and informational.

Attack signatures are used to indicate that an attack is occurring against your network and other network resources, such as hosts and servers, for example. An informational signature doesn’t necessarily indicate an attack. Instead, informational signatures are typically used to gather information about traffic flowing through the network, as well as to validate certain kinds of attacks. Each of these signature classes is then further divided into other subclasses.

Signature Implementations

The design of signatures involves two main components: implementations and structures. Additionally, there are two implementation methods for signatures, which are context and content. With a context implementation, a signature looks only at the packet header of captured packets, but it does not look at the actual contents (payload) within the packet. The information that these signatures look for in the packet header may include source and destination IP addresses, TCP and UDP port numbers, IP fragment parameters, and IP, TCP, and UDP checksum information, for example.

With a content signature implementation, the signature looks at the actual contents of the packet (i.e. the payload) for information that could indicate an attack. For example, a content signature may inspect an FTP session and look for commands executed during the session. 

Signature Structures

The structure of the signature is used to determine how many packets the IDS or IPS device has to examine for a particular signature when looking for attacks. There are two basic signature structures: atomic and stateful.

Atomic signatures represent the simplest signature type. An atomic structure looks at a single packet for a match on a signature. An example would be looking for the SYN flag in a TCP packet header. Because atomic signatures trigger on a single event, the IDS or IPS does not have to maintain connection state information or maintain a state table. The entire inspection can be accomplished in an operation that does not require any knowledge of past or future activities.

Because of their basic nature, atomic structures have the following advantages:

• The first advantage is that an atomic structure uses minimal system resources, such as CPU and memory, on the IDS or IPS system on which they are being used. This allows network security administrators to implement a significant amount of these signatures.

• The second advantage to an atomic structure is that the signatures are easy for network security administrators to understand because they only search for a specific event, e.g. the presence of a particular TCP flag in a TCP-based connection.

• The third advantage offered by atomic structures is that due to their simple nature, network security administrators can perform traffic analysis very quickly and efficiently.

Despite their advantages, atomic structures also have some disadvantages, which are as follows:

• When using atomic structures, network security administrators have to know all the atomic events that they want to look for, and then, for each of these events, they have to create the appropriate signature. As the number of atomic signatures increases, just managing the different signatures significantly increases overhead and can become overwhelming.

• The second disadvantage is that these signatures can be applied only to situations in which the context of the event is not important. For example, a signature can be configured to trigger an alert action whenever any traffic being analyzed contains the .exe extension to prevent executable files from being downloaded by users. However, because atomic structures do not contain state information, an alarm is triggered each time the .exe string is seen, even though an actual session may not have been established.

Unlike atomic signatures, stateful signatures are triggered on a sequence of specific events that requires the system to maintain connection state information. The length of time that the signatures must maintain state, or the maximum amount of time over which an attack signature can successfully be detected, is known as the event horizon.

Because of the fact that stateful structures require a specific event to be detected in a known context, their implementation increases the likelihood that the activity represents legitimate attack traffic. This, in turn, reduces the number of false positives that may be generated by stateful signatures. However, despite this significant advantage over atomic structures, stateful structures also have the following disadvantages that must be taken into consideration:

• The main disadvantage to stateful signatures is that maintaining connection state information requires significant amounts of memory and physical disk space, which increases the monetary cost of these devices.

• The second disadvantage is that stateful structures consume a large amount of resources, e.g. memory and CPU, which can lead to a slow response time, dropped packets, missed signatures, and so on, and this adversely impacts the effectiveness of the system.

Signature Triggers

The most important part of any signature is the mechanism that causes the signature to trigger. Different vendors use different triggering mechanisms. The triggering mechanism refers to the condition, or conditions, that cause the system to generate a signature action, which can be an alert, notification, or both, for example.

Some devices use basic triggering mechanisms, while others use advanced, complex triggering mechanisms. Regardless of the simplicity or complexity of the mechanism used, most IDS and IPS implementations incorporate various triggering mechanisms when developing signatures, the most common being pattern-based, anomaly-based, and behavior-based.

Pattern-based signature triggering is the most common and one of the simplest triggering mechanisms because its main function is to identify a specific pattern. Once the pattern is matched, the signature is triggered. This pattern may be as simple as text or a binary string, or it may be relatively advanced and use regular expressions or de-obfuscation techniques.

A regular expression is simply a pattern-matching language that enables you to define a flexible search pattern. For example, in Cisco IOS software, the regular expression (also referred to as regexp) ^123 matches all patterns that begin with the numbers 123. This would match 1234 and 123123, but it would not match 01234, for example. Going into detail on regular expressions is beyond the scope of the IINS course requirements.

Obfuscation is the concealment of the intended meaning in communication, which results in the communication being confusing, intentionally ambiguous, and more difficult to interpret. This technique is used by attackers to attempt to confuse the IDS or IPS system so that it will miss valid strings that have been obfuscated. This technique is commonly used when requesting web pages from a server. Going into detail on the various obfuscation techniques used by attackers is beyond the scope of the IINS course requirements.

In anomaly-based detection, which is also known as profile-based detection, signatures are triggered when certain activities deviate from what is considered normal. As described earlier in this chapter, anomaly-based detection may be implemented using either statistical or non-statistical anomaly-based detection methods.

Behavior-based signatures are triggered when any behaviors that do not occur normally are observed, as this may be an indication of a network attack or other malicious activity. Before they can be used, behavior-based signatures must be manually defined. However, the drawback to this is that it takes much research to determine behaviors that do not occur normally and that can accurately indicate suspicious behavior.

An example of behavior-based signatures would be a signature that is triggered when a user attempts to download and install an executable file or program from an external web site when company policy clearly forbids such activities. 

Signature Actions

Up to this point, we have learned about the different types of signatures and all of their characteristics. In this section on IDS and IPS signatures, we are going to learn about the different actions that the device can take once a signature has been triggered. When a signature observes the activity that it is configured to detect, the signature triggers one or more actions, which can include, but are not limited to, the following:

• Generating an alert
• Dropping or preventing the activity
• Logging the activity
• Resetting a TCP connection
• Blocking future activity
• Allowing the activity

IDS and IPS devices can generate atomic alerts, summary alerts, or a combination of both. An

atomic alert is generated every time a signature triggers. These are the most common alerts generated by IDS and IPS devices. A summary alert is a single alert that indicates multiple occurrences of the same signature from the same source address and/or port. This reduces the overall number of individual alerts that are generated for the same incident.

For example, if an IDS or IPS system signature is triggered by 100 occurrences of the same event, say from a host with the IP address 172.16.1.254, it would send out 100 alerts for each occurrence. Naturally, this would be redundant information, as it indicated the same event. Therefore, using summary alerts, network security administrators could configure a signature summary interval so that when the length of time specified by the summary interval has elapsed, a summary alarm is sent, indicating the number of alarms that occurred during the time interval specified by the summary interval parameter. This would reduce the overall number of alarms but still capture the same information.

One of the greatest advantages of an IPS device over an IDS device is the capability to directly drop packets when implemented inline. This action enables the device to stop an attack before it has the chance to perform malicious activity. Besides dropping individual packets, the drop action can be expanded to drop all packets for a specific session or even all packets from a specific host for a certain amount of time. This flexibility allows the IPS to conserve resources, such as processor and memory, by dropping traffic without having to analyze each packet separately.

Logging capabilities allow the IDS and IPS systems to log the actions or packets that are seen so that network security administrators can analyze this information in greater detail. The log information is typically stored locally on the device itself, in a specific file. Because the signature also generates an alert, administrators can observe the alert on the management console and then retrieve the log data from the IPS device, so as to analyze the activity that the attacker performed on the network after triggering the initial alarm.

The majority of IPS devices that are implemented inline have the capability to terminate TCP connections that are performing unwanted or unauthorized operations by generating a packet for the connection with the TCP RST flag set. This resets the session for the attacker as well as for the victim (target) systems.

As we learned earlier in this chapter, IDS and IPS devices operating in promiscuous mode can be configured automatically to configure another infrastructure device, such as a firewall, when they detect an attack in the traffic they are monitoring. This ACL stops traffic from an attacking system without requiring the IDS or IPS to consume resources analyzing the traffic. After a configured period of time, the IDS or IPS device can be configured to remove the ACL or leave it in place, preventing similar future activity from occurring.

Finally, allowing the activity refers to the process of defining exceptions to configured signatures. Configuring exceptions enables you to take a more restrictive approach to security because you first deny everything and then allow only the activities that are needed. For example, an exception might be that no other department except for the IT department is allowed to download .exe files from external web servers. If someone who is not in the IT department attempts to download an .exe file, an alert is generated.

IDS and IPS System Types

Now that we have a solid understanding of IDS and IPS systems and their capabilities, operation, and characteristics, we are going to be learning about the different types of IDS and IPS solutions available. Because the IINS exam requirements explicitly state host-based and network-based IDS and IPS types, this section provides a description of both of these, as well as other common IDS and IPS types. 

Intrusion Detection System Types

There are several different kinds of IDS types, and the following section provides a brief overview of each of these, including:

• Network-based Intrusion Detection System (NIDS)
• Protocol-based Intrusion Detection System (PIDS)
• Application Protocol-based Intrusion Detection System (APIDS)
• Host-based Intrusion Detection System (HIDS)
• Hybrid Intrusion Detection System

A Network-based Intrusion Detection System (NIDS) is an independent, standalone platform used to identify intrusions by examining network traffic and monitoring multiple hosts. A NIDS is typically connected to a network switch and relies on Switched Port Analyzer (SPAN) or other similar port mirroring implementation to view network traffic. Cisco Secure IDS 4200 Series Sensors available from Cisco are an example of a Network-based IDS solution. However, the Cisco product range of Network-based IDS solutions is End-of-Life and is no longer sold and, in some cases, no longer supported by Cisco. Instead, Cisco recommends that their Network-based IPS solutions be implemented. NIPS will be described later in this chapter.

A Protocol-based Intrusion Detection System (PIDS) is a system or agent that monitors and analyzes the communication between a server and the devices connected to it.  For example, a PIDS would monitor the HTTP protocol on a web server. Going into detail on PIDS is beyond the scope of the IINS course requirements, and it will not be described any further in this guide.

An Application Protocol-based Intrusion Detection System (APIDS) is somewhat similar to a PIDS; however, instead of monitoring a specific protocol, an APIDS is used to monitor and analyze the communication for specific application protocols. For example, an APIDS could be used to monitor the SQL protocol on an Oracle database server. Going into detail on APIDS is beyond the scope of the IINS course requirements, and it will not be described any further in this guide.

A Host-based Intrusion Detection System (HIDS) is an agent that resides on a network host, e.g. a PC, laptop, or server, which is used to identify intrusions by analyzing system calls, application logs, file-system modifications, and other host activities and state. An example of a HIDS is the Cisco Security Agent (CSA). However, it should be noted that while the CSA provides HIDS capabilities, the latest versions of this software are truly a Host-based IPS solution and are marketed as such. HIPS will be described later in this chapter.

A Hybrid Intrusion Detection System is simply an IDS solution that combines two or more of the other IDS characteristics. For example, a Hybrid IDS could provide both HIDS and APIDS capabilities. Going into detail on Hybrid Intrusion Detection Systems is beyond the scope of the IINS course requirements, and it will not be described any further in this guide.

Intrusion Prevention System Types

As is the case with IDS systems, there are several different types of IPS solutions, as follows:

• Host-based Intrusion Prevention System (HIPS)
• Network-based Intrusion Prevention System (NIPS)
• Content-based Intrusion Prevention System (CIPS)
• Rate-based Intrusion Prevention System (RBIPS)

A Host-based Intrusion Prevention System (HIPS) is an IPS system that resides on a network host, such as a computer, laptop, or server. A Host-based Intrusion Prevention System is typically implemented on these endpoints in conjunction with other security software, such as anti-virus software. The primary function of the HIPS is to prevent malicious code from modifying the system or other software on the system. This is performed by an integrated firewall, system-level action control, sandboxing, or whitelisting.

Sandboxing is simply a security mechanism for separating running programs that is used to execute untested code or untrusted programs. The sandbox on the host provides a tightly controlled set of resources for guest programs to run in, such as free space on disk and memory.

A whitelist is the opposite of a blacklist. In other words, a whitelist is a list or compilation that identifies entities or programs that are permitted, recognized, or provided a particular privilege, service, mobility, access, or recognition. For example, on a Windows-based machine, Microsoft Outlook, which is used for email communication, may be whitelisted, which means that the HIPS on the host will allow this program to execute if initiated by the user. An example of a Host-based IPS is the Cisco Security Agent. HIPS is a core topic and a requirement of the IINS course, and CSA will be described in greater detail later in this chapter.

A Network-based Intrusion Prevention System (NIPS) is an IPS system-dedicated hardware or software platform designed to analyze, detect, and report on network security events. NIPS are designed to inspect traffic and, based on their configuration or security policy, they can drop malicious traffic. Unlike Host-based IPS systems, a Network-based IPS system can detect potential security events all over the network and react to each of them, whereas a HIPS can detect and react to potential security events only on the host on which it resides. An example of a Network-based IPS is the Cisco IPS 4200 Sensors. NIPS is a core topic and a requirement of the IINS course, and Cisco NIPS solutions will be described in greater detail later in this chapter.

A Content-based Intrusion Prevention System (CIPS) is used to inspect the content (i.e. payload or data) of network packets for signatures, which allows it to detect and prevent known types of attacks, such as worm infections and hacks. While there are dedicated CIPS systems, it is important to know that this functionality is typically integrated into other IPS systems. Going into detail on CIPS is beyond the scope of the IINS course requirements, and it will not be described any further in this guide.

A Rate-based Intrusion Prevention System (RBIPS) is primarily used to prevent DoS and DDoS attacks. A Rate-based IPS is similar to a statistical anomaly-based IDS in that it monitors and baselines normal network behavior. Through real-time traffic monitoring and comparison with stored statistics, RBIPS can identify abnormal rates for certain types of traffic. Once an attack is detected, various prevention techniques may be used such as rate-limiting, which is basically slowing down the speed of specific attack-related traffic types, source, or connection tracking, and source-address, blacklisting, or whitelisting.

Although going into detail on RBIPS is technically beyond the scope of the IINS course requirements, the Cisco Traffic Anomaly Detector and the Cisco Guard DDoS Mitigation, which provide RBIPS services as part of the Cisco Anomaly Detection and Mitigation solution, are described later in this chapter, as a related topic.

Cisco Network Intrusion Prevention Systems

The Cisco Network-based Intrusion Prevention solutions are an integral part of the Cisco Self-Defending Network strategy that provides network intelligence to identify and prevent malicious traffic including, but not limited to, network viruses, worms, spyware, adware, and application abuse. The Cisco Network-based Intrusion Prevention solution protects the network from policy violations, vulnerability exploitations, and anomalous activity through detailed inspection of traffic at Layers 2 though 7, throughout the entire network. The Network-based Intrusion Prevention solutions available from Cisco are as follows:

• The Cisco IPS 4200 Series Appliance Sensors
• The Cisco IDS Services Module 2 (IDSM-2)
• The Cisco Advanced Inspection and Protection Security Services Module
• The Cisco IPS Advanced Integration Module (IPS-AIM)
• The Cisco Internetwork Operating System (IOS) IPS

The Cisco IPS 4200 Series Appliance Sensors

The Cisco IPS 4200 Series Appliance Sensors offer a broad range of solutions, providing easy integration into enterprise and service provider networks. The following section describes the five purpose-built appliances that encompass the IPS 4200 Series Appliance Sensor offerings manufactured by Cisco:

• The IPS 4215 Appliance Sensor can deliver up to 80Mbps of transactional performance and is suitable for multiple T1/E1 and T3/E3 environments. The 4215 supports up to five monitoring interfaces per appliance.

• The IPS 4240 Appliance Sensor can deliver up to 250Mbps of transactional performance, providing protection in switched environments on multiple T3/E3 subnets. The 4240 can support multiple 10/100/1000 interfaces and can be deployed on partially utilized GigabitEthernet links or fully saturated FastEthernet links.

• The IPS 4255 Appliance Sensor supports transactional performance at 600Mbps and can be used to protect partially utilized GigabitEthernet links and traffic traversing switches that are being used to aggregate traffic from numerous subnets.

• The IPS 4260 Appliance Sensor can be deployed to deliver transactional performance of up to 1Gbps of dedicated intrusion prevention protection. The 4260 can also be used to protect both GigabitEthernet subnets and aggregated traffic that is traversing switches from multiple subnets.

• The IPS 4270 Appliance Sensor supports unparalleled performance and can be deployed to deliver transactional performance of up to 2Gbps of dedicated intrusion prevention protection. It can also be used to protect both GigabitEthernet subnets and aggregated traffic that is traversing switches from multiple subnets.

The Cisco IDS Services Module 2 (IDSM-2)

The Cisco IDS Services Module 2 (IDSM-2) is a high-speed, high-performance integrated IPS module that can be installed into Cisco Catalyst 6500 switches or Cisco 7600 Series routers. The IDSM-2 can deliver up to 600Mbps of intrusion prevention protection in passive mode and up to 500Mbps in inline mode.

Amongst many other features, the IDSM-2 offers Cisco IPS Device Manager (IDM), which is a GUI-based, Java-enabled, built-in web-based tool for sensor configuration and management. IDM can be accessed through any web browser and is enabled to use Secure Sockets Layer (SSL) by default. Additionally, IDSM-2 provides event monitoring for up to five IPS sensors through the Cisco IPS Event Viewer (IEV).

The Cisco Advanced Inspection and Protection Security Services Module

The Cisco Advanced Inspection and Protection Security Services Module, or AIP-SSM, as it is more commonly referred to as, is designed for the Cisco ASA 5500 Series Security Appliances. The AIP-SSM provides a full-featured intrusion prevention solution to protect against malicious traffic, including network viruses and worms.

Using Cisco IPS sensor software, the AIP-SSM combines inline prevention services that offer all the major functions available in the traditional Cisco IPS sensor solutions. Cisco ASA 5500 Series Security Appliances coupled with AIP-SSM provide the best in-class firewall and intrusion prevention capabilities in a single, easy-to-deploy platform.

The Cisco IPS Advanced Integration Module (IPS-AIM)

The Cisco IPS Advanced Integration Module (IPS-AIM) is part of the Cisco integrated IDS/IPS sensor family portfolio and is an integral part of the Cisco Self-Defending Network. The IPS-AIM can be used with Cisco Integrated Services Routers (ISRs), such as the Cisco 2800 and 3800 Series platforms.

The Cisco IPS-AIM has dedicated processors and Dynamic Random Access Memory (DRAM). Although the IPS-AIM does not provide dedicated command and control (management) interfaces, administrators can use the internal GigabitEthernet port for in-band management.

The Cisco Internetwork Operating System (IOS) IPS

The Cisco Internetwork Operating System (IOS) IPS feature set provides an integrated inline deep-packet inspection solution with router software architecture. This solution is available on all Cisco Integrated Services Routers (ISRs) and offers an integrated security and policy enforcement solution. The Cisco IOS IPS will be described in detail later in this chapter.

NOTE:  It is important to know that prior to the Cisco IPS 4200 Series Appliance Sensors, Cisco offered the Cisco IDS 4200 Series Appliance Sensors that were truly only IDS-capable devices. The appliances offered included the Cisco IDS 4250 Sensor, the IDS 4235 Sensor, the IDS 4230 Sensor, the IDS 4220 Sensor, and the IDS 4210 Sensor. These products are no longer being sold and might not be supported. Therefore, we will not be describing them or their capabilities and functionality in any detail in this guide.

 

Deploying an IPS into the traffic stream (i.e. inline) introduces a new device into the data path that has the potential to fail and prevent traffic from flowing. Fortunately, with Cisco IPS solutions, a failed IPS can be prevented from interrupting the flow of network traffic via a fail-open mechanism, a failover mechanism, or by performing load balancing. While going into detail on these mechanisms is beyond the scope of the IINS course requirements, it is important to have a basic understanding of what they mean.

Cisco NIPS solutions can use a hardware-based or software-based fail-open mechanism to allow traffic to go through uninterrupted in the event of any hardware (e.g. power failure) or software (e.g. signature failure) issues. Cisco IOS software also allows administrators to implement various failover mechanisms, such as HSRP and Spanning-Tree Protocol, in order to provide NIPS redundancy. HSRP stands for Hot Standby Router Protocol; however, it is beyond the scope of the IINS course requirements. And, finally, network security administrators can use load-balancing techniques in conjunction with Cisco NIPS solutions to prevent a single NIPS failure from preventing the flow of traffic.

Sensor Communication Protocols

As we learned earlier in this chapter, the sensor OS applications communicate with each other using IDAPI. However, it is also important to understand the other different communication protocols used by the Cisco IPS sensor software, and they are as follows:

• RDEP2
• IDIOM
• IDCONF
• SDEE
• CIDEE

RDEP stands for Remote Data Exchange Protocol and is a Cisco proprietary Application-level protocol used by Cisco IDS Sensors that uses both a subset of the HTTP/1.1 protocol and a client request/server response model. RDEP2 is used primarily for external communications; specifically, it is used to exchange IDS events, IP logs, configurations, and control messages between IPS clients and IPS servers. RDEP2 is no longer used and has been replaced by Security Device Event Exchange (SDEE) in the current Cisco IPS Sensor software versions. SDEE will be described later in this section.

IDIOM, which stands for Intrusion Detection Interchange and Operations Messages, is a data format standard that defines the event messages that are reported by the IPS, as well as the operational messages that are used to configure control intrusion-detection systems. These messages consist of XML (Extensible Markup Language) documents that conform to the IDIOM XML schema. IDIOM supports two types of interactions: the event and control transactions. Event transactions are used to exchange IPS events, such as IPS alerts. Control transactions utilize four types of IDIOM messages: request, response, configuration, and error messages. Event and control transactions that are communicated between application instances within a host are known as local events or local control transactions, or, collectively, as local IDIOM messages. Events and control transactions that are communicated between different hosts using the RDEP2 protocol are known as remote events and remote control transactions, or, collectively, as remote IDIOM messages. It is important to know that in the latest sensor software OS, IDIOM has for the most part been superseded by IDCONF, SDEE, and CIDEE, all of which are described later in this section.

IDCONF, which stands for Intrusion Detection Configuration, specifies the XML schema including IPS control transactions. Cisco IPS sensor software manages its configuration by using XML. However, it is important to know that the IDCONF schema does not specify the contents of the configuration documents but, rather, the framework from which the configuration documents are developed. IDCONF messages are exchanged over RDEP2 and are wrapped inside IDIOM request and response messages.

SDEE, which stands for Security Device Event Exchange, is a new standard proposed by the International Computer Security Association (ICSA) that specifies the format of messages and protocol used to communicate events generated by security devices. SDEE is an enhancement to the current version of RDEP2 that adds extensibility features that are needed for communicating events generated by various types of security devices. Systems that use SDEE to communicate events to clients are referred to as SDEE providers. SDEE specifies that events can be transported using the HTTP or HTTP over SSL and TLS protocols. When HTTP or HTTPS is used, SDEE providers act as HTTP servers, while SDEE clients are the initiators of HTTP requests.

IPS includes Web Server, which processes HTTP or HTTPS requests. Web Server uses run-time loadable servlets to process the different types of HTTP requests. Each servlet handles HTTP requests that are directed to the URL associated with the servlet. The SDEE server is implemented as a web server servlet and processes only authorized requests. A request is authorized if it originates from a web server to authenticate the identity and determine the privilege level of the client.

CIDEE, which stands for Cisco Intrusion Detection Event Exchange, specifies the extensions to SDEE that are used by the Cisco IPS. The CIDEE standard specifies all possible extensions that are supported by IPS systems. However, it is mandatory that any extension that is designated as being required be supported by all systems. CIDEE supports the following events:

• evError (error event) – this is generated by the CIDEE provider when the provider detects an error or warning condition. The evError event contains error code and textual description of the error.

• evStatus (status message event) – this is generated by CIDEE providers to indicate that something of potential interest occurred on the host. Different types of status messages can be reported in the status event – one message per event. Each type of status message contains a set of data elements that are specific to the type of occurrence that the status message is describing. The information in many of the status messages is useful for audit purposes. Errors and warnings are not considered status information and are reported using evError rather than evStatus.

• evShunRqst (block request event) – this is generated to indicate that a block action is to be initiated by the service that handles network blocking.

NOTE: SDEE is the default protocol used by Cisco IPS Sensor software, as well as by the Cisco IOS IPS feature set used on Cisco IOS routers.

Cisco Host Intrusion Prevention Systems

This section provides details on the Cisco Host-based Intrusion Prevention solution that uses Cisco Security Agent (CSA). This section describes core concepts pertaining to CSA, such as CSA architecture, components, policies, rules, and rule modules, as well as basic details on managing and deploying CSA using CSA Management Center (CSA MC).

CSA provides proactive HIDS and HIPS solutions for endpoint systems, such as desktops, laptops, and servers, for known and unknown (day-zero) threats. CSA does not rely on a signature-based architecture. Instead, CSA uses a flexible policy-based and behavior-based architecture, which allows it to offer defense against targeted attacks, viruses, worms, spyware, adware, rookits, and day-zero attacks that have not been discovered, as well as new exploits and variants taking advantage of known and unknown vulnerabilities.

CSA also has the capability to provide policy compliance controls, offering protection to sensitive information on the system. For example, CSA can use granular controls that restrict the removal of USB keys, prevent copy-and-paste operations between applications, and restrict peer-to-peer applications such as instant messaging, amongst many other things. The following list highlights the characteristics and benefits offered by CSA:

• Endpoint system protection
• Host-based intrusion prevention
• Policy-based and behavior-based architecture
• Personal firewall protection
• Day-zero attack protection
• Regulatory policy compliance enforcement
• Acceptable corporate use policy compliance
• Preventative protection against targeted attacks
• Stability and protection of the underlying operating system
• File and directory protection
• Host application visibility
• Application control
• Correlation of system calls and application functions 

CSA Architecture

The architecture of CSA software is unique in that the host agent resides between the applications and the OS kernel, as illustrated in the following diagram:

The placement of the CSA agent, as illustrated in the diagram above, provides application visibility with minimal impact to the stability and performance of the underlying OS. CSA works at the kernel level by controlling file system and network actions, as well as other components of the OS.

CSA architecture intercepts all OS and application-related calls when access is requested for certain resources, such as file access, device access, registry access, and application execution calls. CSA is also capable of intercepting dynamic runtime resources requested, such as memory pages, services, and shared library modules. This means that when an application attempts an operation or requests access, CSA checks the operation against the security policy, which allows it to make real-time decisions on whether to allow or deny the operation.

CSA Components

CSA has three basic components: CSA endpoints, the CSA Management Server, and the CSA Management Console (CSA MC).

CSA endpoints are computing desktops, servers, laptops, and point-of-sale (POS) terminals. The CSA is responsible for enforcing security policies received from the management server, sending events, and interacting with the user.

The CSA management server is the core component in the CSA deployment. The management server is a repository of the configuration database, and is responsible for deploying the security policies to the endpoints, receiving and storing all events, sending alerts to the network security administrator, and deploying software to the endpoints.

The CSA MC is an administrative web-based user interface and policy configuration tool that provides network security administrators with event views. All communication between the CSA MC and the management server are secured using SSL. The CSA MC binds with the server database, which holds all the policies. All CSA endpoint agents then register with the CSA MC to receive the policy. The CSA MC validates the host and deploys the respective policy pertaining to that host or group of hosts.

Managing CSA Groups

Groups are a logical collection of hosts. The grouping system used in CSA deployment provides several advantages, which include:

• Policy consistency through the application of the same set of policies on multiple hosts across the network;

• The capability to apply alternative mechanisms and set parameters based on group configurations; and

• A test mode feature that can be used to validate policies on groups of hosts before they are actively enforced on production systems.

The CSA group feature simplifies configuration management by associating policy controls and other parameters into group settings. When a user is associated with a group, the host will inherit all of the parameters configured within the group. This ensures consistency across the entire CSA deployment because all hosts have uniform settings when associated to the same group. This scalable approach of grouping hosts allows network security administrators to perform large-scale CSA deployments with great ease and reduced complexity.

Another advantage of having groups is that it makes the creation of agent kits, which are the CSA installation executable files that are installed onto the endpoint system, much easier because the group parameter is the only element required to build agent kits.

CSA MC has several built-in, predefined groups that can be used according to the requirements. Additionally, network security administrators can create custom groups using the CSA MC. Groups in CSA MC are divided into the following three categories:

• Mandatory groups
• Predefined groups
• Custom groups

NOTE: You are not required to demonstrate detailed knowledge on the CSA MC groups.

By default, CSA MC provides three auto-enrollment mandatory groups as follows:

• All Windows
• All Solaris
• All Linux

One of these mandatory groups will be associated with each endpoint host according to its OS architecture upon registration. For example, when Windows-based hosts are registered to the CSA MC, they are automatically enrolled into the All Windows group, in addition to any other custom groups that are mapped to the particular host. Given this, it is important to know that hosts cannot be removed from the mandatory groups, because mandatory groups are used to enforce some of the compulsory security policies that are utilized to prevent critical services from being inadvertently disturbed. For example, a mandatory policy can dictate that a user cannot disable the DNS or DHCP service.

Because hosts can belong to one or more groups, which means that they will inherit multiple policies from these groups, in the event of a rule conflict, the combined rule set will follow the rule precedence process when deciding which rules to override.

CSA Policies, Rule Modules, and Rules

In CSA terminology, a policy is a collection of rule modules, and a rule module is a collection of multiple rules. The rule module acts as the container for rules, whereas the policy serves as the unit of attachment to groups. Rules provide control access to system or network resources.

Multiple rule policies can be attached to a single policy, and multiple policies can be attached to a single group. Different types of rules in a rule module can coexist, and different types of rules within a single policy can also coexist. CSA MC allows for the creation of several types of policies in addition to the default, built-in policies (e.g. the Windows, Solaris, and Linux policies). It is imperative to remember that policies, rule modules, and rules are the enforcement tools that are used in the CSA architecture.

NOTE: You are not required to perform any configuration tasks using CSA in the IINS exam. 

CSA Access Control Process

This next section on CSA outlines, in order, the access process that the CSA performs:

1. Identify the resource being accessed. This may be a network resource, a memory function call, an application execution, files access, a system configuration, or any other OS kernel system calls, for example.

2. Gather data about the operation. For example, if a file access operation is requested, CSA will identify and gather the process name, file path, file name, and file operation, such as read, write, or erase.

3. Determine the state of the system. This includes the current IP address, MAC address, DNS suffix, VPN client status (if applicable), NAC (if application), and virus or worm detection.

4. Consult the security policy by analyzing rules and consulting the local policies. For example, anomaly-based, atomic, pattern-based, or behavioral-based rules, or another type of access control matrix.

5. Take action as per policy defined in the rules. For example, depending on the rule configuration, CSA may permit, deny, query, change the internal state, or monitor.

CSA Interceptor and Correlation

At the core of CSA architecture is the interceptor and correlation mechanism. The interceptor is used to intercept key actions that are attempted on the system and to check the action in question against the rule correlation engine to determine whether a rule set allows or denies it. Based on the information the interceptors receive, either they allow the action to take place or they stop it. Interceptors intercept application-related and system-related functions to enforce policy-based and behavior-based rules. There are five different interceptors used by CSA, depending on the platform on which it is installed, i.e. Windows or UNIX. These interceptors are as follows:

• The network application interceptor is applicable to both Windows- and UNIX-based platforms. It is used to regulate and control which applications are allowed to communicate with the network.

• The network traffic interceptor provides system-hardening features such as SYN flood protection and port scan detection. This interceptor is used on both Windows- and UNIX-based platforms.

• The file interceptor, which is used in UNIX- and Windows-based platforms, controls which applications can read and/or write to specified system files and directories.

• The registry interceptor controls system behavior, preventing applications from writing to particular registry keys. This interceptor is used only for Windows-based platforms.

• The system call interceptor is the UNIX-equivalent of the Windows platform registry interceptor. This interceptor protects the UNIX kernel in the same manner the registry interceptor protects the Windows registry.

As the interceptors are allowing or denying actions, they produce an event each time a rule set is triggered by a system action. These events are stored in the event correlation engine, which forwards them on to the local event manager and global event manager.

NOTE: Before we move on to correlation, it is important to know that some documentation refers to only four interceptors, which are the file system interceptor, the network interceptor, the configuration interceptor, and the execution space interceptor.

The file system interceptor is the same as the file interceptor, and the network interceptor is the same as the network traffic interceptor. The configuration interceptor is responsible for intercepting read/write requests to the Windows registry or rc files in UNIX-based systems. This single interceptor performs the same actions as the current system called interceptor and registry interceptor for both UNIX- and Windows-based platforms, respectively. And, finally, the execution space interceptor maintains the integrity of the dynamic runtime environment of individual applications. The execution space interceptor also blocks attempts by any application to inject code into other applications.

While going into the details pertaining to the CSA interceptor and correlation mechanism is beyond the scope of requirements of the IINS course, the following diagram illustrates the core architecture of the CSA correlation system and shows how it intercepts various system and application-related functions or calls:Correlation is the enforcement engine for CSA, where relationships between events are established to determine whether the behavior is acceptable and whether the event should be allowed or denied. CSA correlation maintains the state of all system calls and application activities on the host system. In addition to this, CSA correlation enhances interoperability between applications, such as protecting the usage of the command shell from unauthorized access and ensuring that legitimate applications can invoke this without interruption. CSA can correlate numerous activities to increase the accuracy of its rules and policies, as well as to enhance its capability to make accurate decisions about what is dangerous.

Cisco IOS IPS Overview and Implementation

Deploying the Cisco IOS IPS inline provides network security administrators with unique filtering capabilities that enable them to stop the attack at the point of origin. The IOS IPS solution can be deployed at various points within the network and can be ideally situated at the network edge to protect the network from malicious and offending traffic entering into the network. As of the time of the writing of this section, Cisco is the only vendor that delivers this integrated functionality on a router.

Cisco IOS IPS Basics

The Cisco IOS IPS feature is a suite of intrusion prevention solutions provisioning a single point of protection at the network perimeter. The IOS IPS offers unparalleled intrusion security, reliability, scalability, and multilevel performance. Some of the key features of the IOS IPS are as follows:

• It protects the network from viruses, worms, and a large variety of threats and exploits;
• It eliminates the need for a standalone IPS device;
• It provides integrated inline deep-packet inspection;
• It complements the Cisco IOS Firewall and VPN solutions for superior threat protection;
• It supports about 2000 attack signatures;
• It uses Cisco IOS routing capabilities to deliver integrated functionality;
• It enables distributed network-wide threat mitigation; and
• It sends a Syslog message or an alarm in SDEE format when a threat is detected.

When a Cisco IOS router will be acting as an IPS device, it needs to have a place to store the signature files, referred to as Signature Definition Files (SDFs), that it will use to identify malicious traffic. A Signature Definition File is a file, usually in XML format, that contains signature definitions that can be used to load signatures on the Cisco IOS router.  In most cases, the SDF is located in the router Flash Memory; however, Cisco IOS routers also have the capability to reference multiple SDFs located on network servers (e.g. TFTP servers) for increased signature coverage.

In order to ensure that the Signature Definition Files are up-to-date, which allows them to be effective against ever-changing and evolving threats, network security administrators need to manually update the SDFs periodically by downloading the latest files from the Cisco website. SDFs have a .sdf file extension name, for example, 128MB.sdf, 256MB.sdf, and attack-drop.sdf.

However, it is important to know and remember that in Cisco IOS 12.4(11)T and later, Cisco IOS IPS uses the Cisco IPS 5.x signature format, which is a version-based signature-definition XML format also used by other Cisco appliance-based IPS products. Support for signatures and SDFs in Cisco IPS Version 4.x are discontinued in this and further Cisco IOS T-Train Software releases. Additionally, it is important to remember that IPS 4.x uses a version format of 2.xxx.xxx while IPS 5.x uses a version format of 3.xxx.xxx. 

Anomaly Detection and Mitigation

DoS and DDoS attacks have become more sophisticated and prevalent over the years. In today’s rapidly evolving networks, attackers are more often than not one step ahead of network security administrators, and effective mitigation of DoS and DDoS attacks has become a pressing problem. Fortunately, network security administrators can leverage several proactive detection and prevention mechanisms to protect networks from these malicious techniques.

Anomaly Detection and Mitigation Systems

Anomaly intrusion detection and prevention is an enhanced solution designed to combat DoS and DDoS attacks. Anomaly detection solutions provide intelligence-based intrusion prevention by monitoring system activity and categorizing the traffic as either normal or anomalous (abnormal). The classification is based on heuristics or rules rather than traditional patterns or signatures used by other Intrusion Prevention Systems.

Anomaly detection systems are initially in learning mode so that they can characterize normal activity and establish a baseline for normal traffic and network conditions. Anomaly detection involves defining or learning normal activity and looking for deviations from the various baseline profiles that have been created. This can be performed using the following methods:

• Protocol anomaly detection
• Behavioral anomaly detection
• Network anomaly detection

Protocol anomaly detection involves looking for deviations from a standard protocol. This method is useful for identifying deviations from normal protocol behavior. For example, this method can be used to detect HTTP embedding methods used by attackers.

Behavioral anomaly detection involves learning normal user behavior and detecting the relational traffic pattern activities of individual hosts or a group of hosts. If a change occurs, an alarm is generated. This method is most useful in a very tightly controlled environment. For example, if company policy mandates that no executable files are to be downloaded, and this has been clearly communicated to all employees, but someone downloaded or attempted to download an executable file, an alarm would be triggered.

Network anomaly detection involves watching or learning the normal network traffic levels, for example, using a time-based classification of normal traffic activity. If deviation from normal traffic activity is detected, an alarm is generated. However, it is important to understand that network conditions change constantly, and so this method can result in many false positives. 

Anomaly Detection Operation and Characteristics

Once baselines have been established, anomaly detection compares all traffic with the baseline profile, and any deviation from the profile is considered a potential attack. However, the issue is that on many occasions, legitimate traffic is integrated with the attack traffic; therefore, network security administrators must examine traffic patterns in real-time so that valid traffic can be processed and still be passed without interruption. Attack traffic can then be diverted to a mitigation device.

Anomaly detection and mitigation algorithms can detect all kinds of attacks, including day-zero attacks. This differs significantly from signature-based systems, which can only detect attacks for which a static signature has been defined. The following section lists the characteristics of anomaly detection techniques:

• They do not require the use of signatures or patterns;
• They are granular and are based on observed traffic pattern behavior;
• They can perform relational and behavioral anomaly detection;
• They detect in real-time, i.e. anything they report is actually happening at that time;
• They support dynamic filtering;
• They include sophisticated anti-spoofing techniques;
• They can detect day-zero and minute-zero attacks;
• They can highlight behaviors that do not indicate an attack but are of some interest; and
• They include a traffic diversion architecture.

Cisco Anomaly Detection and Mitigation Solution

The Cisco Anomaly Detection and Mitigation solution is used to combat complex and sophisticated DDoS attacks. This solution can be used in both enterprise and service provider networks and provides the following capabilities:

• It can detect and mitigate a wide range of DDoS attacks;
• It can classify legitimate traffic and attack traffic in real-time;
• It can block large botnet and zombie attacks;
• It can block attack traffic by using source-based dynamic filters; and
• It delivers multi-gigabit performance at line rate.

The following diagram illustrates packet flow through the defense modules that provide advanced DDoS protection using the Cisco Anomaly Detection and Mitigation solution:

The Cisco DDoS Anomaly Detection and Mitigation solution consists of two core components:

• Cisco Traffic Anomaly Detector
• Cisco Guard DDoS Mitigation

NOTE: While going into detail on both of these solutions is beyond the scope of the IINS course requirements, the following section provides a brief description of each component.

The Cisco Traffic Anomaly Detector

The Cisco Traffic Anomaly Detector devices work in conjunction with the Cisco Guard DDoS Mitigation devices. These devices identify potential DDoS attacks and divert traffic destined to the target device(s) to the Cisco Guard to screen, identify, and block (in real-time), without affecting legitimate traffic flows. When anomalous traffic is detected, the Cisco Traffic Anomaly Detector performs one or more of the following actions:

• It sends an alert to initiate a manual response from network security administrators;
• It triggers an existing management system; and/or
• It launches the Cisco Guard DDoS Mitigation device to begin mitigation.

The Cisco Traffic Anomaly Detector performs the following tasks:

• Traffic learning – the Cisco Traffic Anomaly Detector classifies and categorizes the normal zone traffic pattern by using an algorithm-based process to establish a baseline. During the learning process, the Cisco Traffic Anomaly Detector modifies the default traffic policies and policy thresholds to match the characteristics of normal traffic. The traffic policies and thresholds define the reference points that the Cisco Traffic Anomaly Detector uses to determine when the traffic is normal or when it is abnormal, which indicates an attack.

• Traffic anomaly detection – the Cisco Traffic Anomaly Detector detects anomalies in protected zone traffic based on normal traffic characteristics.

The Cisco Guard DDoS Mitigation

The Cisco Guard DDoS Mitigation is a unique Multi-Verification (MVP) architecture that subjects diverted traffic (from the Cisco Traffic Anomaly Detector) to the most advanced anomaly recognition, protocol analysis, source verification, and anti-spoofing technologies. The Cisco Guard DDoS Mitigation device can identify a broad range of DDoS attacks, such as:

• TCP-based attacks
• UDP-based attacks
• HTTP attacks
• Botnet and zombie attacks
• DNS attacks
• Voice-over-IP attacks

The Cisco Guard DDoS Mitigation performs the following tasks:

• Traffic learning – the Cisco Guard DDoS Mitigation classifies and categorizes the normal zone traffic pattern by using an algorithm-based process to establish a baseline. During the learning process, the Cisco Guard DDoS Mitigation modifies default traffic policies and policy thresholds to match the characteristics of normal traffic. The traffic policies and thresholds define the reference points that the Cisco Guard DDoS Mitigation uses to determine when the traffic is normal or when it is abnormal, which indicates an attack.

• Traffic protection – the Cisco Guard DDoS Mitigation distinguishes between legitimate and malicious traffic and filters the malicious traffic so that only the legitimate traffic is allowed to pass on to the protected zone.

• Traffic diversion – the Cisco Guard DDoS Mitigation diverts the zone traffic from its normal network path to the Guard learning and protection processes and then returns legitimate zone traffic back to the network.

Because the Cisco Anomaly Detection and Mitigation solution is typically hard to understand, the following network diagram illustrates how these two components complement each other:

 NOTE: As previously stated, you are not expected to answer questions on specific details pertaining to the Cisco DDoS Anomaly Detection and Mitigation solution. Additionally, you do not need to know intricate details about the communication between the two components.

Chapter Summary

 The following section is a summary of the major points you should be aware of in this post:

Intrusion Detection and Prevention Systems

• An Intrusion Detection System is used to detect malicious behavior
• IDS devices can only perform promiscuous monitoring based on an out-of-packet stream
• In order to allow an IDS to view traffic, the switch must be configured for SPAN
• SPAN can be configured to monitor all traffic from a port, ports, VLAN or group of VLANs
• An IDS cannot block traffic on its own
• An IDS can notify another infrastructure device (e.g. firewall or router) to shun an attack
• An IPS monitors traffic in the same way as an IDS
• An IPS can be implemented inline, in promiscuous mode, or a combination of both
• Unlike an IDS, an IPS is reactive, and can block traffic on its own
• In promiscuous mode, an IPS can notify other devices to shun an attack
• IDS /IPS devices use protocol analyzers to decode Application Layer protocols
• IDS /IPS devices are composed of a sensor, a console and an engine
• The IDS/IPS console monitors events and controls the sensors
• The IDS/IPS engine uses rules to generate alerts
• The IDS/IPS sensor provides packet capturing and analyzing capabilities
• The IDS/IPS sensor uses multiple micro-engines to analyze traffic against signatures
• The following table lists and describes IDS and IPS you are expected to know:

Term	Description
Alert or Alarm	A signal suggesting a system has been or is being attacked
True attack stimulus	An event that triggers an system to produce an alarm and react as though a real attack were in progress
False attack stimulus	The event signaling the system to produce an alarm when no attack has taken place
False Positive	An alert or alarm that is triggered when no actual attack has taken place; i.e. an incorrect positive alert or alarm
False negative	A failure of the system to detect an actual attack
Noise	Data or interference that can trigger a false positive
Site policy	Guidelines within an organization that control the rules and configurations of the system
Site policy awareness	The ability the system has to dynamically change its rules and configurations in response to changing environmental activity
Confidence value	A value an organization places based on past performance and analysis to help determine its ability to effectively identify an attack
Alarm filtering	The process of categorizing attack alerts produced by the system in order to distinguish false positives from actual attacks

• The four methods used by IDS and IPS devices to detect attacks are:

1. Policy-based Detection
2. Anomaly-based Detection
3. Honey Pot Detection
4. Signature-based Detection

• Policy-based detection is based on administrator definition of normal network behavior
• Anomaly-based detection uses either statistical or non-statistical anomaly-based detection
• A honey pot is a trap set to lure attackers away from legitimate targets
• Signature-based detection examines network traffic for defined attack patterns
• Signature-based detection is the primary method used by IDS/IPS devices

IDS and IPS Signatures

• Signatures are at the heart of signature-based detection solutions
• Signatures can be grouped into two general classes: attack and informational
• Attack signatures are used to indicate that an attack is occurring
• Informational signatures are used to gather information about traffic
• The design of signatures involves implementations and structures
• There are two implementation methods for signatures: context and content
• With context implementation, a signature looks only at the packet header
• With content implementation, the signature looks at the packet payload
• Signature structure determines how many packets the IDS/IPS device has to examine
• There are two basic signature structures: atomic and stateful
• An atomic structure looks at a single packet for a match on a signature
• Stateful signatures are triggered on a sequence of specific events
• The most important part of a signature is the mechanism that causes it to trigger
• The most common triggering mechanisms are pattern, anomaly, and behavior-based
• When a signature is triggered, it can perform one of the following activities:

1. Generating an alert
2. Dropping or preventing the activity
3. Logging the activity
4. Resetting a TCP connection
5. Blocking future activity
6. Allowing the activity

• IDS and IPS devices can generate atomic alerts, summary alerts or a combination of both
• Atomic alerts are the most common; they are generated every time a signature triggers
• A summary alert is a single alert that indicates multiple occurrences of the same signature

IDS and IPS System Types

• There are several different types of IDS types, which are:

1. Network-based Intrusion Detection System (NIDS)
2. Protocol-based Intrusion Detection System (PIDS)
3. Application Protocol-based Intrusion Detection System (APIDS)
4. Host-based Intrusion Detection System (HIDS)
5. Hybrid Intrusion Detection System

• There are several different types of IPS solutions. These are:

1. Host-based Intrusion Prevention Systems (HIPS)
2. Network-based Intrusion Prevention Systems (NIPS)
3. Content-based Intrusion Prevention Systems (CIPS)
4. Rate-based Intrusion Prevention Systems (RBIPS)

Cisco Network Intrusion Prevention Systems

• The Network-based Intrusion Prevention solutions available from Cisco are:

1. The Cisco IPS 4200 Series Appliance Sensors
2. The Cisco IDS Services Module 2 (IDSM-2)
3. The Cisco Advanced Inspection and Protection Security Services Module
4. The Cisco IPS Advanced Integration Module (IPS-AIM)
5. The Cisco Internetwork Operating System (IOS) IPS

• The Cisco IPS 4200 Series sensors are purpose-built appliances manufactured by Cisco
• The Cisco IDSM-2 can be installed into Catalyst 6500 switches or 7600 series routers
• The Cisco AIP-SSM is designed for the Cisco ASA 5500 series security appliances
• The Cisco IPS-AIM can be used with Cisco Integrated Services Routers, such as the 2800
• The Cisco IOS IPS provides an integrated solution with the router software architecture
• Cisco IPS Sensor software can use the following protocols to exchange IPS events:

1. RDEP2
2. IDIOM
3. IDCONF
4. SDEE
5. CIDEE

Cisco Host Intrusion Prevention Systems

• The Cisco Security Agent is the Cisco Host-based Intrusion Prevention solution
• CSA provides proactive HIDS and HIPS solutions for endpoint systems
• Unlike other IPS solutions, CSA does not rely on a signature-based architecture
• CSA uses a flexible policy-based and behavior-based architecture
• CSA also has the capability to provide policy compliance controls
• The following list highlights the characteristics and benefits offered by CSA:

1. Endpoint system protection
2. Host-based intrusion prevention
3. Policy-based and behavior-based architecture
4. Personal firewall protection
5. Day-zero attack protection
6. Regulatory policy compliance enforcement
7. Acceptable corporate use policy compliance
8. Preventative protection against targeted attacks
9. Stability and protection of the underlying Operating System
10. File and directory protection
11. Host application visibility
12. Application control
13. Correlation of system calls and application functions

• The CSA software is unique in that the host agent resides between the applications and OS
• CSA architecture intercepts all OS and application-related calls
• CSA is also capable of intercepting dynamic runtime resources requested
• CSA has three basic components:

1. CSA endpoints – e.g. computing desktops, servers, laptops, and POS terminals
2. CSA Management Server – which is the core component of the CSA architecture
3. CSA Management Console – an administrative web-based user interface

• CSA architecture uses a grouping system
• Groups are a logical collection of hosts
• Groups in CSA MC are divided into the following three categories:

1. Mandatory groups
2. Predefined groups
3. Custom groups

• By default, CSA MC provides three auto-enrollment mandatory groups, which are:

1. All Windows
2. All Solaris
3. All Linux

• In CSA terminology, policy is a collection of rule modules
• In CSA terminology, a rule module is a collection of multiple rules
• Rules provide control access to system or network resources
• At the core of CSA architecture is the interceptor and correlation mechanism
• Correlation is the enforcement engine for CSA

Cisco IOS IPS Overview and Implementation

• Cisco is the only vendor that delivers integrated IPS functionality on a router
• The IOS IPS feature is a suite of intrusion prevention solutions
• Some of the key features of the IOS IPS are:

1. It protects the network from viruses, worms and a large variety of threats and exploits
2. It eliminates the need for a standalone IPS device
3. It provides integrated inline deep-packet inspection
4. It complements the Cisco IOS Firewall and VPN solutions for superior threat protection
5. It supports about 2000 attack signatures
6. It uses Cisco IOS routing capabilities to deliver integrated functionality
7. It enables distributed network-wide threat mitigation
8. It sends a Syslog message or an alarm in SDEE format when a threat is detected

• Cisco IOS IPS routers use Signature Definition Files (SDFs) to identify malicious traffic
• An SDF is a file, usually in XML format, that contains signature definitions
• In most cases, the SDF is located in the router Flash Memory
• Signature Definition Files use an .sdf file extension, for example 128MB.sdf
• Before configuring Cisco IOS IPS, the following two prerequisites must be satisfied:

1. Download the signature package from the Cisco website
2. Download the Public Crypto Key from the Cisco website

Anomaly Detection and Mitigation

• Anomaly intrusion detection and prevention is used to combat DoS and DDoS attacks
• Anomaly detection solutions categorize traffic as either normal or anomalous (abnormal)
• Anomaly detection classification is based on heuristics, rather than traditional signatures
• Anomaly detection defines or learns normal activity using the following methods:

1. Protocol anomaly detection
2. Behavioral anomaly detection
3. Network anomaly detection

• Protocol anomaly detection involves looking for deviations from a standard protocol
• Behavioral anomaly detection involves learning normal user behavior
• Network anomaly detection involves watching or learning the normal network traffic levels
• The following section lists the characteristics of anomaly detection techniques:

1. They do not require the use of signatures or patterns
2. They are granular and are based on observed traffic pattern behavior
3. They can perform relational and behavioral anomaly detection
4. They detect in real-time; i.e. anything they report is actually happening at that time
5. They support dynamic filtering
6. They include sophisticated anti-spoofing techniques
7. They can detect day-zero and minute-zero attacks
8. They can highlight behaviors that do not indicate an attack, but are of some interest
9. They include a traffic diversion architecture

• The Cisco Anomaly Detection and Mitigation solution provides the following capabilities:

1. It can detect and mitigate a wide range of DDoS attacks
2. It can classify legitimate traffic and attack traffic in real-time
3. It can block large botnet and zombie attacks
4. It can block attack traffic by using source-based dynamic filters
5. It delivers multi-gigabit performance at line rate

• The Cisco DDoS Anomaly Detection and Mitigation solution consists of two components:

1. Cisco Traffic Anomaly Detector
2. Cisco Guard DDoS Mitigation

• The Cisco Traffic Anomaly Detector identifies potential DDoS attacks
• When anomalous traffic is detected, the Cisco Traffic Anomaly Detector does the following:

1. It sends an alert to initiate a manual response from network security administrators
2. It triggers and existing management system
3. It launches the Cisco Guard DDoS Mitigation device to begin mitigation

• The Cisco Traffic Anomaly Detector performs the following tasks:

1. Traffic learning
2. Traffic anomaly detection

• The Cisco Guard DDoS Mitigation is a unique Multi-Verification (MVP) architecture
• The Cisco Guard DDoS Mitigation can identify a broad range of DDoS attacks, such as:

1. TCP-based attacks
2. UDP-based attacks
3. HTTP attacks
4. Botnet and Zombie attacks
5. DNS attacks
6. Voice over IP attacks

• The Cisco Guard DDoS Mitigation performs the following tasks:

1. Traffic learning
2. Traffic protection
3. Traffic diversion

Read more on the Cisco website.