Given a scenario, use appropriate software tools to troubleshoot connectivity issues. This chapter describes software tools and applications that include protocol analyzers, throughput testers, ping, traceroute, dig, ipconfig, arp, nbtstat, netstat, and route. You learn all the important Linux commands and tools in our CompTIA Linux+ course.

Back to book index.

Software Protocol Analyzers

As mentioned in the previous chapter, protocol analyzers are tools used to investigate upper layer protocol issues such as application problems, because they reveal the content of the packets on the network. Protocol analyzers work in both wired and wireless environments. Using a protocol analyzer involves the following steps:

• Make the necessary network adjustments to capture desired traffic and direct it to the protocol analyzer (using port mirroring in most cases)
• Start the packet capture on the desired interface
• Generate network application traffic that you want to investigate
• Stop the capture
• Save the captured traffic in a file
• Filter the traffic based on the desired criteria:
  • Timestamps
  • Protocols
  • Port numbers
  • Source IP
  • Destination IP
  • Packet size
• Examine the relevant traffic and find the problem

Most software protocol analyzers are able to convert the network bits into relevant information that can be easily readable and interpreted. For example, they auto-identify port numbers used in different packets and translate them into the corresponding protocol and message type. Some software protocol analyzers also include other useful tools, including:

• Traffic accounting applications
• Traffic analytic tools
• Features that can rebuild the traffic flow from the captured packets

The most powerful open source software protocol analyzer on the market is Wireshark (we offer Wireshark training on www.howtonetwork.com by the way). The Wireshark packet capture interface offers many useful fields and features, as can be seen in the screenshot below:

Figure 23.1 – Wireshark Packet Capturing Interface 

The major components of the Wireshark packet capturing interface are as follows:

• Tool bar, including filtering options
• Live packet count and capture, with a brief description of each (time, source, destination, protocol)
• Detailed information per each layer; each component field per layer is presented, depending on the packet type
• Hex packet format

Throughput Testers

As you build the network and install all kinds of devices, one of the questions you should ask yourself is the performance you should expect as traffic passes through all those devices. Dedicated testing equipment and software can push the network equipment to its limits, and this accomplishes several tasks:

• Measures network throughput
• Measures application performance

These tasks are accomplished by generating different types and amounts of traffic and injecting it into the network. This allows you to evaluate the devices’ performance under different conditions, by simulating possible production traffic.

As mentioned, this can be achieved using either hardware or software tools. Software tools can be either free or open source and are less expensive than dedicated hardware appliances because they can be used on existing machines (workstations or servers) in the network. The downside to using software tools is their limited capacity and functionality compared with hardware equipment.

On the other hand, throughput testing hardware appliances are usually very expensive but they offer increased performance and functionality. These tools are used in critical environments that need to be thoroughly tested from a performance perspective before placing them into production.

A very common open source throughput measuring/traffic generator tool is called Iperf, which also comes with a GUI frontend called Jperf, which is shown in Figure 23.2 below. This tool works in a Client-Server model, so you need to set up one device that will generate traffic and another device that will receive the traffic.

Figure 23.2 – Jperf Interface 

Some of the most important parameters that can be configured from the Jperf interface include the following:

• Client-Server configuration:
  • IP address
  • Ports
  • Number of connections
• Application Layer options:
  • Report interval
  • MSS output
  • Port
  • Output format
• Transport Layer options:
  • TCP or UDP
  • Buffer size
  • TCP window size
  • MSS
• IP Layer options:
  • TTL
  • Type of Service
  • IPv6

Ping

Ping is a command line tool commonly used to test reachability between two network locations. This is often considered a tool that allows you to see whether a device is up and active on the network. In addition to the basic reachability test, ping also provides round-trip time (RTT) information to the specific destination. This reflects the time that it took the packet to get to the other side and return to the originating device. Ping is also used to test packet loss on the connection.

Ping uses ICMP (Internet Control Message Protocol) as the underlying protocol (Layer 3), so when trying to test the network using this tool, you must be sure that no devices between the source and the destination are filtering ICMP traffic. Ping is usually the first troubleshooting test deployed when an application error or reachability error occurs because it is a simple way to test Layer 3 connectivity. If the ping works, you can be sure that no Layer 1, Layer 2, or Layer 3 problems exist, so you can start focusing your efforts on troubleshooting upper layer protocols.

The ping utility functions by sending ICMP Echo Request messages to a machine to verify that it is up and running. If the specific machine is operating, it will send an ICMP Echo Reply message back to the source, confirming its availability.

ICMP packets are not usually treated with priority on network devices. For example, if a router is busy completing critical tasks (e.g., routing processes), it will only answer an ICMP query after it is finished with the other tasks. This can lead to inaccurate time information received from the ping utility, so you should always measure the RTT multiple times, at different times of the day.

The ping utility was developed by Mike Muss in 1983 and he considered it an emulation of the sonar process, which sends out a signal and expects a response back.

Note:     Ping is not an acronym; it is just the name of the utility. Some say that ping is an abbreviation of Packet Internet Groper, but its author has stated that the name comes from the sound that sonar makes.

Every operating system has a utility to ping and this is usually built into the system. Based on the OS, the ping can have different options and parameters, which can be seen by typing the ping command and then the IP address of the destination. Next, we will look at a few ping examples on different platforms.

Windows Example

By default, issuing a simple ping on a Windows platform will send four packets and calculate the average statistics from this operation, as can be seen in the screenshot below:

Figure 23.3 – Windows Standard Ping 

Some of the relevant ping output information includes:

• Default packet size: 32 bytes
• TTL: 41
• Average RTT: 57 ms
• No packet loss

However, if you need advanced parameters, the ping help page lists all the available options, as shown in the screenshot below:

Figure 23.4 – Windows Ping Parameters 

Some of the Windows ping parameters include:

• Number of packets
• Packet size
• TTL
• Timeout between each reply
• Source address

Linux Example

As opposed to the Windows utility, Linux pings do not send a limited amount of ping packets, so they must be stopped by pressing the Ctrl+C key combination. At that point the OS will create statistics based on the RTT for each packet. This can be seen in the screenshot below:

Figure 23.5 – Linux Standard Ping

The Linux ping options include the following, which can be seen in Figure 23.6 below:

• TTL
• Source interface
• Seconds to wait for response
• Quiet mode (output only at ping finish)

Figure 23.6 – Linux Ping Parameters

Cisco IOS Example

Network devices operating at Layer 3 (e.g., routers, firewalls, Layer 3 switches, etc.) are also capable of sending pings. The default ping on a Cisco router sends five packets and the statistics are similar to the ones offered by the Windows utility, as can be seen in the screenshot below:

Figure 23.7 – Cisco IOS Standard Ping 

However, if you need to modify the ICMP packets sent by the ping, you can use the advanced options available in the Cisco IOS, as shown in the screenshot below:

Figure 23.8 – Cisco IOS Ping Parameters 

Some of these parameters include:

• Repeat count
• Datagram size
• Timeout
• Source interface
• Type of Service

Traceroute

On modern networks there might be a number of routers between two machines on the network. The traceroute utility can give information about the number of hops traffic passes to reach a destination. Every OS has a built-in traceroute utility that can be used via different commands:

• Windows: tracert
• Linux: traceroute
• Cisco IOS: traceroute

You can use traceroute as a baseline test for network connectivity. You can also document the intermediary hops between two critical points in normal network operations and compare traceroute results any time there is a problem. This will allow you to see which point along the line has issues.

Traceroute works by sending UDP (for Cisco and Linux; the Windows tracert command uses ICMP Echo Request datagrams instead of UDP datagrams as probes) Request messages and manipulating the Time To Live (TTL) field. By increasing the TTL and then having each router decrement it as it sends it along to the next router, you will get a hop count from your source to your destination. A router hop is a packet sent from one router to another router. When the TTL on the packet reaches 0, the router sends an ICMP Time Exceeded message back to the source computer. When the destination receives the packet, it sends back an ICMP Port Unreachable message and the traceroute program stops.

Note:     On some networks ICMP packets are filtered for security reasons, so the traceroute command might provide inaccurate results. If the traceroute is blocked after a specific host, this might not mean that the network is down. It might just mean that ICMP packets are filtered at that point.

In Figure 23.9 below, you can see a traceroute command issued on a Cisco router for a destination located one hop away. This means that traffic from the router to that destination (192.168.10.2) will pass through a single hop (router) on the way: 10.10.10.2.

Figure 23.9 – Traceroute Example 

Next, we will look at the traceroute parameters applicable to different operating systems.

Windows Parameters

Some of the Windows traceroute parameters include the following, which can be seen in the screenshot below:

• Maximum hops
• Wait timeout
• Source address
• IPv6

Figure 23.10 – Windows Traceroute Parameters

Linux Parameters

The Linux platform offers increased flexibility in terms of traceroute parameters. Some of these, which can be seen in the screenshot below, include the following:

• Set the DF bit
• Use ICMP Echo Requests instead of UDP datagrams
• Print numeric address
• Verbose
• Number of probes per TTL
• Time to wait for response

Figure 23.11 – Linux Traceroute Parameters

Cisco IOS Parameters

Some of the Cisco IOS traceroute parameters, shown in the screenshot below, include the following:

• Source address
• Numeric display
• Timeout
• Probe count
• TTL
• Port number

Figure 23.12 – Cisco IOS Traceroute Parameters 

DNS Troubleshooting Tools

One of the most common communication types over the Internet involves DNS (Domain Name Service) traffic, in which hosts request the IP address based on the name of a website. There are two utilities that can help in the DNS troubleshooting process:

• Nslookup
• Dig (domain information groper)

These two utilities are very similar in functionality, so you can usually use one or the other. Nslookup and dig provide information about:

• IP addresses that are being resolved
• Canonical names configured in the DNS servers
• Cache timers

Nslookup is a utility present in both Windows and Linux-based operating systems that allows you to perform name queries and reverse name queries for troubleshooting purposes. On some Linux machines, nslookup is no longer supported so the recommendation is to use dig instead. Dig is a more functional program that provides more in-depth details compared with nslookup.

Dig is not included with Windows but you can install it as a third-party application. In most Linux environments, dig is installed by default.

Figure 23.13 – Windows nslookup Command 

Analyzing the nslookup output in Figure 23.13 above, you can see the following useful information:

• The IP address cisco.com was queried
• The primary DNS server queried for this was 192.168.178.1 (a local service)
• The local DNS service sent the query further to the non-authoritative server, which provided the queried website addresses

IPconfig and IFconfig

One of the first and most basic steps in the Layer 3 troubleshooting process is verifying the IP address and other associated parameters of the local interfaces. Depending on the platform you are using, two commands will take care of this operation:

• Windows: ipconfig
• Linux: ifconfig

To further troubleshoot any type of connection issue, you must know the devices’ IP address, subnet mask, and gateway. For example, if you want to perform a connectivity test using the ping command to a destination device but you do not know the IP address of that device, you can find it using the ipconfig or ifconfig commands on that machine.

Figure 23.14 – Windows ipconfig Command Output

As you can see in the screenshot above, if you type ipconfig in Windows CLI, a lot of useful information about the interfaces is provided:

• List of interfaces (both wired and wireless)
• IPv4 address of each interface
• Subnet mask of each interface
• IPv6 prefix of each interface

If you want to find out more detailed information for each interface, you can issue the ipconfig/? command for a list of available parameters. An interesting option is using the ipconfig/all command, as shown in Figure 23.15 below. This provides the following additional information:

• General OS addressing information:
  • Routing
  • DNS servers
  • WINS servers
• Interface description
• Layer 2 (MAC) address for each interface
• DNS suffix for each interface
• DHCP status for each interface
• DNS server information
• NetBIOS information

Figure 23.15 – Windows ipconfig/all Command Output 

In Linux environments, issuing the ifconfig command provides the following information, which can be seen in the screenshot below:

• List of interfaces (including interface id)
• Link encapsulation
• Layer 2 address
• Layer 3 address
• Network mask
• Broadcast address
• Interface packet counters, including dropped packets and collisions

Figure 23.16 – Linux ifconfig Command Output 

You can further tune the output using specific commands. For example, the ifconfig eth0 command will provide information only for the Ethernet0 interface.

Note:     Although the ipconfig and ifconfig commands provide similar outputs, one major difference is the ifconfig command does not provide default gateway information. This can be verified using other commands (such as route), which will be covered in detail in later sections.

ARP

Address Resolution Protocol (ARP) is a fundamental network protocol that allows hosts to learn the Layer 2 address of a device it wants to communicate with by sending a query based on the Layer 3 address of the specific device.  Details about the way ARP functions can be found in Chapter 5.

After the addresses are learned using the ARP process, Layer 2 addresses are cached for a period of time on the machine (e.g., workstations, network devices, etc.) so they can be used immediately in future communications without the need to send additional ARP requests.

If you want to examine the ARP cache to verify that the ARP request was successfully responded to, you can use the following commands based on the OS you are using (the outputs for each are shown in the screenshots below):

• Windows: arp -a
• Linux: arp

Figure 23.17 – Windows ARP Cache

Figure 23.18 – Linux ARP Cache 

Based on your needs, the arp commands in both Windows and Linux can be used with a series of keywords to provide more granular results. One interesting option with Windows is viewing the ARP cache per interface. This can be done using the arp –av command.

Nbtstat

Nbtstat is a utility used in networks that use NetBIOS communicating over TCP/IP and this is usually present in the Windows platform. This utility is built into the Windows operating system and offers information about other devices on the network, for example:

• Device name
• Device IP address
• Workgroup
• Logged users

The nbtstat command can be used to query the local machine or to send queries to a remote device, referenced by name or IP. Some of the command options include the following:

• nbtstat –n: lists local NetBIOS names
• nbtstat –a <name>: lists remote NetBIOS names, based on the name of the particular machine
• nbtstat –A <ip>: lists remote NetBIOS names, based on the IP of the particular machine.

If you issue the simple nbtstat command in a Windows console, it will list all the available options and functionalities, as can be seen in the screenshot below:

Figure 23.19 – Windows nbtstat Command Output

Netstat

The netstat (network statistics) command is included in most modern operating systems and it shows the active connections on a machine. You would generally use it when you want to find out what type of services are running on a workstation or what connections are open on that machine.

In a Windows environment, the following related commands are commonly used:

• netstat –a: lists all active connections (see the screenshot below)
• netstat –b: lists the applications (executables) that use the active connections
• netstat –n: lists the connections without doing a DNS resolution

Figure 23.20 – Windows netstat -a Command Output 

Some of the most important fields in the output above include the following:

• Protocol
• Local address
• Remote address
• Connection state (listening, established, syn_sent)

Figure 23.21 below shows the output for the netstat command in a Linux environment:

Figure 23.21 – Linux netstat Command Output

Route

Every Layer 3 device on the network has a routing table, even if it is not a router. This routing table, sometimes transparent to the user, decides where to send network traffic and on what interface based on the destination address. For workstations with a single NIC, this process is very simple, as every packet is sent out of the only interface available. However, if you have a server with multiple network interfaces, things get complicated because there are multiple ways to send traffic out to the network.

On most devices there is a command that allows the user to view the existing routing table or even to modify it. Most of the time you will use it only to view the routing table for troubleshooting purposes, but there might be situations in which you want to influence path selection decisions so you might want to edit the routing table. Caution must be taken in such situations because modifying the devices’ routing tables without knowing how the traffic flows can cause a lot of harm. In Windows and Linux environments, the route command is used to achieve this purpose.

Route on Windows

On Windows machines, if you type the route command you will see all the available options. To view the routing table (both for IPv4 and IPv6), you can issue the route print command, as shown in the screenshot below:

Figure 23.22 – Routing Table in Windows 

Some of the most important information you can see in the route print output include:

• Network destinations and network mask
• Gateway for each destination
• Outgoing interface
• Metric for each route

If you need to add a route in the routing table, can issue the route add <destination> mask <mask> <gateway> metric <metric> command. If you need to delete a route in the routing table, you can issue the route delete <prefix> command.

Route on Linux

On Linux machines, you can use the same command used on Windows machines to show the routing table, as shown in the screenshot below:

Figure 23.23 – Routing Table in Linux 

Some of the most important information you can see in the output above includes:

• Destination network
• Gateway
• Network mask
• Flags
• Route metric
• Outgoing interface

Summary

Protocol analyzers are tools used to investigate upper layer protocol issues such as application problems, because they reveal the content of the packets on the network. Protocol analyzers work in both wired and wireless environments.

As you build the network and install all kinds of devices, one of the questions you should ask yourself is the performance you should expect as traffic passes through all those devices. Dedicated testing equipment and software can push the network equipment to its limits and this accomplishes several tasks:

• Measures network throughput
• Measures application performance

Ping is a command line tool commonly used to test reachability between two network locations. This is often considered a tool that allows you to see whether a device is up and active on the network. In addition to the basic reachability test, ping also provides round-trip time information to the specific destination. This reflects the time that it took the packet to get to the other side and return to the originating device. Ping is also used to test packet loss on the connection.

On modern networks there might be a number of routers between two machines. The traceroute utility can give information about the number of hops traffic passes to reach a destination.

One of the most common communication types over the Internet involves DNS (Domain Name Service) traffic, in which hosts request the IP address based on the name of a website. There are two utilities that can help in the DNS troubleshooting process:

• Nslookup
• Dig (domain information groper)

One of the first and most basic steps in the Layer 3 troubleshooting process is verifying the IP address and other associated parameters of the local interfaces. Depending on the platform you are using, two commands take care of this operation:

• Windows: ipconfig
• Linux: ifconfig

The netstat (network statistics) command is included in most modern operating systems and it shows the active connections on a machine. You would generally use it when you want to find out what type of services are running on a workstation or what connections are open on that machine.

Every Layer 3 device on the network has a routing table, even if it is not a router. This routing table, sometimes transparent to the user, decides where to send network traffic and on what interface based on the destination address. The command used to view Windows and Linux routing tables is route.

Basic Linux commands guide.

Configure all the important Linux commands in our book 101 Labs – CompTIA Linux+