Troubleshooting Switches at the Physical Layer

Cisco IOS switches support several commands that can be used to troubleshoot Layer 1, or at least suspected Layer 1, issues. However, in addition to being familiar with the software command suite, it is also important to have a solid understanding of physical indicators (i.e., LEDs) that can be used to troubleshoot link status or that indicate an error condition.

We cover troubleshooting in detail in our Cisco CCNP ENCOR course.

Troubleshooting Link Status Using Light Emitting Diodes (LEDs)

If you have physical access to the switch or switches, LEDs can be a useful troubleshooting tool. Different Cisco Catalyst switches provide different LED capabilities. Understanding the meaning of the LEDs is an integral part of Catalyst switch link status and system troubleshooting. Cisco Catalyst switches have front-panel LEDs that can be used to determine link status, as well as other variables such as system status. The supported LEDs, as well as what they indicate, are listed and described in Table 1 below:

Table 1. Cisco Ethernet Module LED Status

LED	Color or State	Description
Status	Green	The color green indicates that all diagnostics have passed and that the module is operational.
	Orange	The color orange indicates that the module is booting or running diagnostics. This color could also indicate that an over-temperature condition (i.e., a minor temperature threshold) has been exceeded during environmental monitoring.
	Red	The color red indicates that the module is resetting. The switch has just been powered on or the module has been hot inserted during the normal initialization sequence. The color red could also indicate that an over temperature condition (i.e., a major temperature threshold) has been exceeded during environmental monitoring. If the module fails to download code and configuration information successfully during the initial reset, the LED stays red; the module does not come online.
	Off	If the status LED is off, this indicates that the module is not receiving power or has been powered off.
LINK	Green	The color green indicates that the port is active and the link is connected and operational.
	Orange	The color orange indicates that the port is disabled through the CLI command (i.e., the shutdown command is configured).
	Flashing Orange	A flashing orange LED indicates that the port is faulty and has been disabled by the system.
	Off	If the link LED is off, then this indicates that the port is not active or the link is not connected. It does not mean no cable is connected.
PHONE	Green	If the phone LED is green, then this indicates that the voice daughter card is installed.
	Off	If the phone LED is off, then this indicates that the voice daughter card is not detected or is not installed.

Other popular Catalyst switches have LEDs that can be used for link and system troubleshooting. The Ethernet module LEDs, as well as the meanings of these LEDs, are listed and described in Table 2 below:

Table 2. Cisco Catalyst 4500 Ethernet Module LED Status

LED	Color or State	Description
Status	Green	The color green indicates that all diagnostic tests have passed.
	Red	The color red indicates that a test, other than an individual port test, has failed.
	Orange	The color orange indicates system boot-up, that self-test diagnostics are running, or that the module is disabled.
Link	Green	The color green indicates that the port is operational and that a signal has been detected.
	Orange	The color orange indicates that the link has been disabled by software (i.e., the shutdown command has been issued).
	Flashing Orange	A flashing orange LED indicates that the link has been disabled due to a hardware failure.
	Off	If the link LED is off, then this means no signal has been detected. It does not necessarily mean that the port is not connected.
Port Status	Green	The color green indicates that the port is operational and that a signal has been detected.
	Orange	The color orange indicates that the link or port has been disabled by software.
	Flashing Orange	A flashing orange LED indicates that the link has been disabled due to a hardware failure.
	Off	If the LED is off, then this indicates that no signal is detected.

In addition to understanding what the different LED colors mean, it is also important to have an understanding of what action to take to remedy the issue. For example, assume that you are troubleshooting a Catalyst 6500 Series switch and you notice that the status LEDs on the supervisor engine (or any switching modules) is red or off. In such cases, it is possible that the module might have shifted out of its slot or, in the event of a new module, was not correctly inserted into the chassis. In this case, the recommended action is to reseat the module. In some cases, it also may be necessary to reboot the entire system.

While a link or port LED color other than green typically indicates some kind of failure or other issues, it is important to remember that a green link light does not always mean that the cable is fully functional. For example, a single broken wire or one shutdown port can cause the problem of one side showing a green link light while the other side does not. This could be because the cable encountered physical stress that causes it to be functional at a marginal level. In such cases, the CLI can be used to perform additional troubleshooting.

Using the Command Line Interface to Troubleshoot Link Issues

Several Command Line Interface (CLI) commands can be used to troubleshoot Layer 1 issues on Cisco IOS Catalyst switches. Commonly used commands include the show interfaces, the show controllers, and the show interface [name] counters errors commands. In addition to knowing these commands, you also are required to be able to interpret accurately the output or information that these commands provide.

The show interfaces command is a powerful troubleshooting tool that provides a plethora of information, which includes the following:

• The administrative status of a switching port
• The port operational state
• The media type (for select switches and ports)
• Port input and output packets
• Port buffer failures and port errors
• Port input and output errors
• Port input and output queue drops

Following is the output of the show interfaces command for a GigabitEthernet switch port:

Catalyst-3750-1#show interfaces GigabitEthernet3/0/1 GigabitEthernet0/1 is up, line protocol is down (notconnect) Hardware is GigabitEthernet, address is 000f.2303.2db1 (bia 000f.2303.2db1) MTU 1500 bytes, BW 10000 Kbit, DLY 1000 usec, reliability 255/255, txload 1/255, rxload 1/255 Encapsulation ARPA, Loopback not set Keepalive not set Auto-duplex, Auto-speed, link type is auto, media type is unknown input flow-control is off, output flow-control is desired ARP type: ARPA, ARP Timeout 04:00:00 Last input never, output never, output hang never Last clearing of “show interface” counters never Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0 Queueing strategy: fifo Output queue: 0/40 (size/max) 5 minute input rate 0 bits/sec, 0 packets/sec 5 minute output rate 0 bits/sec, 0 packets/sec 0 packets input, 0 bytes, 0 no buffer Received 0 broadcasts (0 multicasts) 0 runts, 0 giants, 0 throttles 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored 0 watchdog, 0 multicast, 0 pause input 0 input packets with dribble condition detected 0 packets output, 0 bytes, 0 underruns 0 output errors, 0 collisions, 1 interface resets 0 babbles, 0 late collision, 0 deferred 0 lost carrier, 0 no carrier, 0 PAUSE output 0 output buffer failures, 0 output buffers swapped out

Most Cisco Catalyst switch ports default to the ‘notconnect’ state as illustrated in the first line of the output printed by this command. However, a port can also transition to this state if a cable is removed from the port or is not correctly connected. This status is also reflected when the connected cable is faulty or when the other end of the cable is not connected to an active port or device (e.g., if a workstation connected to the switch port is powered off).

NOTE: When troubleshooting GigabitEthernet ports, this port status may also be a result of incorrect gigabit interface converters (GBICs) being used between the two ends.

The first part of the output in the first line printed by this command (i.e., [interface] is up) refers to the Physical layer status of the particular interface. The second part of the output (i.e., line protocol is down) indicates the Data Link layer status of the interface. If this indicates an ‘up’, then it means that the interface can send and receive keepalives. Keep in mind that it is possible for the switch port to indicate that the Physical layer is up, while the Data Link layer is down, for example, such as when the port is a SPAN destination port or if the local port is connected to a CatOS switch with its port disabled.

The Input queue indicates the actual number of frames dropped because the maximum queue size was exceeded. The flushes column counts Selective Packet Discard (SPD) drops on the Catalyst 6000 Series switches. SPD drops low-priority packets when the CPU is overloaded in order to save some processing capacity for high-priority packets. The flushes counter in the show interfaces command output increments as part of SPD, which implements a selective packet drop policy on the IP process queue of the router. Therefore, it applies only to process-switched traffic. The different Cisco IOS switching methods will be described again later in this guide.

The Total output drops indicates the number of packets dropped because the output queue is full. This is often seen when traffic from multiple inbound high bandwidth links (e.g., GigabitEthernet links) is being switched to a single outbound lower bandwidth link (e.g., a FastEthernet link). The output drops increment because the interface is overwhelmed by the excess traffic due to the speed mismatch between the inbound and outbound bandwidths.

In addition to the show interfaces command, the show interfaces [name] counters errors command can also be used to view interface errors and facilitate Layer 1 troubleshooting. Following is the output that is printed by the show interfaces [name] counters errors command:

Catalyst-3750-1#show interfaces GigabitEthernet3/0/1 counters errors   Port        Align-Err    FCS-Err   Xmit-Err    Rcv-Err UnderSize Gi3/0/1         0          0          0          0         0   Port     Single-Col Multi-Col  Late-Col Excess-Col Carri-Sen   Runts  Giants Gi3/0/1       0         0         0          0         0        0       0

The following section describes some of the error fields included in the output of the show interfaces [name] counters errors command and which issues or problems are indicated by non-zero values under these fields.

The Align-Err field reflects a count of the number of frames received that do not end with an even number of octets and that have a bad cyclic redundancy check (CRC). These errors are usually the result of a duplex mismatch or a physical problem, such as cabling, a bad port, or a bad network interface controller (NIC). When the cable is first connected to the port, some of these errors can occur. In addition, if there is a hub connected to the port, collisions between other devices on the hub can cause these errors.

The FCS-Err field reflects the number of valid-size frames with Frame Check Sequence (FCS) errors but no framing errors. This is typically a physical issue, such as cabling, a bad port, or a bad NIC. Additionally, a non-zero value under this field could indicate a duplex mismatch.

A non-zero value in the Xmit-Err field is an indication that the internal send (Tx) buffer is full. This is commonly seen when traffic from multiple inbound high bandwidth links (e.g., GigabitEthernet links) is being switched to a single outbound lower bandwidth link (i.e., a FastEthernet link), for example.

The Rcv-Err field indicates the sum of all receive errors. This counter is incremented when the interface receives an error such as a runt, a giant, or an FCS, for example.

The Undersize field is incremented when the switch receives frames that are smaller than 64 bytes in length. This is commonly caused by a faulty sending device.

The various collisions fields indicate collisions on the interface. This is common for half-duplex Ethernet, which is almost non-existent in modern networks. However, these counters should not increment for full-duplex links. In the event that non-zero values are present under these counters, this typically indicates a duplex mismatch issue. When a duplex mismatch is detected, the switch prints a message similar to the following on the console or in the log:

%CDP-4-DUPLEX_MISMATCH: duplex mismatch discovered on FastEthernet0/1 (not full duplex), with R2 FastEthernet0/0 (full duplex)

As will be described in the section pertaining to Spanning Tree Protocol (STP), duplex mismatches can cause STP loops in the switched network if a port is connected to another switch. These mismatches can be resolved by manually configuring the speed and the duplex of switch ports.

The Carri-Sen (carrier sense) counter increments every time an Ethernet controller wants to send data on a half-duplex connection. The controller senses the wire and ensures that it is not busy before transmitting. A non-zero value under this field indicates that the interface is operating in half-duplex mode. This is normal for half-duplex.

Non-zero values can also be seen under the Runts field due to a duplex mismatch or because of other Physical layer problems, such as a bad cable, port, or NIC on the attached device. Runts are received frames with a bad CRC that are smaller than the minimum IEEE 802.3 frame size, which is 64 bytes for Ethernet.

Finally, the Giants counter is incremented when frames are received that exceed the IEEE 802.3 maximum frame size, which is 1518 bytes for non-jumbo Ethernet, and that have a bad FCS. For ports or interfaces connected to a workstation, a non-zero value in this counter is typically caused by a bad NIC on the connected device. However, for ports or interfaces that are connected to another switch (e.g., via a trunk link), this field will contain a non-zero value if 802.1Q encapsulation is used. With 802.1Q, the tagging mechanism implies a modification of the frame because the trunking device inserts a 4-byte tag and then re-computes the FCS.

Inserting a 4-byte tag into a frame that already has the maximum Ethernet size creates a 1522-byte frame that can be considered a baby giant frame by the receiving equipment. Therefore, while the switch will still process such frames, this counter will increment and contain a non-zero value. To resolve this issue, the 802.3 committee created a subgroup called 802.3ac to extend the maximum Ethernet size to 1522 bytes; however, it is not uncommon to see a non-zero value under this field when using 802.1Q trunking.

Finally, the show controllers ethernet-controller <interface> command can also be used to display traffic counter and error counter information similar to that printed by the show interfaces and show interfaces <name> counters errors commands. Following is the output of the show controllers ethernet-controller <interface> command:

Catalyst-3750-1#show controllers ethernet-controller GigabitEthernet3/0/1 Transmit GigabitEthernet3/0/1       Receive 4069327795 Bytes                    3301740741 Bytes 559424024 Unicast frames           376047608 Unicast frames 27784795 Multicast frames           1141946 Multicast frames 7281524 Broadcast frames           1281591 Broadcast frames 0 Too old frames           429934641 Unicast bytes 0 Deferred frames          226764843 Multicast bytes 0 MTU exceeded frames      137921433 Broadcast bytes 0 1 collision frames               0 Alignment errors 0 2 collision frames               0 FCS errors 0 3 collision frames               0 Oversize frames 0 4 collision frames               0 Undersize frames 0 5 collision frames               0 Collision fragments 0 6 collision frames 0 7 collision frames         257477 Minimum size frames 0 8 collision frames      259422986 65 to 127 byte frames 0 9 collision frames       51377167 128 to 255 byte frames 0 10 collision frames      41117556 256 to 511 byte frames 0 11 collision frames       2342527 512 to 1023 byte frames 0 12 collision frames       5843545 1024 to 1518 byte frames 0 13 collision frames             0 Overrun frames 0 14 collision frames             0 Pause frames 0 15 collision frames 0 Excessive collisions            0 Symbol error frames 0 Late collisions                 0 Invalid frames, too large 0 VLAN discard frames      18109887 Valid frames, too large 0 Excess defer frames             0 Invalid frames, too small 264522 64 byte frames                  0 Valid frames, too small 99898057 127 byte frames 76457337 255 byte frames                 0 Too old frames 4927192 511 byte frames                 0 Valid oversize frames 21176897 1023 byte frames                0 System FCS error frames 127643707 1518 byte frames                0 RxPortFifoFull drop frame 264122631 Too large frames 0 Good (1 coll) frames 0 Good (>1 coll) frames

NOTE: The output above will vary slightly depending on the switch platform on which this command is executed. For example, Catalyst 3550 Series switches also include a Discarded frames field, which shows the total number of frames whose transmission attempt is abandoned due to insufficient resources. A large number in this field typically indicates a network congestion issue. In the output above, you would look at the RxPortFifoFull drop frame field, which indicates the total number of frames received on an interface that are dropped because the ingress queue is full.

Read the Cisco troubleshooting pages.