Network troubleshooting is not some mysterious skill that can only be carried out by the select few. With a methodical plan and using a step-by-step process, you should quickly be able to troubleshoot, diagnose, and resolve most of the problems you encounter in a computer network.
For the CCNA exam, you will be expected to troubleshoot some problems in a simulated network. You must be able to look at network topologies and identify the cause of the problem, and then quickly resolve it. This chapter is primarily focused on showing you the kinds of problems you could come across in the CCNA exam. It is the troubleshooting system that is important here, not the specific problems we will look at. We cover network troubleshooting in great detail in our CCNP ENSARI course.
Before you delve into troubleshooting, it’s important that you have a strong grasp of the technology from reading the theory pages several times over and, of course, completing the labs many times over until you no longer need to look up the commands.
When configuring a lab, you will find that the use of IOS show commands will ensure that everything is working as desired. When you are troubleshooting, it will almost always be in a network that has already been configured, or at least partly configured, so you should start off by using the show commands.
Show commands can give you a general overview of the status of a device or drill down to a very low level. You can also use debug commands to track events per service or protocol. Please use debug commands with great care on a live network because they can consume massive amounts of device resources and quickly cause it to crash, which can leave you open to disciplinary or legal issues.
There are many books written about troubleshooting in a Cisco networking environment, and as you progress through your career, you should plan to read several of them. Before starting our review of the kinds of problems you will see on the CCNA exam, let’s first discuss some general background on troubleshooting.
In the exam, you could be tested on your troubleshooting skills in several ways:
- Fix a partly configured or broken network using the simulator
- Log in to devices and answer questions about which parts are broken
- Answer theory-only questions about probable causes of network problems
- Answer questions about the best show or debug command to use in a situation
In fact, you may well have to employ your troubleshooting skills during the configuration lab if it isn’t performing as required. Bear in mind that you will be working with a simulator and not on a live device, so the debug and show commands available will be limited.
Your Network Troubleshooting Plan
A structured approach is generally the best method to use when troubleshooting networking issues. In a larger organization, this includes how problems are reported to the networking team, the troubleshooting procedures followed by the team, and the processes used to include other departments for problems that span organizational units. In smaller organizations, the formal structure may not be as well defined, as much of the approach will reside in a single person, namely you.
The initial steps in troubleshooting involve isolating the source of the problem. During this step (if it were a live network), you will want to discuss the symptoms seen by the people reporting the problem. Users in the network may mislead you with wrong information due to their lack of knowledge on the reported symptoms, which means that the symptoms may not be the root cause of the problem. Sometimes users think they know what the problem is and present facts that support their conclusion. The best advice during this step is to determine how widespread the problem is.
Attempt to isolate the issue to a single location, building, floor, and, finally, specific network segment or node. One method often used is termed “divide and conquer”, whereby you parse the network into progressively smaller sections until you isolate the location of the problem.
It’s important to gather information from the users but not be misled by them. Some could call in complaining that their e-mail isn’t working, which could be part of a far larger problem or simply due to their PCs running slowly. I’ve also noticed that when working on large networks managed by multiple teams, everyone is quick to blame the network for the problem.
Take in all the information provided and then apply your plan. If you know that all services are working on a device but only one user is having an issue, then there is little point in checking to see whether the power supply has failed. Because only one user is affected, it would be unlikely that the problem was caused by a configuration issue or network outage. As such, you would probably check the port the user is plugged into on the network switch.
Once you have isolated the problem in the network topology, then continue to apply structured critical thinking to isolate the root cause of the problem. You may use the OSI model as a reference to help you organize the troubleshooting commands and tools that you have at your disposal. In order to solve many network problems, you will need to be familiar with Cisco troubleshooting tools as well as those provided by other vendors for use on their products, such as Microsoft for Windows operating systems and the Open Source Foundation for Linux operating systems.
If you work in a structured support team, you may well need to involve senior engineers or vendor support in your troubleshooting process. There should be escalation procedures in place, as well as agreed methods to verify the solution and monitor the situation via SNMP or manual monitoring.
OSI layer 1 (physical) problems include:
- Faulty or broken cables
- Broken or faulty pins/connectors
- No power
- No cable connected or wrong interface
- Failing or damaged interface
- Incorrect cable for the interface
Layer 2 (data link) problems include:
- Incorrect configuration on the interface
- Clock rate missing or incorrect
- Incorrect layer 2 protocol settings
- Faulty network card
- Interface shut down
Layer 3 (network) problems include:
- Misconfigured routing protocol
- Incorrect IP/network addressing
- Incorrect subnet masking
Each of the layers above requires a different set of commands or debugs, although some show commands will display information for multiple layers, such as show interface serial 0/0, for example. The show ip ospf 10 command will not tell you whether an interface has gone down.
A great book to help you with real-life troubleshooting is:
Internetworking Troubleshooting Handbook (Cisco Press). ISBN: 1587050056
If I had to carry only one book in my briefcase when out in the field, it would be:
Cisco Field Manual: Router Configuration (Cisco Press). ISBN: 1587050242
These books are somewhat out of date, and the Cisco IOS and devices have come a long way since their publication, but the principles are still sound. With a methodical approach, you should be able to quickly and effectively troubleshoot and resolve any problem you encounter in the exam and on a live network in the real world.
Network Debugging
We have actually already covered this in various theory lessons and labs, but it’s time to directly address debugging Cisco devices.
Built into the Cisco IOS are a large number of show commands, as you already know, but you also have the ability to view status and error messages in real time, which Cisco refers to as debugging. Some of these messages are printed on the console screen automatically, such as interfaces coming up or going down, duplex errors, or VLAN mismatches. Others require you to issue a specific debug command for that protocol or technology.
If you need to debug a live network (commonly referred to as a production network), then you must exercise extreme caution and probably seek expert advice first. Some debug commands will return a huge amount of information very quickly, meaning that you won’t see a command prompt on the screen to turn them off, and you will quickly overload your CPU, causing the router to crash. I’ve seen too many careers end prematurely due to this mistake.
You may not actually have the ability to issue debug commands in the CCNA exam because you are working on router simulators, and there is no real traffic passing over the link. I can’t guarantee this, of course, but you could easily be asked theoretical debugging questions in the exam.
If you type debug ? at a router prompt, you will see several pages of available debugs:
R1#debug ?
IUA ISDN adaptation Layer options
aaa AAA Authentication, Authorization and
Accounting
aal2_xgcpspi AAL2_XGCP Service Provider Interface
access-expression Boolean access expression
acircuit Attachment Circuit information
adjacency adjacency
arp IP ARP and HP Probe transactions
asnl Application Subscribe Notify Layer
aspp ASPP information
async Async interface information
bgp BGP information
–More—
Usually, I advise using the question mark (?), which helps you see what options are available:
R1#debug ip ?
access-list IP access-list operations
address IP address activity
admission Network admission control debug
auth-proxy Authentication proxy debug
bgp BGP information
cache IP cache operations
dhcp Dynamic Host Configuration Protocol
director Director agent information
dns DNS
eigrp IP-EIGRP information
–More—
And many of these commands have options available or options within options:
R1#debug ip eigrp ?
<1-65535> Autonomous System
neighbor IP-EIGRP neighbor debugging
notifications IP-EIGRP event notifications
summary IP-EIGRP summary route processing
vrf Select a VPN Routing/Forwarding instance
<cr>
For troubleshooting, you will want to use debug commands after you have checked all of your configurations and used your available show commands but still can’t see the issue. If you are troubleshooting a WAN connection, you would use the debug outputs to prove to your ISP that the issue isn’t on your side of the connection because they will inevitably immediately blame you (as I’m sure you are already aware of if you work on a network team).
We will cover layer 2 debug commands in the relevant WAN sections, but they include:
debug serial packet
debug serial interface
debug ppp packet
debug ppp authentication
debug ppp negotiation
You can issue a debug ip packet command but I advise that you never do this on a live network due to the huge amount of output it produces. If you have a small home network, you can try it out. It’s worth noting that you can debug an ACL, but Cisco has made the command somewhat difficult to find:
R1#debug ip packet ?
<1-199> Access list
<1300-2699> Access list (expanded range)
detail Print more debugging detail
Here is some output from the command above if you really want to see it. Included are s=, which is the source IP address, and d= for the destination address:
R1#debug ip packet
IP packet debugging is on
*Mar 1 03:42:13.359: IP: s=192.168.1.1 (local), d=224.0.0.5 (FastEthernet0/0), len 80, sending broad/multicast
You can also drill into more detail if you append the tag detail to the end of the debug:
R1#debug ip packet detail
IP packet debugging is on (detailed)
*Mar 1 03:48:07.059: IP: s=192.168.1.2 (FastEthernet0/0), d=224.0.0.5, len 80, rcvd 0, proto=89
Protocol number 89 is used by OSPF. You may recognize 224.0.0.5 as the All OSPF Routers address. Each routing protocol offers its own set of debugs that we will address as they arise.
If you are connected to a remote device via a Telnet connection, then the debug output won’t show automatically (because it is printed on the console session by default). You will need to issue the terminal monitor command. You can send the logging messages (events) to an internal buffer with the logging buffered command. We covered logging in detail earlier.
You may need to add millisecond-level timestamps onto your debugs. Cisco will usually request this for chatty protocols or rapid message exchange such as Integrated Services Digital Network (ISDN) or Point-to-Point Protocol (PPP). You would add the service timestamps debug datetime msec command to do this, and if you issue a show run command, you may see that it’s on by default depending on your IOS release:
version 15.1
service timestamps debug datetime msec
Most importantly, turn off all debugs with the undebug all command, or un all for short. I’ve managed to type this when the terminal screen was flooded with debug output, and I thought the router was about to crash. The command took effect after several agonizing seconds.
Layer 1 Troubleshooting
Troubleshooting network devices at layer 1 might seem like a simple matter but it can become very complicated because there are a lot of parameters to check at this layer.
Although you may laugh, many layer 1 issues can be easily diagnosed, tested, and resolved by swapping out a cable. All Cisco devices have interface LEDs, which indicate various states (read the user’s manual for a complete list). No light or a red light usually indicates a faulty cable, so swap it out to test this theory. An amber (orange) light can mean that no keepalives are seen on the line. It can also mean that an incorrect cable has been installed, so ensure that the cable is straight-through (if going from a switch to a host).
Figure 28.1 below shows the common LEDs found on the Cisco 2960 model. The documentation runs into many pages so check the online manual for this model.
FIG 28.1 – Cisco 2960 LEDs (Image © Cisco Systems)
On Cisco routers and switches, you can focus on two commands:
- show controllers – displays hardware component statistics
- show interfaces – displays interface-level statistics
A show controllers output might look similar to the following:
R1#show controllers
Interface FastEthernet0/0
Hardware is GT96K FE ADDR: 6627DD48, FASTSEND: 6072E6E4, MCI_INDEX: 0
Bytes_recvd 0 Bytes_sent 0 Bytes_sent 0 Frames_sent 0
total_bytes_RX 0 Total_frames_RX 0 Bcast_frames_recvd 0
Mcast_frames_RX 0 CRC_err 0 Ovr_sized_frames 0
Fragments 0 Jabber 0 Collision 0
Late_collision 0 64B frame 0; 65_127B_frames 0
128_255B_frames 0 256_511B_frames 0 512_1023B_frames 0
1023_maxB_frames 0 Rx_error 0 Dropped_frames 0
Mcast_frames_tx 0 Bcast_frames_tx 0 Sml_frame_recvd 0
[output omitted]The command above is usually used on a Serial interface, however.
The output above presents hardware-level statistics for all router/switch interfaces and modules. Some of the most interesting parameters are as follows (most important items in bold):
- Bytes_recvd – total number of bytes received on the interface
- Bytes_sent – total number of bytes sent on the interface
- Frames_sent – total number of frames sent on the interface
- Bcast_frames_recvd – total number of broadcast frames received on the interface
- Mcast_frames_RX – total number of multicast frames received on the interface
- CRC_err – total number of CRC errors; incremented when the checksum calculated by the sender does not match the checksum calculated by the network device and usually indicates transmission problems that altered the packet (collisions or physical issues)
- Ovr_sized_frames – total number of oversized frames (frames larger than 1514 bytes)
- Fragments – total number of fragments; packets are fragmented when they cross interfaces with a smaller MTU (maximum transmission unit) than the packet’s size
- Collision – number of collisions
- Late_collision – number of late collisions (collisions detected late in the transmission process); usually indicates a duplex mismatch so you should check to ensure that the duplex and autonegotiation settings are consistent at both ends of the link
- Dropped_frames – number of frames dropped (among the multiple reasons a frame might get dropped, the most common ones include collisions and interface queue full)
Another tool you can use when troubleshooting layer 1 issues is the show interfaces X command:
R1#show interfaces FastEthernet0/1
FastEthernet0/1 is up, line protocol is up (connected)
Hardware is Gt96k FE, address is c201.0f00.0001 (bia c201.0f00.0001)
MTU 1500 bytes, BW 10000 Kbit/sec, DLY 1000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, Loopback not set
Keepalive set (10 sec)
Half-duplex, 10Mb/s, 100BaseTX/FX
ARP type: ARPA, ARP Timeout 04:00:00
Last input never, output never, output hang never
Last clearing of show interface counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
0 packets input, 0 bytes
Received 0 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored
0 watchdog
0 input packets with dribble condition detected
0 packets output, 0 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 unknown protocol drops
0 babbles, 0 late collisions, 0 deferred
0 lost carrier, 0 no carrier
0 output buffer failures, 0 output buffers swapped out
Some of the most interesting parameters you can inspect in the output above include the following (most important items in bold):
- Interface status – up, line protocol is up means that the interface is physically up and connected to another device at the other end
- MAC address
- MTU (maximum transmission unit) – this is 1500 for Ethernet by default
- Interface bandwidth
- Duplex parameters
- Last input, output – the time since the last packet was successfully received or transmitted by the interface
- Input queue – the number of packets in the input queue (if this number starts to get close to the maximum value, it means that the network device has issues with processing the packets compared to the rate at which it receives them, and it might soon start to discard packets)
- Total output drops – the number of packets dropped because the output queue is full (a common cause might be traffic from a high-bandwidth link being transmitted to a lower-bandwidth link)
- Output queue – the status of the output queue (if the number of packets in this queue gets close to the maximum value, packets might get dropped)
- 5 minute input rate and 5 minute output rate – the average input and output rate of the interface in the last five minutes
- Runts – the number of frames that are smaller than the minimum IEEE 802.3 frame size (64 bytes for Ethernet), with a bad CRC; usually caused by a duplex mismatch or physical problem (bad cable or port)
- Giants – the number of frames that are larger than the maximum IEEE 802.3 frame size (1518 for non-jumbo Ethernet); usually caused by a faulty interface module
- Throttles – the number of times the traffic received on the port is disabled; usually caused by a buffer or processor overload
- Input errors – includes runts, giants, no buffer, CRC, frame, overrun, and ignored counts
- CRC – this is incremented when the checksum calculated by the sender does not match the checksum calculated by the network device; usually indicates transmission problems that altered the packet (collisions or physical issues)
- Frame – the number of packets received incorrectly; usually caused by collisions or physical cabling problems
- Overrun – the number of times the receiver hardware was unable to forward the received data to a hardware buffer; caused by traffic rates that exceed the ability of the receiver hardware to handle data
- Ignored – the number of received packets ignored because the interface hardware ran low on internal buffers; usually caused by broadcast storms
- Underruns – the number of times the transmitter has ran faster than the switch can handle; this can happen in a high-throughput situation
- Output errors – the sum of all errors that prevented correct data transmission out of the interface
- Collisions – the number of times a collision occurred before the interface transmitted a frame successfully; usually seen on half-duplex interfaces
- Late collisions – the number of times a collision is detected late in the transmission process; can be caused by a duplex mismatch or physical cable/port issues
- Deferred – the number of frames that have been transmitted after they were put in hold because the media was busy; usually seen on half-duplex links, where sending and receiving packets cannot happen at once because of the shared media
Most commonly you will be troubleshooting input errors, which will be accompanied by CRC errors. This is a sign that the other side of the connection is set to half-duplex.
Switch1#show int f0/11
FastEthernet0/11 is up, line protocol is up (connected)
Hardware is Fast Ethernet, address is 0011.9247.db0b (bia 0011.9247.db0b)
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, Loopback not set
Keepalive set (10 sec)
Full-duplex, 100Mb/s
Queueing strategy: fifo
Output queue: 0/40 (size/max)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
1117 packets input, 91352 bytes, 0 no buffer
Received 108 broadcasts (0 multicast)
0 runts, 0 giants, 0 throttles
665 input errors, 660 CRC, 0 frame, 0 overrun, 0 ignored
[output truncated]However, you will also see the error message below printed on your console screen:
00:06:14: %CDP-4-DUPLEX_MISMATCH: duplex mismatch discovered on FastEthernet0/11 (not full duplex), with Switch1 FastEthernet0/11 (full duplex).
00:06:14: %CDP-4-DUPLEX_MISMATCH: duplex mismatch discovered on FastEthernet0/11 (not full duplex), with Switch1 FastEthernet0/11 (full duplex).
The same command on the switch connected to f0/11 reveals that the duplex setting is in fact incorrect.
Switch2#show int f0/11
FastEthernet0/11 is up, line protocol is up (connected)
Hardware is Fast Ethernet, address is 000d.292e.118b (bia 000d.292e.118b)
MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, Loopback not set
Keepalive set (10 sec)
Half-duplex, 100Mb/s
Switch2(config-if)#int f0/11
Switch2(config-if)#duplex full
You can also use the show ip interface brief command to quickly establish whether any interface is up or down and whether you need to move on to troubleshooting the interface or the encapsulation, duplex settings, etc.
Troubleshooting VLAN Issues
When troubleshooting end-to-end intraVLAN connectivity, you have to make sure that you verify a number of configuration aspects, including:
- IP addressing issues within the VLAN
- VLANs are created
- VLANs are correctly associated to switch ports
Usually, you want each VLAN to be assigned a unique IP subnet to avoid interVLAN communication issues. This means that two hosts that are part of the same VLAN will need to be assigned IP addresses from the same range. Therefore, if two hosts in the same VLAN cannot communicate, the first thing to check is whether they have IP addresses from the same subnet. Don’t forget to also check whether they have the same subnet mask.
If there is still no VLAN communication after the IP addressing verification, the next thing to check is whether the specific VLAN is correctly defined on all switches between the two hosts. This is accomplished using the show vlan command:
Switch#show vlan
VLAN Name Status Ports
1 default active Fa1/1
[output truncated]If the VLAN is not configured, you can create it using the vlan x command on the switch:
Switch#configure terminal
Switch(config)#vlan 5
Switch(config-vlan)#name CCNA
Switch(config-vlan)#exit
Switch#show vlan
VLAN Name Status Ports
1 default active Fa1/1
5 CCNA active Fa1/1
[output truncated]After properly creating the VLAN on all switches between the two hosts, you need to check whether the correct ports are associated with the respective VLAN. At the very least, the following ports have to be associated:
- The port connecting to the sender host
- The port connecting to the receiver host
- Any interswitch port connecting multiple switches between the two hosts
To check port-to-VLAN mapping, you can use the show vlan command. If you also have trunk connections between switches, you can check whether the specific VLAN is allowed on the trunk using the following command:
Switch#show interfaces trunk
Port Mode Encapsulation Status Native vlan
Fa1/11 on 802.1q trunking 1
Fa1/12 on 802.1q trunking 1
Port Vlans allowed on trunk
Fa1/11 1,3,4,
Fa1/12 1,3,4,
Port Vlans allowed and active in management domain
Fa1/11 1,3,4,
Fa1/12 1,3,4,
Port Vlans in spanning tree forwarding state and not pruned
Fa1/11 1,3,4,
Fa1/12 1,3,4,
If the specific VLAN is not allowed on the trunk, you can add it using the following command:
Switch(config)#int fast1/11
Switch(config-if)#switchport trunk allowed vlan add 5
Switch(config-if)#int fast1/12
Switch(config-if)#switchport trunk allowed vlan add 5
 |
Use caution in real-world scenarios: if you forget the add keyword, all previously allowed VLANs on the particular trunk will be removed from the trunk. |
Switch#show interfaces trunk
Port Mode Encapsulation Status Native vlan
Fa1/11 on 802.1q trunking 1
Fa1/12 on 802.1q trunking 1
Port Vlans allowed on trunk
Fa1/11 1,3,4,5
Fa1/12 1,3,4,5
Port Vlans allowed and active in management domain
Fa1/11 1,3,4,5
Fa1/12 1,3,4,5
Port Vlans in spanning tree forwarding state and not pruned
Fa1/11 1,3,4,5
Fa1/12 1,3,4,5
Now that all the trunk ports are carrying the relevant VLAN, you must also add it to the access ports connecting to the hosts:
Switch(config)#interface FastEthernet1/2
Switch(config-if)#switchport access vlan 5
Switch#show vlan
VLAN Name Status Ports
—- ——————————– ——— ——————–
1 default active Fa1/1
5 CCNA active Fa1/1, Fa1/2
At this point, you should have end-to-end VLAN connectivity between two hosts in the same VLAN.
Another useful command is show interface X switchport, which will tell you whether the interface is set to access/trunk/dynamic, which VLAN is native on it, encapsulation type, and other useful information:
Switch#show int f0/10 switchport
Name: Fa0/10
Switchport: Enabled
Administrative Mode: dynamic auto
Operational Mode: static access
Administrative Trunking Encapsulation: dot1q
Operational Trunking Encapsulation: native
Negotiation of Trunking: On
Access Mode VLAN: 1 (default)
Trunking Native Mode VLAN: 1 (default)
Remember that you can apply an IP address to a VLAN. This will also create an SVI for the switch. You can easily see this with the show ip interface brief command. The SVI must be opened with the no shut command in order for it to operate.
|
You do need to add the no shut command for older versions of IOS. |
Switch#show ip interface brief
Interface IP-Address OK? Method Status Protocol
FastEthernet0/1 unassigned YES manual up up
GigabitEthernet1/2 unassigned YES manual down down
Vlan1 192.168.1.2 YES manual up up
[output truncated]
Switch#show int vlan 1
Vlan1 is up, line protocol is up
Hardware is CPU Interface, address is 0002.4a4b.27a7 (bia 0002.4a4b.27a7)
Internet address is 192.168.1.2/24
Troubleshooting Trunks
Apart from a hardware or cable fault, most of the trunking problems originate from configuration errors. Common problems are discussed below:
- Trunk will not come up – First, check whether the interface status is up/up using the show ip interface brief or show interface [interface] The second thing to check is the mode configured on the switch port. This can be done using the show interface [interface] switchport command. This command will show an output similar to the one given below:
SwitchA#show interface fa1/1 switchport
Name: Fa1/1
Switchport: Enabled
Administrative Mode: trunk
Operational Mode: trunk
Administrative Trunking Encapsulation: dot1q
Operational Trunking Encapsulation: dot1q
Negotiation of Trunking: Disabled
Access Mode VLAN: 0 ((Inactive))
Important points to note are the Administrative Mode, Administrative Trunking Encapsulation, and Negotiation of Trunking lines. These will tell you the mode, the trunking protocol on the port, and DTP (Dynamic Trunking Protocol) status, respectively. Remember that ports set on Auto/Auto mode will not trunk (this was covered previously). A trunking protocol mismatch will not allow the trunk to come up. Also, remember that the only trunking protocol is dot1Q on the 2960 Switch and it might be ISL on older models of Cisco switches.
- Trunk does not carry traffic from relevant VLANs – Trunks carry the traffic for all VLANs by default. Only two things can cause this problem: allowed list and pruning. The show interface trunk command will show which VLANs are allowed across the trunk and which VLANs are pruned.
Switch(config-if)#switchport trunk allowed vlan 10-20
Switch(config-if)#end
Switch#show int trunk
Port Mode Encapsulation Status Native vlan
Fa0/1 on 802.1q trunking 1
Port Vlans allowed on trunk
Fa0/1 10-20
- Trunk link does not have the same encapsulation at both ends – This might happen because some Cisco switches allow both ISL and dot1Q encapsulation types. If, by mistake, you configure ISL at one end and dot1Q at the other, the trunk link will not work. This can be checked using the following command:
Switch#show interfaces trunk
Port Mode Encapsulation Status Native vlan
Fa0/1 on 802.1q trunking 1
Fa0/2 on 802.1q trunking 1
Troubleshooting VTP
VTP problems include the following:
- VTP client does not receive or apply information from the server – The first thing to check is whether the trunk link is configured and active between the VTP server and the client. This includes trunk links between any switches between the VTP server and client if the client in consideration is not directly connected.
Second, ensure that the VTP domain and password are correct. You can check the domain name with the show vtp status command and the password with the show vtp password command.
Another important factor is the revision number. If the VTP client is an old switch with pre-existing configuration, then it might have a higher revision than the one being advertised by the server. In such situations, change the domain of the client to something else and then revert it back to the correct domain. This will reset the revision number on the client. The show vtp status command helps to verify the VTP configuration. You can see plenty of examples of the show commands in the theory and lab sections.
- New VTP client caused a change in the VLAN database in the entire network – This can happen only if the client was brought from a lab or another network (using the same domain name and password, if one was set) and had a higher revision number. This can be verified using the show vtp status
- VTP pruning is not working correctly – If there is a VTP transparent switch in between the VTP server and the VTP client, then VTP pruning will not work. Another reason VTP pruning will appear not to be working correctly is the configuration of allowed VLANs on the trunk links. Some VLANs might have been removed manually. This can be verified using the show interface trunk
Mini-Lab – Troubleshooting VTP, VLANS, and Trunking
For this lab, you will have to play along by breaking it before you fix it. Figure 28.2 below shows a very simple network. A host on either end should be in VLAN 10 and connect across a trunk link to the other host. You have been called into a small business to troubleshoot and resolve the issue. In the CCNA exam, you may just have to check configurations and answer questions, or log in to one or more devices and resolve the issue.
FIG 28.2 – Mini-lab: Troubleshooting VTP, VLANs, and trunking
You need to add a faulty configuration to both sides before you fix it:
SwitchA(config)#int f0/10
SwitchA(config-if)#shut
SwitchA(config-if)#vtp domain howtonetwork
Changing VTP domain name from NULL to howtonetwork
SwitchA(config)#vtp password cisco
Setting device VLAN database password to cisco
SwitchA(config)#int f0/1
SwitchA(config-if)#switchport mode trunk
SwitchA(config-if)#switchport trunk allowed vlan 1-9
And you can break a few things on Switch B for good measure:
SwitchB#conf t
SwitchB(config)#int f0/1
SwitchB(config-if)#switchport mode access
SwitchB(config-if)#duplex half
SwitchB(config-if)#int f0/10
SwitchB(config-if)#switchport mode access
SwitchB(config-if)#switchport access vlan 10
% Access VLAN does not exist. Creating vlan 10
SwitchB(config-if)#no shut
SwitchB(config-if)#exit
SwitchB(config)#vtp domain howtonetwork
Domain name already set to howtonetwork.
SwitchB(config)#vtp password hello
Setting device VLAN database password to hello
SwitchB(config)#
Now pretend that you are troubleshooting the network for the first time. Looking at the diagram above, you would expect certain things to be true from what you already know about VLANs, VTP, and trunking, such as:
- The listed ports should be up
- Hosts should be connected to access ports
- Access ports should be in VLAN 10
- Links should be trunking via being set to trunk or via DTP
- VLAN 10 and the native VLAN should pass across the trunk
- VTP domain and password should match
Being methodical about this, you know that if a port is down, nothing interesting will happen. I’ve removed much of the irrelevant output in the commands below to save space:
- Are the relevant interfaces up?
SwitchA#show ip int brief
Interface IP-Address OK? Method Status Protocol
Fa0/1 unassigned YES manual up up
Fa0/10 unassigned YES manual administratively down down
Alarm bells should be ringing because your access port is administratively down. You can easily fix this with the no shut command. You don’t need to reissue the show ip interface command again because you will see the interface come up.
- Is the interface placed into the correct VLAN and set to access?
SwitchA#show int f0/10 switchport
Name: Fa0/10
Switchport: Enabled
Administrative Mode: dynamic auto
Operational Mode: static access
Administrative Trunking Encapsulation: dot1q
Operational Trunking Encapsulation: native
Negotiation of Trunking: On
Access Mode VLAN: 1 (default)
Trunking Native Mode VLAN: 1 (default)
The port is set to access but it’s in VLAN 1. If you wanted to check this another way, you could issue a show vlan brief command. I’ve removed some output to save space:
VLAN Name Status Ports
—- —————————- ——— ——————
1 default active Fa0/2, Fa0/3,
Fa0/4, Fa0/5
This, of course, is easily fixed:
SwitchA(config)#int f0/10
SwitchA(config-if)#switchport access vlan 10
% Access VLAN does not exist. Creating vlan 10
SwitchA(config-if)#end
SwitchA#show vlan brief
VLAN Name Status Ports
—- ————————— ——— ——————-
1 default active Fa0/2, Fa0/3,
Fa0/4, Fa0/5
10 VLAN0010 active Fa0/10
- You can now move on and check the trunking:
SwitchA#show interfaces trunk
Port Mode Encapsulation Status Native
vlan
Fa0/1 on 802.1q trunking 1
Port Vlans allowed on trunk
Fa0/1 1-9
Good news and bad news. The interface is trunking but only VLANs 1 through 9 are permitted. There is a configuration line blocking your VLAN. In the exam, you will probably have only the show run command available as opposed to the show run interface X, for example. I’ve pasted in the relevant show run output below to save space:
interface FastEthernet0/1
switchport trunk allowed vlan 1-9
switchport mode trunk
You will be told in the exam what to allow. You might want to remove the entire configuration line by typing it out again with no in front, or just add VLAN 10.
SwitchA(config)#int f0/1
SwitchA(config-if)#switchport trunk allowed vlan 1-10
SwitchA(config-if)#end
SwitchA#show int trunk
Port Mode Encapsulation Status Native
vlan
Fa0/1 on 802.1q trunking 1
Port Vlans allowed on trunk
Fa0/1 1-10
- Check the VTP. This is the final part you need to check. As you know, both sides need to match the domain and password:
SwitchA#show vtp status
VTP Version: 2
Configuration Revision: 1
Maximum VLANs supported locally: 255
Number of existing VLANs: 6
VTP Operating Mode: Server
VTP Domain Name: howtonetwork
VTP Pruning Mode: Disabled
VTP V2 Mode: Disabled
VTP Traps Generation: Disabled
MD5 digest: 0xAA 0x0E 0xB1 0xB4 0xEE 0x2F 0xAC 0xC5
Configuration last modified by 0.0.0.0 at 3-1-93 04:26:01
Local updater ID is 0.0.0.0 (no valid interface found)
You can’t see the password in the configuration but there is a command you can use to check it:
SwitchA#show vtp password
VTP Password: cisco
So far, you can see that the interfaces are correctly trunking or set to access, and the VLAN is applied to the interface and allowed across the trunk. The VTP configuration is also correct. You can move over to Switch B. I’ll keep this part shorter because you know the drill by now. Interfaces up, correctly set to access/trunk, VLANs exist and permitted, and VTP configuration agrees:
SwitchB#show ip int brief
Interface IP-Address OK? Method Status Protocol
FastEthernet0/1 unassigned YES manual up up
FastEthernet0/10 unassigned YES manual up up
SwitchB#show int f0/10 switchport
Name: Fa0/10
Switchport: Enabled
Administrative Mode: static access
Operational Mode: static access
Administrative Trunking Encapsulation: dot1q
Operational Trunking Encapsulation: native
Negotiation of Trunking: Off
Access Mode VLAN: 10 (VLAN0010)
Trunking Native Mode VLAN: 1 (default)
SwitchB#show interface trunk
SwitchB#
SwitchB(config)#int f0/1
SwitchB(config-if)#switchport mode trunk
%LINEPROTO-5-UPDOWN: Line protocol on Interface FastEthernet0/1, changed state to up
SwitchB#show int trunk
Port Mode Encapsulation Status Native
vlan
Fa0/1 on 802.1q trunking 1
Port Vlans allowed on trunk
Fa0/1 1-1005
SwitchB#show vtp status
VTP Version: 2
Configuration Revision: 1
Maximum VLANs supported locally: 255
Number of existing VLANs: 6
VTP Operating Mode: Server
VTP Domain Name: howtonetwork
VTP Pruning Mode: Disabled
VTP V2 Mode: Disabled
VTP Traps Generation: Disabled
MD5 digest: 0x31 0x48 0xCC 0x20 0x45 0xFB 0x46 0x1F
Configuration last modified by 0.0.0.0 at 3-1-93 03:58:08
Local updater ID is 0.0.0.0 (no valid interface found)
SwitchB#show vtp password
VTP Password: hello
SwitchB(config)#vtp password cisco
Setting device VLAN database password to cisco
You know that the interface was set to half-duplex on Switch B; however, because autoconfiguration is running, Switch A has set itself to the same. If you had hard-set it to full-duplex and 100 Mbps, you would have seen errors. This is a small gotcha because the network will work as is but you want the link to work at full capacity, so set full-duplex on both ends (or at least change it back on Switch B):
SwitchB(config)#int f0/1
SwitchB(config-if)#duplex full
Now, all things being equal, you should be able to ping across the link. The output below shows 192.168.1.1 pinging its neighbor on the other side of VLAN 10:
PC>ping 192.168.1.2
Pinging 192.168.1.2 with 32 bytes of data:
Reply from 192.168.1.2: bytes=32 time=1ms TTL=128
Reply from 192.168.1.2: bytes=32 time=0ms TTL=128
Reply from 192.168.1.2: bytes=32 time=0ms TTL=128
Reply from 192.168.1.2: bytes=32 time=0ms TTL=128
Ping statistics for 192.168.1.2:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 0ms, Maximum = 1ms, Average = 0ms
[END OF MINI-LAB]IP Addressing Troubleshooting
There are a few things to consider when troubleshooting host connectivity at layer 3. Let’s work on the following example:
FIG 28.3 – Troubleshooting host IP addressing issues
Usually, when troubleshooting scenarios that involve verifications only at the IP layer, the recommendation is to start the investigation from one end of the network. Let’s assume that you will start the troubleshooting process from the Internet cloud and work your way to the user’s workstation. The first thing you would do is verify interface Fast Ethernet 0/0 on R1 and ensure that:
- The interface is in up/up status; and
- The interface is properly addressed.
You can check the interface up/up status (as well as the IP address and subnet mask) using the following command:
R1#show interfaces fa0/1
FastEthernet0/1 is up, line protocol is up
Hardware is Gt96k FE, address is c201.0f00.0000 (bia c201.0f00.0000)
Internet address is 192.168.1.1/24
MTU 1500 bytes, BW 10000 Kbit/sec, DLY 1000 usec,
reliability 255/255, txload 1/255, rxload 1/255
[output truncated]To verify correct IP addressing (but without seeing the subnet mask) on the Internet-facing interface, use the following command to display a summary:
R1#sho ip int br
Interface IP-Address OK? Method Status Protocol
FastEthernet0/1 192.168.1.1 YES manual up up
[output omitted]You can also use the show ip interface fa0/1 command to see both pieces of information.
After you make sure that everything is okay at the interface level, you will want to make sure that the router has connectivity to the Internet:
R1#ping 8.8.8.8
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 8.8.8.8, timeout is 2 seconds:
!!!!!
Success rate is 80 percent (4/5), round-trip min/avg/max = 20/29/52 ms
Based on the output above, the ping is successful so you can shift your attention to the LAN-facing part of the network. Start by inspecting the LAN-facing router interface:
R1#sho ip interface fa0/0
FastEthernet0/0 is up, line protocol is up
Internet address is 192.168.1.1/24
Broadcast address is 255.255.255.255
Address determined by setup command
MTU is 1500 bytes
[output omitted]The interface is up and the correct IP address is assigned. One additional thing you might want to check on the router is the DHCP server configuration if the host is configured as a DHCP client. In this case, you need to pay attention to a few parameters:
- Correct network and subnet statement in the DHCP pool
- Correct default gateway statement in the DHCP pool
- Correct DNS server statement in the DHCP pool
- Correctly configured excluded addresses range
Next, check the switch for correct layer 2 configuration (VLAN assignment). The process for this was demonstrated in a previous section, so it won’t be repeated here.
The last element in the troubleshooting process is the user workstation. From an IP connectivity perspective, check the following:
- Correct IP address assignment – from the same range as R1’s Fast Ethernet 0/0 interface
- Correct subnet mask assignment
- Correct default gateway assignment – this should match the address on R1’s interface (192.168.1.1)
- Correct DNS server assignment – this can be manually configured or automatically assigned from the DHCP server
- Make sure that you don’t have other workstations in the network with the same IP address because communication errors might occur if duplicate IP addresses are present in the same VLAN
We have already covered the extended ping command throughout this guide. You can also use the extended option with the traceroute command. Please try it out on some of the multi-router labs you configured.
R1#traceroute
Protocol [ip]:
Target IP address: 192.168.1.2
Source address:
Numeric display [n]:
Timeout in seconds [3]:
Probe count [3]:
Minimum Time to Live [1]:
Maximum Time to Live [30]:
Port Number [33434]:
Loose, Strict, Record, Timestamp, Verbose[none]:
Type escape sequence to abort.
Tracing the route to 192.168.1.2
1 *
192.168.1.2 32 msec *
We also covered troubleshooting IP addressing in the relevant chapter.
Troubleshooting Access Control Lists
You can check which access control lists are configured on your router with the show ip access-lists command:
Router#show ip access-lists
Standard IP access list 10
permit 10.1.1.2
Extended IP access list 100
permit tcp any any eq domain
permit udp any any eq 25
permit tcp any any eq www
You can see an ACL on an interface by issuing the show ip interface [interface type/number] command, including in which direction it has been placed:
Router#show ip interface Serial0/0
Serial0/0 is up, line protocol is up
Internet address is 192.168.1.1/24
Broadcast address is 255.255.255.255
Address determined by setup command
MTU is 1500 bytes
Helper address is not set
Directed broadcast forwarding is disabled
Outgoing access list is not set
Inbound access list is 10
You can also type show run interface X. Note that this command may not be available on your router depending on the IOS release you are using. It probably won’t work on Packet Tracer or in the exam.
Router#show run interface Serial0/0
interface Serial0/0
ip address 192.168.1.1 255.255.255.0
ip access-group 10 in
end
It is possible to run a debug on ACLs. This is outside the scope of the CCNA syllabus (please check the latest syllabus). You would specify the traffic you want to monitor with an ACL, and then enter the command below in global configuration mode:
Router#debug ip packet 101 [detail] – Where 101 is your ACL
detail – gives more detailed information in the debug
|
The debug ip packet command in general will show only process-switched traffic, so it is not much use if CEF (Cisco Express Forwarding) is turned on. |
You can see actual statistics per ACL, including how many matches there have been, with the show access-lists X command:
Router#show access-lists 100
Extended IP access list 100
permit tcp host 192.168.1.1 any established (308 matches)
permit udp host 192.168.1.2 any eq domain (12 matches)
permit icmp host 192.168.1.3 any
permit tcp host 192.168.4.5 host 10.0.0.1 gt 1023
permit tcp host 192.168.4.8 host 10.0.0.2 eq smtp (4 matches)
You can clear the counters on an ACL with the clear access-list counters command:
Router#clear access-list counters ?
[0-199] Access list number
WORD Access list name
[cr]
Router#clear access-list counters 100
Router# show access-lists 100
Extended IP access list 100
permit tcp host 192.168.1.1 any established
permit udp host 192.168.1.2 any eq domain
permit icmp host 192.168.1.3 any
permit tcp host 192.168.4.5 host 10.0.0.1 gt 1023
permit tcp host 192.168.4.8 host 10.0.0.2 eq smtp
The majority of ACL issues crop up either from a configuration error, such as putting the wrong network address or wildcard mask in, or forgetting to apply the ACL to an interface. Remember that if you add the ACL to an internal interface, it will only inspect traffic when it hits that interface. If, for example, you are trying to block Telnet access coming in from the Internet, your internal interface ACL won’t block it.
Also, remember that ACLs are processed top-down; if there is a match higher up, the more specific entry further down won’t be reached. The router will NOT filter self-generated traffic.
Common RIP Troubleshooting Issues
The vast majority of RIP troubleshooting issues will be configuration mistakes. RIP implements split horizon, route poisoning, and holddown timers, which were covered earlier. These problems will usually be taken care of during the design phase; if not, they will become apparent during the initial configuration.
One common issue with RIP is that the incorrect network is advertised or not advertised at all. Even directly connected networks must be advertised in order for RIP to work. Many junior engineers just configure the edge networks and wonder why the routing tables are empty.
In Figure 28.4 below, the 10 network needs to be advertised on both routers, as well as the LAN networks, so that the routes will show up in the routing table, despite the fact that the 10 network will show as “Connected” and not as a RIP route in the routing table:
FIG 28.4 – Common RIP troubleshooting issue
Other common issues are that the no auto-summary command is missing or is configured on the wrong router. You may find that the passive-interface command has been added somewhere that it should not have been. Remember that you should avoid discontiguous subnets when designing your network to prevent having to do extra configurations to deal with the issue.
Your friends will be the show ip route and show ip protocols commands. It is not likely that the debug commands will work in the exam. Also, try the show ip rip database command. Remember that we covered RIP troubleshooting in the RIP section also.
End of Chapter Questions
Please visit https://www.howtonetwork.com/ccnasimplified to take the free Chapter 28 exam.
