We covered network troubleshooting simple issues in the previous chapter. As you can imagine, this troubleshooting section will be more complex because of the technologies involved. Initial configurations are more difficult, the commands are more granular, and you will find that your network could partially work or intermittently work, which will make your task all the more difficult.
In the exam, you can expect your troubleshooting to fall into the categories below:
- General theory questions about the likely cause of an issue
- A network diagram and questions about the likely cause of an issue
- Logging into equipment and using show commands to determine the cause of an issue
- Having to log in to a network, diagnose, and resolve configuration problems
The last issue will be the most difficult because you did not complete the initial configurations, so you will have to quickly familiarize yourself with the topology, use the relevant show commands to check outputs and configurations, and quickly resolve the issue so you can move on to the next question.
Randomly typing show commands or typing show run on every device hoping to stumble on the root cause is a recipe for disaster. Bear in mind that there may well be multiple issues on multiple OSI layers, such as physical, data link, and network on both routers and switches, so you will have your work cut out for you!
The secret sauce is really understanding the technology and doing lots of hands-on labs throughout this guide. You will make mistakes as you go along and discover which show command will reveal the root cause of the issue to you. Once you get comfortable with my labs, start making your own up using different IP addresses and interfaces. Like any master, you get there by doing tasks over and over until they become second nature. If you go into the exam not knowing which command will show you the Spanning Tree root (for example), then failure is inevitable because the show run command won’t tell you.
Troubleshooting STP
This section touches on common STP problems and ways to troubleshoot them. The steps given here apply to both 802.1D and 802.1W running different STP process on each VLAN.
STP is a very maintenance-free protocol and generally does not require troubleshooting. STP will mostly have the following problems:
- Incorrect root bridge
- Incorrect root port
- Incorrect designated port
Let’s look at each of the problems and ways to troubleshoot them.
Incorrect Root Bridge
The root bridge is selected based on the bridge ID (BID), which consists of priority and the base MAC address of the switch. The show spanning-tree vlan [vlan#] command will show the current root bridge. Note the MAC address and the priority of the root bridge and compare it with those of the switch that you want to make the root bridge. Decreasing the priority of the correct switch should resolve the problem. This can be done using the spanning-tree vlan [vlan#] priority [priority] or the spanning-tree [vlan #] root primary command.
You can glean specific root bridge information with the show spanning-tree bridge command, which displays the local bridge information. This isn’t likely to work on the exam simulator though.
ALS1#show spanning-tree bridge
Hello Max Fwd
Vlan Bridge ID Time Age Dly Protocol
———— —————————— —– —- —- ——–
VLAN0001 32769 (32768, 1) 0011.9247.db00 2 20 15 ieee
Incorrect Root Port
The root port is the fastest path from a switch to the root bridge. The cost is the cumulative cost of all the links in the path, with each port adding its cost to the VLAN. For example, if there are two 100 Mbps links between a switch and a root bridge, then the cost is 38. The show spanning-tree vlan [vlan #] command will show the current root port and its cost. Compare that with the cost of the desired path. The cost of the desired path can be changed using the spanning-tree cost [cost] interface command.
In Figure 29.1 below, Switch 0 is the root bridge for VLAN 20 and all switches are connected by Fast Ethernet cables. You would expect Switch 2 to show a cost of 38. Fa0/1 points to the root bridge, so it should be the root port and Fa0/2 points away, so it should be the designated port.
FIG 29.1 – Incorrect root port example
Switch2#show spanning-tree vlan 20
VLAN0020
Spanning tree enabled protocol ieee
Root ID Priority 24596
Address 0003.E469.8D9B
Cost 38
Port 1(FastEthernet0/1)
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Bridge ID Priority 32788 (priority 32768 sys-id-ext 20)
Address 0002.4A05.849B
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Aging Time 20
Interface Role Sts Cost Prio.Nbr Type
—————- —- — ——— ——– ———————–
Fa0/1 Root FWD 19 128.1 P2p
Fa0/2 Desg FWD 19 128.2 P2p
Switch 3 should add another 19 to the cost and have one root port pointing toward Switch 2:
Switch3#show spanning-tree vlan 20
VLAN0020
Spanning tree enabled protocol ieee
Root ID Priority 24596
Address 0003.E469.8D9B
Cost 57
Port 2(FastEthernet0/2)
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Bridge ID Priority 32788 (priority 32768 sys-id-ext 20)
Address 0009.7CD4.6A95
Hello Time 2 sec Max Age 20 sec Forward Delay 15 sec
Aging Time 20
Interface Role Sts Cost Prio.Nbr Type
—————- —- — ——— ——– ———————–
Fa0/2 Root FWD 19 128.2 P2p
If you wanted to alter this behavior, you could change the cost directly on a port with the above stated command.
Incorrect Designated Port
The designated port is the lowest cost port connecting a network segment to the rest of the network. The designated port cost can be seen and changed using the show spanning-tree vlan [vlan#] command and the spanning-tree cost [cost] command.
A summary of what you should see with a correctly configured Spanning Tree domain is shown in Figure 29.2 below. Make sure that you work out why each interface is in the relevant state and why the root bridge has become the root.
FIG 29.2 – STP operating correctly
To troubleshoot STP and VLAN issues, a very useful command to display a summary of interface settings, status, speed, and duplex settings is show interfaces status. As always, please type this out during any switching labs so the command is burned into your brain.
ALS1#show interfaces status
Port Name Status Vlan Duplex Speed Type
Fa0/1 notconnect 1 auto auto 10/100BaseTX
Fa0/2 connected 1 a-full a-100 10/100BaseTX
Fa0/3 connected 1 a-full a-100 10/100BaseTX
Fa0/4 connected 1 a-full a-100 10/100BaseTX
Fa0/5 notconnect 1 auto auto 10/100BaseTX
Fa0/6 notconnect 1 auto auto 10/100BaseTX
Fa0/7 connected 1 a-full a-100 10/100BaseTX
Fa0/8 connected 1 a-full a-100 10/100BaseTX
[output truncated]Troubleshooting InterVLAN Routing Problems
InterVLAN routing is mentioned in exam syllabus. We covered the theory and configuration earlier. When troubleshooting interVLAN routing, there are a few standard things you should check to ensure connectivity between hosts connected in different VLANs.
One of the first things to verify is layer 1 connectivity. Make sure that:
- User workstations are connected to the correct switch ports and the ports are active. Verify that the link is up, both on the workstations and on the switches.
- The links between the switches and the router(s) are functional and in up/up mode.
The next thing to do is verify that the user-facing switch ports are assigned to the correct VLANs. You can do this using the show vlan command on the switches:
Switch#show vlan
VLAN Name Status Ports
—- ——————————– ——— ——————–
2 VLAN2 active Fa1/1, Fa1/2
3 VLAN3 active Fa1/3, Fa1/4
It would be a good idea to also verify the trunk links between switches, and between switches and the routers that perform the interVLAN routing to make sure that all relevant VLANs are carried on all trunk links up to the routers. You should also check the native VLAN match on both ends of the trunk. If it doesn’t match, untagged traffic will not be transmitted as expected. You can check this using the show interfaces trunk command:
Switch#show interfaces trunk
Port Mode Encapsulation Status Native vlan
Fa1/11 on 802.1q trunking 1
Fa1/12 on 802.1q trunking 1
Port Vlans allowed on trunk
Fa1/11 1,3,4,5
Fa1/12 1,3,4,5
Port Vlans allowed and active in management domain
Fa1/11 1,3,4,5
Fa1/12 1,3,4,5
Port Vlans in spanning tree forwarding state and not pruned
Fa1/11 1,3,4,5
Fa1/12 1,3,4,5
If the trunk links are not forming, you have to check the encapsulation configured on them. A link configured as ISL on one end and dot1Q on the other link will not work as expected (this is not an issue for the Cisco 2960 Switch model). The correct encapsulation can be analyzed with the show interfaces trunk command (see the output above). Another useful piece of information this command provides is the trunk mode (which is on in the output above).
Moving up to layer 3 verification, you need to make sure that correct IP addresses are configured on:
- User workstations
- Router subinterfaces or layer 3 switch SVIs
Each VLAN is usually associated with a dedicated IP subnet. You need to verify that all workstations and interfaces in a particular VLAN have a correct IP address and subnet mask configuration. You can check the IP addressing on the router subinterfaces using the following command:
R1#show ip interface brief
Interface IP-Address OK? Method Status Protocol
FastEthernet0/0 unassigned YES unset up up
FastEthernet0/0.2 192.168.2.1 YES manual up up
FastEthernet0/0.3 192.168.3.1 YES manual up up
If you want to see the configured subnet, then, of course, use the show interface X command.
If all layer 1, layer 2, and layer 3 verifications successfully pass, you should have layer 3 connectivity between hosts in different VLANs. You can test this simply with a standard ping between workstations.
Troubleshooting Routing Issues
If you have access to network diagrams and documentation (which you will in the exam), you should be able to look at it and know what the configuration commands should be. For example, if you see a router with an OSPF area 0 interface showing IP address 192.168.1.1/30, you would expect to see the following configuration on the interface:
R1#show int f0/0
FastEthernet0/0 is up, line protocol is up
Hardware is Gt96k FE, address is c200.0700.0000 (bia c200.0700.0000)
Internet address is 192.168.1.1/30
If it is anything but this, you will have an issue. You would also expect the interface to be up physically at layer 2. If it isn’t, then there will likely be an issue with the clock rate or the no shut command having not been applied (clock rates are only a Serial line issue for DCE).
R1#show ip interface brief
Interface IP-Address OK? Method Status Protocol
FastEthernet0/0 192.168.1.1 YES manual up up
If this is all correct, you would expect to see the route correctly advertised in the configuration.
R1#show ip protocols
Routing Protocol is “ospf 1”
Outgoing update filter list for all interfaces is not set
Incoming update filter list for all interfaces is not set
Router ID 192.168.1.1
Number of areas in this router is 1. 1 normal 0 stub 0 nssa
Maximum path: 4
Routing for Networks:
192.168.1.0 0.0.0.255 area 0
However, there is an issue: the mask is /30 on the interface but the configuration is for a mask of /24. This will cause problems, and you need to resolve this by removing the incorrect configuration and entering the correct one. If the interface had been down, you could have used other show commands to determine the cause of the issue. Once you believe that you have fixed the issue, you should check on a neighbor router to ensure that the routing table is showing the correct network.
As I mentioned, if you are randomly typing various show commands, you will either never find the cause of the issue above or it will take you so long that you run out of time in the exam and can’t complete the other questions and labs.
There are only a few things you would find not working in the exam, and for routing protocol challenge labs these include:
- No clock rate on the DCE interface
- Interface hasn’t had the no shut command applied
- Incorrect IP address or subnet mask on the interface
- Incorrect network advertised by the routing protocol
- Incorrect area or ASN configured for the network
- Auto summary turned off or on
- Access list blocking the routing protocol
The last entry is less likely in a routing troubleshooting challenge lab but it’s worth bearing in mind.
We will cover the generic commands and then for each protocol, the specific commands. On a live network, you may use debug commands but the emulator in the exam will likely not support this. I always use the commands below until I find the cause of the problem:
- show ip route – on a neighbor router to check that the route is there
- show ip interface brief – is the interface up/up?
- show controllers s0/0 – which is the DCE interface/is there a clock rate added?
- show interface s0/0 – is the IP address/subnet mask and layer 2 protocol correct?
- show ip protocols – route correctly advertised/passive interface/auto summary?
You would use the commands above no matter what routing protocol you are troubleshooting. These should be more than enough for CCNA-level issues, but we will dig into protocol-specific troubleshooting below. A command that can come in handy is clear ip route *. This clears the routing table and forces the routing protocol to exchange routing tables.
When troubleshooting routing issues, the general recommendation is to follow the debugging process specific to each protocol (EIGRP, OSPF, etc.). We covered debug commands for these in the relevant chapters.
When using IPv6, you have to make sure that you configure the device for IPv6 routing. This is disabled by default on most devices.
Router(config)#ipv6 unicast-routing
The next step is to check whether the routing table is showing the correct entries learned by the specific OSPF protocol you configured and that the device is learning the necessary networks as advertised by the routing protocol. This is accomplished using the show ip route command:
R1#show ip route
[output truncated]Gateway of last resort is not set
10.0.0.0/24 is subnetted, 2 subnets
C 10.0.12.0 is directly connected, FastEthernet0/0
O IA 10.0.23.0 [110/20] via 10.0.12.2, 00:15:28, FastEthernet0/0
192.168.1.0/32 is subnetted, 1 subnets
C 192.168.1.1 is directly connected, Loopback0
In the output above, you can see that the router is learning about network 10.0.23.0 via OSPF. This means that the routing protocol is functioning, but if you want to make sure that it’s functioning as it should (e.g., maybe you should have received multiple routes via OSPF or with other metrics), you should troubleshoot further, as explained in the next sections.
When learning the same prefix via multiple routing protocols, you might also need to check whether the correct route is installed into the routing table based on the administrative distance of each protocol. For example, if you learn the same route via EIGRP and OSPF, only the EIGRP route will be installed in the routing table due to its lower AD value.
A truncated show ip route output with RIP advertising a remote network of 172.16.0.0 is shown below:
R 172.16.0.0/16 [120/1] via 192.168.1.2, 00:00:06, F0/0
C 10.0.0.0/8 is directly connected, Loopback0
C 192.168.1.0/24 is directly connected, F0/0
I added EIGRP on the remote router and even though RIP was still active, the route was replaced in the routing table due to the preferred AD:
D 172.16.0.0/16 [90/409600] via 192.168.1.2,00:00:04, F0/0
C 10.0.0.0/8 is directly connected, Loopback0
C 192.168.1.0/24 is directly connected, F0/0
Troubleshooting OSPF
There are several commands you can use to verify and troubleshoot OSPF:
- show ip protocols – shows all routing protocol information on the router
- show ip route – shows the routing table
- show ip ospf interface – shows which interfaces are in which OSPF areas
- show ip ospf – shows the link state update interval and SPF details
- show ip ospf neighbor detail – shows a list of OSPF neighbors in detail
- show ip ospf database – shows the OSPF topological database
Of course, there are several OSPF debug commands available depending on which issue you are troubleshooting:
R2#debug ip ospf ?
adj OSPF adjacency events
database-timer OSPF database timer
events OSPF events
hello OSPF Hello events
lsa-generation OSPF lsa generation
packet OSPF packets
spf OSPF spf
tree OSPF database tree
[output truncated]You can test these commands when you configure the OSPF labs in the Labs section.
Before you dive into troubleshooting, ensure that all the interfaces are up at layers 1 and 2. Also, check IP connectivity by pinging across all point-to-point links. Ensure that all subnet masks are correct and any layer 2 protocols such as PPP are operating correctly.
Bear in mind that access control lists will block any routing traffic unless specifically permitted. Another gotcha is incomplete IPv6 parts in your configurations. For example, the command below will enable OSPF on the interface:
ip ospf 1 area 0
For OSPFv3, you need to type (not in the syllabus):
ipv6 ospf 1 area 0
I’ve made this mistake myself when hurriedly typing out configuration commands, and I’ve seen others do it and then spend a long time troubleshooting the issue.
When troubleshooting an OSPF implementation, you can approach the process in two ways:
- Start by examining the OSPF adjacencies, LSDB, and then correct route installation
- Start by examining whether the routes are present in the routing table, then the LSDB, and, finally, OSPF adjacencies
Either way, there are a few specific things that you need to verify depending on the topology. One of the first things to check is whether the OSPF neighbor adjacencies have been formed. If logging is enabled on your device, you should see a message similar to this one when two OSPF neighbors form an adjacency:
*Mar 1 00:03:24.687: %OSPF-5-ADJCHG: Process 1, Nbr 2.2.2.2 on FastEthernet0/0 from LOADING to FULL, Loading Done
This is a clear sign that the adjacency has been formed. If you want to dig a bit deeper into this, you can use one of the following commands to inspect neighbor relationships:
R2#show ip protocols
Routing Protocol is ospf 1
Outgoing update filter list for all interfaces is not set
Incoming update filter list for all interfaces is not set
Router ID 2.2.2.2
It is an area border router
Number of areas in this router is 2. 2 normal 0 stub 0 nssa
Maximum path: 4
Routing for Networks:
Routing on Interfaces Configured Explicitly (Area 0):
FastEthernet0/0
Routing on Interfaces Configured Explicitly (Area 23):
FastEthernet0/1
Reference bandwidth unit is 100 mbps
Routing Information Sources:
Gateway Distance Last Update
3.3.3.3 110 00:11:38
1.1.1.1 110 00:12:22
Distance: (default is 110)
R2#show ip ospf ?
[1-65535] Process ID number
border-routers Border and Boundary Router Information
database Database summary
flood-list Link state flood list
interface Interface information
max-metric Max-metric origination information
mpls MPLS related information
neighbor Neighbor list
request-list Link state request list
retransmission-list Link state retransmission list
rib Routing Information Base (RIB)
sham-links Sham link information
statistics Various OSPF Statistics
summary-address Summary-address redistribution Information
timers OSPF timers information
traffic Traffic related statistics
virtual-links Virtual link information
| Output modifiers
[cr]
R2#show ip ospf neighbor
Neighbor ID Pri State Dead Time Address Interface
1.1.1.1 1 FULL/BDR 00:00:37 10.0.12.1 F0/0
3.3.3.3 1 FULL/DR 00:00:39 10.0.23.3 F0/1
The show ip ospf neighbor command gives you an explicit summary of the neighbor relationships. In the output above, you can see that R2 has two neighbors, and you can also inspect the state of those neighbors, which is useful in broadcast networks in order to see which router has been selected as the DR.
If the adjacencies do not form, you might have a timer mismatch between the two neighbors. Remember, the default OSPF timers are 10/40 in broadcast networks. If you don’t see OSPF adjacencies forming, one of the first debug commands you can issue is the debug ip ospf hello, as shown in the output below:
R1#debug ip ospf hello
OSPF hello events debugging is on
*Mar 1 00:23:26.887: OSPF: Send Hello to 224.0.0.5 area 0 on FastEthernet0/0 from 10.0.12.1
R1#
*Mar 1 00:23:31.055: OSPF: Rcv Hello from 2.2.2.2 area 0 from FastEthernet0/0 10.0.12.2
*Mar 1 00:23:31.059: OSPF: Mismatched Hello parameters from 10.0.12.2
*Mar 1 00:23:31.059: OSPF: Dead R 40 C 7, Hello R 10 C 5 Mask R 255.255.255.0 C 255.255.255.0
*Mar 1 00:23:31.867: OSPF: Send Hello to 224.0.0.5 area 0 on FastEthernet0/0 from 10.0.12.1
In the output above, you are informed about a timer mismatch with neighbor 10.0.12.2, and you can see that the timers received from the neighbors (labeled R in the output) are 10 for Hello and 40 for Dead, while the locally configured timers (labeled C in the output) are 5 for Hello and 7 for Dead.
You can fix this by configuring timers with the same value at both ends:
R1(config)#int fa0/0
R1(config-if)# ip ospf hello-interval 10
R1(config-if)# ip ospf dead-interval 40
R1(config-if)#
*Mar 1 00:26:56.563: %OSPF-5-ADJCHG: Process 1, Nbr 2.2.2.2 on FastEthernet0/0 from LOADING to FULL, Loading Done
The next thing you want to check if adjacencies are not formed is the OSPF area ID configured on each side of the connection. If the area ID differs, the adjacency will not form. In this case, a warning message will appear directly on the console, without any debug command activated:
*Mar 1 00:43:09.931: %OSPF-4-ERRRCV: Received invalid packet: mismatch area ID, from backbone area must be virtual-link but not found from 10.0.12.2, FastEthernet0/0
You can also see this by debugging the OSPF adjacencies:
R2#deb ip ospf adj
OSPF adjacency events debugging is on
*Mar 1 00:46:07.083: OSPF: Rcv pkt from 10.0.12.1, FastEthernet0/0, area 0.0.0.0
mismatch area 0.0.0.1 in the header
To correct this, simply configure the same area ID on both ends of the peers:
R1#conf t
Enter configuration commands, one per line. End with CNTL/Z.
R1(config)#int fa0/0
R1(config-if)#no ip ospf 1 area 1
R1(config-if)#ip ospf 1 area 0
*Mar 1 00:46:32.019: %OSPF-5-ADJCHG: Process 1, Nbr 2.2.2.2 on FastEthernet0/0 from LOADING to FULL, Loading Done
Another very commonly seen reason for OSPF adjacencies not forming is an MTU mismatch between the OSPF peers. When this happens, the OSPF adjacency is put in DOWN mode:
*Mar 1 00:58:13.975: %OSPF-5-ADJCHG: Process 1, Nbr 2.2.2.2 on FastEthernet0/0 from EXSTART to DOWN, Neighbor Down: Too many retransmissions
The reason for this is that during the OSPF adjacency building process, the different MTUs do not allow OSPF DBD packets to be exchanged. The routers try to exchange DBDs over a period of time and if this does not work, the neighbor is put in DOWN mode. You can see the warning about mismatched MTUs in the debugging output below:
R2#deb ip ospf adj
OSPF adjacency events debugging is on
*Mar 1 00:56:43.155: OSPF: Send DBD to 1.1.1.1 on FastEthernet0/0 seq 0x12D9 opt 0x52 flag 0x7 len 32
*Mar 1 00:56:43.155: OSPF: Retransmitting DBD to 1.1.1.1 on FastEthernet0/0 [6]
*Mar 1 00:56:43.859: OSPF: Rcv DBD from 1.1.1.1 on FastEthernet0/0 seq 0x1D31 opt 0x52 flag 0x7 len 32 mtu 1000 state EXSTART
*Mar 1 00:56:43.859: OSPF: Nbr 1.1.1.1 has smaller interface MTU
*Mar 1 00:56:43.863: OSPF: First DBD and we are not SLAVE
In order to see which MTU values are configured at both ends, you can use the show interface command:
R1#show int fa0/0
FastEthernet0/0 is up, line protocol is up
Hardware is Gt96k FE, address is c201.2f70.0000 (bia c201.2f70.0000)
Internet address is 10.0.12.1/24
MTU 1000 bytes, BW 10000 Kbit/sec, DLY 1000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, Loopback not set
[output truncated]R2#show interface fa0/0
FastEthernet0/0 is up, line protocol is up
Hardware is Gt96k FE, address is c202.09d0.0000 (bia c202.09d0.0000)
Internet address is 10.0.12.2/24
MTU 1500 bytes, BW 10000 Kbit/sec, DLY 1000 usec,
reliability 255/255, txload 1/255, rxload 1/255
Encapsulation ARPA, Loopback not set
[output truncated]The MTU is the maximum packet size that the interface can transmit in one instance, and it differs per interface type and protocol (1500 for Ethernet). You can clearly see that R1’s interface has an MTU of 1000 bytes, while R2’s interface MTU is 1500. Let’s configure both of them to 1500 to form an OSPF adjacency:
R1#conf t
Enter configuration commands, one per line. End with CNTL/Z.
R1(config)#int fa0/0
R1(config-if)#mtu 1500
*Mar 1 01:02:27.491: %OSPF-5-ADJCHG: Process 1, Nbr 2.2.2.2 on FastEthernet0/0 from LOADING to FULL, Loading Done
The MTU has to match on both sides in order for an OSPF adjacency to form. If you issued the debug ip ospf adj command, you would see something like the following:
OSPF: Nbr 192.168.100.2 has larger interface MTU
Another “fix” is to issue the command below, but beware because this can lead to other issues because there will still be the underlying MTU mismatch:
R1(config)#interface Gig1/0
R1(config-if)#ip ospf mtu-ignore
R1(config-if)#end
Another reason for OSPF adjacencies not forming might be an OSPF network type mismatch on the connecting interfaces. OSPF requires the same network type to be configured on the peers: broadcast, non-broadcast, point-to-multipoint, or point-to-point.
If there is a network type mismatch, the adjacency is down and the neighbor does not come up in the show ip ospf neighbor command:
R1#show ip ospf nei
You can check the configured network type using the following command:
R1#show ip ospf int fa0/0
FastEthernet0/0 is up, line protocol is up
Internet Address 10.0.12.1/24, Area 0
Process ID 1, Router ID 1.1.1.1, Network Type NON_BROADCAST, Cost: 10
Enabled by interface config, including secondary ip addresses
Transmit Delay is 1 sec, State WAITING, Priority 1
No designated router on this network
No backup designated router on this network
Timer intervals configured,Hello 30,Dead 120, Wait 120, Retransmit 5
[output truncated]R2#show ip ospf int fa0/0
FastEthernet0/0 is up, line protocol is up
Internet Address 10.0.12.2/24, Area 0
Process ID 1, Router ID 2.2.2.2, Network Type BROADCAST, Cost: 10
Enabled by interface config, including secondary ip addresses
Transmit Delay is 1 sec, State DR, Priority 1
Designated Router [ID] 2.2.2.2, Interface address 10.0.12.2
No backup designated router on this network
Timer intervals configured, Hello 10, Dead 40, Wait 40, Retransmit 5
[output truncated]In the output above, R1’s interface is configured as non-broadcast and R2’s interface is configured as broadcast from an OSPF perspective. This prevents the neighbor adjacencies from forming. In order to fix this, let’s configure both interfaces as broadcast (in the exam you will be told or will see in a network diagram which it should be):
R1(config)#int fa0/0
R1(config-if)#ip ospf network broadcast
*Mar 1 01:15:45.511: %OSPF-5-ADJCHG: Process 1, Nbr 2.2.2.2 on FastEthernet0/0 from LOADING to FULL, Loading Done
To check the neighbor state on an interface, the simplest command to use is show ip ospf neighbor:
R2#show ip ospf neighbor
Neighbor ID Pri State Dead Time Address Interface
1.1.1.1 1 FULL/BDR 00:00:34 10.0.12.1 F0/0
3.3.3.3 1 FULL/DR 00:00:36 10.0.23.3 F0/1
In the output above, you can see that both adjacencies are in a FULL state. The neighbor on interface Fast Ethernet 0/0 acts as the BDR (which means R2 is the DR on that link, if the link is broadcast), and the neighbor on interface Fast Ethernet 0/1 acts as the DR. You can see that R2 is the DR on the link with R1 by also examining the output of the following command:
R2#sho ip ospf int fa0/0
FastEthernet0/0 is up, line protocol is up
Internet Address 10.0.12.2/24, Area 0
Process ID 1, Router ID 2.2.2.2, Network Type BROADCAST, Cost: 10
Enabled by interface config, including secondary ip addresses
Transmit Delay is 1 sec, State DR, Priority 1
Designated Router (ID) 2.2.2.2, Interface address 10.0.12.2
Backup Designated router (ID) 1.1.1.1, Interface address 10.0.12.1
Timer intervals configured,Hello 10,Dead 40, Wait 40, Retransmit 5
[output truncated]Let’s see what happens on a point-to-point link. You will have to use a pipe to drill down to the interface type:
R1#show ip ospf int fa0/0 | i Type
Process ID 1,Router ID 1.1.1.1, Network Type POINT_TO_POINT,Cost: 10
R2#show ip ospf int fa0/0 | i Type
Process ID 1,Router ID 2.2.2.2, Network Type POINT_TO_POINT,Cost: 10
R1#sho ip ospf nei
Neighbor ID Pri State Dead Time Address Interface
2.2.2.2 0 FULL/ – 00:00:32 10.0.12.2 F0/0
R2#show ip ospf neighbor
Neighbor ID Pri State Dead Time Address Interface
1.1.1.1 0 FULL/ – 00:00:38 10.0.12.1 F0/0
3.3.3.3 1 FULL/DR 00:00:39 10.0.23.3 F0/1
In the output above, you can see that the adjacency on the point-to-point link is in a FULL state but you have no DR or BDR, as expected, because they aren’t needed on this type of network.
As a last step, if the expected routes are still not installed in the routing table, you should check the OSPF link state database (LSDB) to analyze which LSAs (link state advertisements) are received and from which neighbors. You would do this using the following command:
R2#show ip ospf database
OSPF Router with ID (2.2.2.2) (Process ID 1)
Router Link States (Area 0)
Link ID ADV Router Age Seq# Checksum Link count
1.1.1.1 1.1.1.1 725 0x80000011 0x00CBF6 2
2.2.2.2 2.2.2.2 724 0x8000000D 0x00734E 2
Summary Net Link States (Area 0)
Link ID ADV Router Age Seq# Checksum
10.0.23.0 2.2.2.2 1572 0x80000003 0x002DDA
192.168.1.1 1.1.1.1 386 0x80000005 0x00A622
192.168.3.3 2.2.2.2 1572 0x80000003 0x00C6F1
Router Link States (Area 23)
Link ID ADV Router Age Seq# Checksum Link count
2.2.2.2 2.2.2.2 1572 0x80000004 0x00DDF0 1
3.3.3.3 3.3.3.3 1583 0x80000006 0x00AD96 2
Net Link States (Area 23)
Link ID ADV Router Age Seq# Checksum
10.0.23.3 3.3.3.3 1583 0x80000003 0x006D86
Summary Net Link States (Area 23)
Link ID ADV Router Age Seq# Checksum
10.0.12.0 2.2.2.2 1574 0x80000003 0x00A66C
192.168.1.1 2.2.2.2 724 0x80000001 0x00F4C9
Don’t be put off by most of the commands. At the CCNA level, Cisco is more likely to try to trick you by adding either the wrong area to the configuration or advertising the wrong network/wildcard. Trying to catch you out with hard-to-find timer mismatches is very unlikely as a hands-on troubleshooting lab.
Mini-Lab – Troubleshooting OSPF
Although I’ve already mentioned it, please do double-check that you are advertising the correct OSPF network. OSPF will only do what you ask it to, unlike EIGRP, which can in some circumstances work out what you intended to configure. Take the interface below, for example, in Figure 29.3, which you can easily configure:
FIG 29.3 – Mini-lab: Troubleshooting OSPF
If you add the configuration below for OSPF, you might expect the neighbors to form an OSPF relationship and exchange LSAs, but this doesn’t happen.
R1(config-router)#network 192.168.1.0 0.0.0.3 area 0
No neighbor relationship formed and if you started debugging the OSPF process, you would see no outputs. Bear in mind that for the exam, debug commands will probably not be available.
R1#debug ip ospf hello
OSPF Hello events debugging is on
R1#debug ip ospf adj
OSPF adjacency events debugging is on
R1#debug ip ospf packet
OSPF packet debugging is on
There is no OSPF neighbor present on the link, in fact.
R1#show ip ospf neighbor
R1#
But you have configured OSPF to advertise the hosts within the 192.168.1.0/30 network, which are (if you recall your subnetting) 192.168.1.1 and 192.168.1.2.
R1#show ip protocols
Routing Protocol is “ospf 1”
Outgoing update filter list for all interfaces is not set
Incoming update filter list for all interfaces is not set
Router ID 192.168.1.1
Number of areas in this router is 1. 1 normal 0 stub 0 nssa
Maximum path: 4
Routing for Networks:
192.168.1.0 0.0.0.3 area 0
Reference bandwidth unit is 100 mbps
Routing Information Sources:
Gateway Distance Last Update
192.168.1.1 110 00:10:27
192.168.1.2 110 00:10:03
Distance: (default is 110)
You can fix this by configuring OSPF to advertise the correct network on both sides (and removing the incorrect configuration).
R2(config-router)#no network 192.168.1.0 0.0.0.3 area 0
R2(config-router)#network 192.168.1.4 0.0.0.3 area 0
*Mar 1 00:27:24.755: %OSPF-5-ADJCHG: Process 1, Nbr 192.168.1.1 on Serial0/0 from LOADING to FULL, Loading Done
R2#show ip ospf nei
Neighbor ID Pri State Dead Time Address Interface
192.168.1.1 0 FULL/ – 00:00:32 192.168.1.5 Serial0/0
R2#
[END OF MINI-LAB]Troubleshooting EIGRP
As with OSPF, the first place to start when troubleshooting an EIGRP problem is ensuring that layers 1 and 2 are operating correctly. Also, ensure that the IP addresses and subnets have all been entered correctly and that neighbors are in the same subnet. You also need to check for any ACLs that could be blocking EIGRP traffic. Once this is all done, the first step in troubleshooting EIGRP is analyzing the output of the show ip protocols command:
R2#show ip protocols
Routing Protocol is eigrp 100
Outgoing update filter list for all interfaces is not set
Incoming update filter list for all interfaces is not set
Default networks flagged in outgoing updates
Default networks accepted from incoming updates
EIGRP metric weight K1=1, K2=0, K3=1, K4=0, K5=0
EIGRP maximum hopcount 100
EIGRP maximum metric variance 1
Redistributing: eigrp 100
EIGRP NSF-aware route hold timer is 240s
Automatic network summarization is in effect
Maximum path: 4
Routing for Networks:
10.0.12.0/24
10.0.23.0/24
Routing Information Sources:
Gateway Distance Last Update
10.0.12.1 90 00:01:17
10.0.23.3 90 00:00:56
Distance: internal 90 external 170
In the output above, you can see the following things:
- R2 is part of EIGRP 100
- R2 is advertising two networks in EIGRP: 10.0.12.0/24 and 10.0.23.0/24
- R2 has two EIGRP neighbors: 10.0.12.1 and 10.0.23.3
You can illustrate what you now know (after using some of the show commands) with Figure 29.4 below. Other information isn’t relevant for this example.
FIG 29.4 – Troubleshooting EIGRP
You can also see the EIGRP neighbor adjacencies using the show ip eigrp neighbors command. I’ll show you all the options below first:
R2#show ip eigrp ?
[1-65535] Autonomous System
accounting IP-EIGRP accounting
interfaces IP-EIGRP interfaces
neighbors IP-EIGRP neighbors
topology IP-EIGRP Topology Table
traffic IP-EIGRP Traffic Statistics
vrf Select a VPN Routing/Forwarding instance
R2#show ip eigrp neighbors
IP-EIGRP neighbors for process 100
H Address Interface Hold Uptime SRTT RTO Q Seq
(sec) (ms) Cnt Num
1 10.0.23.3 Fa0/1 10 00:05:23 102 612 0 5
0 10.0.12.1 Fa0/0 12 00:05:40 149 894 0 3
Another way to check EIGRP neighbor adjacencies is by issuing the following command:
R2#show ip eigrp interfaces
IP-EIGRP interfaces for process 100
Xmit Mean Pacing
Queue Un/Reliable Time Multicast Pending Flow
Interface Peers SRTT Un/Reliable Timer
Routes
Fa0/0 0 0/0 0 0/2 624 0
Fa0/1 1 0/0 104 0/2 432 0
In the output above, you can see that R2 is configured with EIGRP on both Fa0/0 and Fa0/1 but it has only one peer, on Fa0/1. The most commonly seen reasons for a neighbor adjacency not forming after EIGRP is enabled on the interface are:
- Mismatched EIGRP ASN
- Passive interface configured
Let’s check the EIGRP ASNs on both ends on that link:
R1#show ip protocols
Routing Protocol is eigrp 200
Outgoing update filter list for all interfaces is not set
Incoming update filter list for all interfaces is not set
[output omitted]
R2#sho ip protocols
Routing Protocol is eigrp 100
Outgoing update filter list for all interfaces is not set
Incoming update filter list for all interfaces is not set
[output omitted]After modifying R1’s configuration so that the EIGRP ASN is 100, the adjacency still does not form. Let’s run a show ip protocols command on this router to further inspect the situation:
R1#sho ip prot
Routing Protocol is eigrp 100
Outgoing update filter list for all interfaces is not set
Incoming update filter list for all interfaces is not set
Default networks flagged in outgoing updates
Default networks accepted from incoming updates
EIGRP metric weight K1=1, K2=0, K3=1, K4=0, K5=0
EIGRP maximum hopcount 100
EIGRP maximum metric variance 1
Redistributing: eigrp 100
EIGRP NSF-aware route hold timer is 240s
Automatic network summarization is in effect
Automatic address summarization:
10.0.0.0/8 for Loopback0
Summarizing with metric 281600
Maximum path: 4
Routing for Networks:
10.0.12.0/24
192.168.1.1/32
Passive Interface(s):
FastEthernet0/0
Routing Information Sources:
Gateway Distance Last Update
(this router) 90 00:00:52
Gateway Distance Last Update
10.0.12.2 90 00:01:02
Distance: internal 90 external 170
In the output above, you can see that R1’s EIGRP process is configured as passive on the interface toward R2 (FastEthernet0/0). What this means is that the network configured on that specific interface is advertised into EIGRP but the router cannot form EIGRP adjacencies on that interface. Let’s fix this:
R1#conf t
Enter configuration commands, one per line. End with CNTL/Z.
R1(config)#router eigrp 100
R1(config-router)#no passive-interface fa0/0
*Mar 1 02:27:21.871: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 100: Neighbor 10.0.12.2 (FastEthernet0/0) is up: new adjacency
Another thing you can inspect in the show ip protocols command is the unequal-cost load balancing configuration. As you know, as opposed to OSPF, EIGRP allows for such a configuration, where routes with different metrics may be installed in the routing table at the same time. They will route traffic in proportion to their metric, but in order for this to happen, a variance higher than 1 has to be configured on the router.
R1#sho ip protocols
Routing Protocol is eigrp 100
Outgoing update filter list for all interfaces is not set
Incoming update filter list for all interfaces is not set
Default networks flagged in outgoing updates
Default networks accepted from incoming updates
EIGRP metric weight K1=1, K2=0, K3=1, K4=0, K5=0
EIGRP maximum hopcount 100
EIGRP maximum metric variance 2
[output omitted]In the output above, you can see that the router is configured with a variance of 2, which means that the EIGRP process will install in the routing table routes with a metric up to double in value when compared to the metric of the best route.
Another common issue in hub-and-spoke topologies is when routes are advertised by a spoke that do not know how to reach other spokes in the EIGRP domain. The reason for this might be split horizon being enabled on the hub interface. This prevents prefixes being received on the specific interface from being advertised back on that interface toward the other spokes. In order to fix this, you have to disable split horizon functionality on the hub interface. This is done at the interface level:
Hub(config-router)#int Serial0
Hub(config-if)#no ip split-horizon eigrp 100
Another solution to this issue might be to configure the hub-and-spoke connections as point-to-point, using different dedicated IP subnets on each of them.
Troubleshooting WAN Connectivity
When investigating any reported problems in a WAN, there are several commands that you can use to locate the source of the problem. You will not be able to call your service provider to report a problem unless you can actually show that it is not your equipment that is at fault, rather it’s theirs. Just as when you report an issue with your home network, your service provider help desk will want to blame you for any issues.
The first command used by most network engineers is ping. Try to ping the faulty host and ping the default gateway from the network segment. This will tell you whether you have IP connectivity. Traceroute can also be used to identify where the packet may be failing.
The show ip interface brief command gives a snapshot of all of your interfaces and whether they are up or down, which is a quick and easy way of seeing what is happening. If the interfaces are operating correctly at layers 1 and 2, they should be Status up and Protocol up. The output below shows that they are administratively down:
RouterB#show ip interface brief
Interface IP-Address OK? Method Status Protocol
Fa0 unassigned YES unset administratively down down
Serial0/0 192.168.1.2 YES manual up up
Serial1/0 unassigned YES unset administratively down down
The show interface X command provides a lot of useful information, such as statistics and the configuration. The default bandwidth on Serial interfaces is 1544 Kbits or 1.544 Mbps. This is purely for routing protocols to decide which path to take. It does not affect the physical speed of the interface at all.
RouterB#show interface Serial0/0
Serial0 is up, line protocol is up
Hardware is HD64570
Internet address is 192.168.1.2/24
MTU 1500 bytes, BW 1544 Kbit, DLY 20000 usec,
reliability 255/255, txload 1/255, rxload 1/255 – Transmission and receive load
Encapsulation HDLC, Loopback not set – Encapsulation settings
Keepalive set (10 sec)
Last input 00:00:07, output 00:00:00, output hang never
Last clearing of show interface counters never
Input queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 0
Queueing strategy: weighted fair
Output queue: 0/1000/64/0 (size/max total/threshold/drops)
Conversations 0/1/256 (active/max active/max total)
Reserved Conversations 0/0 (allocated/max allocated)
5 minute input rate 0 bits/sec, 0 packets/sec
5 minute output rate 0 bits/sec, 0 packets/sec
18 packets input, 2813 bytes, 0 no buffer – Traffic statistics
Received 18 broadcasts, 0 runts, 0 giants, 0 throttles
0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
19 packets output, 1728 bytes, 0 underruns
0 output errors, 0 collisions, 4 interface resets – Interface resets
0 output buffer failures, 0 output buffers swapped out
2 carrier transitions
DCD=up DSR=up DTR=up RTS=up CTS=up
You should always issue a clear counters command when troubleshooting interface problems. Interface counters are cumulative, so you may be looking at errors that occurred some time ago.
RouterA#clear counters
Clear show interface counters on all interfaces [confirm]
Another useful command is debug serial interface.
 |
Counters can be cleared on any specific interface you are troubleshooting; you don't have to reset the statistics on all the interfaces.
R1#clear counters FastEthernet0/0 |
R1#clear counters FastEthernet0/0
RouterB#debug serial interface
Serial network interface debugging is on
RouterB#
02:33:41: Serial0: HDLC myseq 39, mineseen 39*, yourseen 39, line up
02:33:51: Serial0: HDLC myseq 40, mineseen 40*, yourseen 40, line up
02:34:01: Serial0: HDLC myseq 41, mineseen 41*, yourseen 41, line up
show controllers serial x is a very useful command and has been discussed in detail.
Troubleshooting EtherChannels
In the CCNA exam, you may be presented with an EtherChannel diagram or the output of show commands and asked what the probable issue is. Ensure that you are familiar with which settings PAgP and LACP require in order to form an EtherChannel.
Remember that each port in the EtherChannel must have matching settings in order for the EtherChannel to form. If one site is full-duplex and the other half-duplex, for example, then the channel can’t form.
When you follow my configuration commands for all the examples in the EtherChannel chapter, ensure that you check all the configurations and test out the relevant show commands. Also, try each of the commands below and note what each one reveals. We covered these in the theory section and EtherChannel labs.
Switch#show etherchannel ?
<1-64> Channel group number
detail Detail information
load-balance Load-balance/frame-distribution scheme among ports in
port-channel
port Port information
port-channel Port-channel information
protocol protocol enabled
summary One-line summary per channel-group
| Output modifiers
<cr>
End of Chapter Questions
Please visit https://www.howtonetwork.com/ccnasimplified to take the free Chapter 29 exam.
