Troubleshooting Network Problems – Part 2: Device Configuration Issues

We will be providing a four part blog series on troubleshooting network problems.  The series will address:

• Part 1:  Network Device Performance
• Part 2:  Network Device Configurations
• Part 3:  Network Traffic and Bandwidth Consumption
• Part 4:  IP Address Issues

One of the first questions network administrators should ask themselves when troubleshooting is “did something on my network change?” More than 80% of network issues are the result of device configuration errors, many of which were unplanned, unauthorized, or not fully tested prior to deployment.

Archiving Your Configurations

Hopefully you have been keeping an archive of your device configurations so you can compare the current version to the previously archived versions.  If you haven’t been, then you need to start immediately.

For a Cisco router, the archive config command allows you to save your IOS configuration in the configuration archive using a standard location and file name prefix that is automatically appended with an incremental version number as each consecutive file is saved.

Router# configure terminal
Router(config)# archive
Router(config-archive)# path disk0:myconfig

 You then save the current running configuration in the configuration archive as follows:

Router# archive config

The show archive command displays information on the files saved in the configuration archive as shown in the following sample output:

Router# show archive

There are currently 1 archive configurations saved.
The next archive file will be named disk0:myconfig-2
Archive #  Name
0
1       disk0:myconfig-1 <- Most Recent
2

Comparing Configurations

Assuming that you have a config archive, you can perform a line-by-line comparison of any two configuration files and generate a list of the differences between them using the show archive config differences command.

show archive config differences[filename1(path)[filename2(path)][ignorecase]]

The output will display the results of the diff operation performed on the configuration files.  A plus symbol (+) indicates that the configuration line exists in filename2(path) but not in filename1(path) while a minus symbol (-) indicates that the configuration line exists in filename1(path) but not in filename2(path).  An exclamation point (!) with descriptive comments is used to identify order-sensitive configuration lines whose location is different in filename1(path) than in filename2(path).

+ip subnet-zero
+ip name-server 10.4.4.4
+voice dnis-map 1
+dnis 111
interface Ethernet1/0
+no ip address
+shutdown
+ip default-gateway 10.5.5.5
+ip classless
+access-list 110 deny ip any host 10.1.1.1
+access-list 110 deny ip any host 10.1.1.2
+access-list 110 deny ip any host 10.1.1.3
+snmp-server community private RW
-no ip subnet-zero
interface Ethernet1/0
-ip address 10.7.7.7 255.0.0.0
-no ip classless
-snmp-server community public RO

Alternatives to CLI

As opposed to relying on a cumbersome and hard to decipher CLI troubleshooting process, the network administrator may want to consider one of the many commercially available network change and configuration management tools as part of a network management system that will automate and simplify the process of managing device configurations.  SolarWinds Network Configuration Manager provides automated network configuration and compliance management and includes an easy-to-use compare and rollback functionality.

Configuration Rollback

Once you determine that a config has changed, you can replace the current running config with any saved config file using the configure replace command.  This functionality can be used to revert to a previous configuration state, effectively rolling back any configuration changes that were made since the previous configuration state was saved.

Router# configure replace disk0:myconfig

This will apply all necessary additions and deletions
to replace the current running configuration with the
contents of the specified configuration file, which is
assumed to be a complete configuration, not a partial
configuration. Enter Y if you are sure you want to proceed. ? [no]: Y

Total number of passes: 1
Rollback Done

Tools for Network Troubleshooting

There are literally many open source, free, or commercially licensed products available to manage configurations.  Below are some guidelines on picking the right tool for your needs.

• Multiple vendor device support – It would be very difficult in this day and age to find a network that consists of equipment from a single vendor.  While all vendors provide some type of tool or utility that will manage and monitor their own equipment, it is critical that you look for a tool that allows you to monitor all of your different vendors in a single pane of glass.
• Support for multiple standard protocols including: SNMP, ICMP, and Syslog for network management; RDP, WMI, and WS –Management for Windows management; and NetFlow, J-Flow, sFlow, IPFIX, and NetStream for flow based traffic monitoring.
• Real-time and historical analysis capabilities. Although most problems in network administration directly relate to how the network operates right now, the only effective way to ascertain today’s behaviors is to view them in comparison with yesterday’s or last week’s.
• Visualizations accessible from anywhere. As a network administrator, you’re not always sitting in your office. Problems and issues tend to pop up all across the network, some of which require on-site support. In these cases, having visualizations that can be accessed from anywhere—for example, using a standard Web browser—gives you the ability to take your toolset to wherever the problem exists.
• Drill-down support. With drill-down support it is possible to quickly move from the highest-level view down into specific problems as needed. Drill-down support reduces on-screen clutter, enabling a single-glimpse and high-level view during periods of nominal activity.
• Affordability. Lastly, any toolset used in troubleshooting and resolving issues must cost less than the amount of benefit it provides. Expensive solutions take longer to pay for themselves and may be more difficult to obtain in a time of shrinking IT budgets. Finding the tool that meets your needs at an acceptable cost is important to gaining the biggest return on your investment.

By Brad Hale, Product Marketing Principal for SolarWinds. SolarWinds (NYSE: SWI) provides powerful and affordable IT management software to customers worldwide.