This excerpt is from the book, ‘DevOps Troubleshooting: Linux Server Best Practices’, authored by Kyle Rankin, published by Pearson/Addison-Wesley Professional, Nov 2012, ISBN 0321832043, Copyright 2013 Pearson Education, Inc. For more info please visit: http://www.informit.com/title/0321832043
Want the whole thing? We’re giving away a free copy.
Is the Server Down? Tracking Down the Source of Network Problems
Most servers are attached to some sort of network and generally use the network to provide some sort of service. Many different problems can creep up on a network, so network troubleshooting skills become crucial for anyone responsible for servers or services on those servers. Linux provides a large set of network troubleshooting tools, and this chapter discusses a few common network problems along with how to use some of the tools available for Linux to track down the root cause.
Network troubleshooting skills are invaluable for every member of a DevOps team. It’s almost a given that software will communicate over the network in some way, and in many applications, network connectivity is absolutely vital for the software to function. When there is a problem with the network, everyone from the sysadmin, to the QA team, to the entire development staff will probably take notice. Whether your networking department is a separate group or not, when your entire DevOps team works together on diagnosing networking problems, you will get a better overall view of the problem. Your development team will give you the deep knowledge of how your software operates on the network; your QA team will explain how the application behaves under unusual circumstances and provide you with a backlog of networking bug history; and your sysadmin will provide you with an overall perspective of how networked applications work under Linux. Together you will be able to diagnose networking problems much faster than any team can individually.
Server A Can’t Talk to Server B
Probably the most common network troubleshooting scenario involves one server being unable to communicate with another server on the network. This section will use an example in which a server named dev1 can’t access the web service (port 80) on a second server named web1. Any number of different problems could cause this, so we’ll run step by step through tests you can perform to isolate the cause of the problem.
Normally when troubleshooting a problem like this, you might skip a few of these initial steps (such as checking the link), since tests further down the line will also rule them out. For instance, if you test and confirm that DNS works, you’ve proven that your host can communicate on the local network. For this example, though, we’ll walk through each intermediary step to illustrate how you might test each level.
Client or Server Problem
One quick test you can perform to narrow down the cause of your problem is to go to another host on the same network and try to access the server. In this example, you would find another server on the same network as dev1, such as dev2, and try to access web1. If dev2 also can’t access web1, then you know the problem is more likely on web1, or on the network between dev1, dev2, and web1. If dev2 can access web1, then you know the problem is more likely on dev1. To start, let’s assume that dev2 can access web1, so we will focus our troubleshooting on dev1.
Is It Plugged In?
The first troubleshooting steps to perform are on the client. You first want to verify that your client’s connection to the network is healthy. To do this you can use the ethtool program (installed via the ethtool package) to verify that your link is up (the Ethernet device is physically connected to the network). If you aren’t sure what interface you use, run the /sbin/ifconfig command to list all the available network interfaces and their settings. So if your Ethernet device was at eth0
$ sudo ethtool eth0
Settings for eth0:
Supported ports: [ TP ]
Supported link modes: 10baseT/Half 10baseT/Full
Supports auto-negotiation: Yes
Advertised link modes: 10baseT/Half 10baseT/Full
Advertised auto-negotiation: Yes
Port: Twisted Pair
Supports Wake-on: pg
Current message level: 0x000000ff (255)
Link detected: yes
Here, on the final line, you can see that Link detected is set to yes, so dev1 is physically connected to the network. If this was set to no, you would need to physically inspect dev1’s network connection and make sure it was connected. Since it is physically connected, you can move on.