Troubleshooting Network Problems – Part 4: IP Address Conflicts

This is the final installment of a four part blog series on troubleshooting network problems.  The other parts of the series address:

• Part 1:  Network Device Performance
• Part 2:  Network Device Configurations
• Part 3:  Network Traffic and Bandwidth Consumption
• Part 4:  IP Address Issues
  

The System Has Detected an IP Address Conflict!  We’ve all seen this message before.  But what does it really mean and how do we go about identifying and resolving the problem?

IP address conflicts occur when two devices on a network are assigned the same IP address resulting in one or both being disabled and losing connectivity until the conflict is resolved.  IP address conflicts are almost always the result of configuration errors including:  assignment of the same static IP address by a network administrator; assignment of a static IP address within the DHCP range (dynamic range) resulting in the same address being automatically assigned by the local DHCP server; an error in the DHCP server; or a system coming back online after an extended period in stand-by or hibernate mode with an IP address that has been re-assigned and is in use on the network.

Here are a number of steps that you can take to troubleshoot this pesky problem.

Step 1 – Look For Overlapping IP Address Ranges on Your DHCP Server

If you are using multiple DHCP servers, you will first want to verify that no two servers have overlapping IP address ranges.  This can be as simple as comparing the IP address ranges and looking for overlaps when the servers are using dynamic or automatic allocation of IP addresses.  If they are using static allocation, then you will need to review each hard coded IP address assignment.

Step 2 – Look for Duplicate Static IP Addresses

Look for devices on the network segment that have been statically configured with the duplicate IP address.  Once found, you can either reconfigure the device to use DHCP or you can configure the DHCP server to stop assigning the duplicated IP address.

Step 3 – Find the Conflicting MAC Addresses

If steps 1 and 2 do not produce results, you will need to find the MAC addresses of the conflicting devices.  Since the MAC address is unique for each device on the network, you can look for devices that contain the same IP address but with different MAC addresses. You can use the Address Resolution Protocol (ARP) to establish a correspondence between the IP address and the MAC address.  Start at your core router and use the show ip arp command:

Router# show ip arp

Protocol  Address               Age(min)  Hardware Addr      Type     Interface
Internet  172.16.233.229         -            0000.0c59.f892    ARPA   Ethernet0/0
Internet  172.16.233.218         -            0000.0c07.ac00    ARPA   Ethernet0/0
Internet  172.16.233.19            -            0000.0c63.1300   ARPA   Ethernet0/0
Internet  172.16.233.309         -            0000.0c36.6965   ARPA   Ethernet0/0
Internet  172.16.168.11             -            0000.0c63.1300   ARPA   Ethernet0/0
Internet  172.16.168.254         9            0000.0c36.6965   ARPA   Ethernet0/0

If you were to see two IP addresses with differing hardware addresses then you have located your problem devices.

Step 4 – Trace the Location of the Device

Perhaps you want to know the physical location or at least the switch port that the offending devices are connected to.  One way is to go to the switch and use the show mac-address table command.  This will show you the MAC address for each port.

switch# show mac-address-table

           Mac Address Table
——————————————-
Vlan    Mac Address          Type                Ports
—–     —————          ——–             ——
1           0007.e9e2.2d7d    DYNAMIC     Fa0/5
1           0009.0f30.07e9    DYNAMIC     Fa0/48
1           0009.5bbc.af04    DYNAMIC     Fa0/28
1           00e0.bb2c.30d1    DYNAMIC     Gi0/1
1           00e0.bb2c.3e5f     DYNAMIC     Gi0/1
Total Mac Addresses for this criterion: 5

Switch#

Unfortunately, you need to run this command from each switch and, if the network is down, you will have to go to the console of each switch.  This can be very tedious and time consuming not to mention logistically challenged in the case of geographically distributed networks.

Another alternative is the use of commercially available network monitoring tools

SolarWinds User Device Tracker Endpoint Details

that will trace the location of a device on a network automatically.  SolarWinds User Device Tracker is a device tracking and switch port management software solution that quickly locates a device on the network by searching on the IP address, Hostname or MAC address.

Preventing Conflicts in the Future

Once you have identified and corrected IP address conflicts, here are some tips to prevent future conflicts:

• Use DHCP to reduce the chances of manually assigning duplicate addresses.
• Set your DHCP server to detect IP address conflicts.
• Modify the DHCP lease duration to something less than the default lease time of 8 days.
• Use multiple DHCP servers, each having it own specific scope.
• Reserve IP addresses instead of assigning static IP addresses.
• Use automated DHCP, DNS, and IP address management and monitoring tools

Even by following these tips, there still remains the possibility that IP conflicts will occur.  I would encourage you to evaluate a commercially available IP address management product that allows you to centrally manage, monitor, alert, and report on your IP infrastructure.  By proactively managing and monitoring your IP address space you can significantly reduce the chances of IP address conflicts.

SolarWinds IP Address Manager

SolarWinds IP Address Manager(IPAM) enables you and your team to ditch your spreadsheets for an easy-to-use, centralized IP address monitoring and management solution. With IPAM you can manage Microsoft® DHCP services, monitor Microsoft DNS and Cisco® DHCP servers, and manage your IP address space; all from an intuitive, centralized Web console.

• Centrally manage, alert, & report on your IP address space
• Manage & monitor Microsoft DHCP/DNS services & monitor Cisco DHCP servers
• Delivers role-based access & control from an intuitive web based interface
• Alert notifications help prevent your subnets & DHCP scopes from filling up
• Automatically discovers used & unused addresses & typically deploys in less than an hour

By Brad Hale, Product Marketing Principal for SolarWinds. SolarWinds (NYSE: SWI) provides powerful and affordable IT management software to customers worldwide.

Troubleshooting Network Problems – Part 3: Bandwidth & Traffic Analysis

We will be providing a four part blog series on troubleshooting network problems.  The series will address:

• Part 1:  Network Device Performance
• Part 2:  Network Device Configurations
• Part 3:  Network Traffic and Bandwidth Consumption
• Part 4:  IP Address Issues

The only things in life that are certain are death, taxes and network issues.  Okay, I added the last one but we all know that you have undoubtedly heard complaints from end users that “the network is slow”.  In Troubleshooting Network Problems – Part 1: Network Device Performance I provided some tips on troubleshooting network performance issues by establishing a performance baseline and then collecting network device performance statistics.  Today’s post will provide some insight on using bandwidth monitoring and traffic analysis to troubleshoot network performance.

Performing each correctly assists the network administrator with identifying network bottlenecks that could be the cause of your “slow network”. It helps the admin identify the network needs and uses of servers and their hosted applications, as well as how the network needs of one IT service impacts the needs of another. It also delivers hard data that objectively verifies the ability of the network to meet stated Service Level Agreements (SLAs).

The two most common ways in which network traffic can be monitored and measured for performance are through packet analysis and flow analysis. Traditional packet-based monitoring tools enable peering into individual packets to determine their contents, the transactions between systems, and the details of communications being passed along that network.  The packet-based approach is a lot like attempting to determine the cause of a traffic jam by peeking into each individual vehicle. Knowing what people and cargo are travelling within each vehicle may be helpful in answering some questions, but it’s not likely to illuminate the cause of the system-wide slowdown.

Flow analysis, on the other hand, provides insight into the flow of traffic within the network, specifically the “who” and “what” of traffic consumption.  Flow analysis allows us to step back to see conditions on the system as a whole. To help you understand the differences in perspective here, let’s take a look at common ways used to measure traffic on a network:

• Protocol analyzers – Protocol analyzers take a look at network conditions from the perspective of the packet. These tools analyze conversations between devices on the network from the location where the analyzer is measuring. This information gives the network administrator an extremely detailed view of individual transactions between two devices and the specific data being transferred between them.
• Hardware probes and distributed analyzers – Hardware probes and distributed analyzers are an early attempt to overcome the limitations of an individual protocol analyzer. These tools can be positioned all across the network for the gathering of information. They go far in providing the whole-system perspective that is so difficult to gather through the previous two perspectives.
• Traffic flow analyzers – These tools overcome the administration headaches of hardware probes and distributed analyzers by leveraging the data flow capture capabilities of the network device (router) itself. Traffic flow analyzers receive flow data directly from monitored devices and analyze that data to gain the high-level perspective needed for troubleshooting incidents across the network system.

NetFlow

NetFlow is a network traffic monitor protocol developed by Cisco Systems for collecting IP traffic information and monitoring network traffic. While the term NetFlow has become a de-facto industry standard many other manufacturers support alternative flow technologies including; Juniper (Jflow); 3Com/HP, Dell and Netgear (s-flow); Huawei (NetStream); Alcatel-Lucent (Cflow); and Ericsson (Rflow).

Routers and switches that support NetFlow collect IP traffic statistics on all interfaces where NetFlow is enabled, and later export those statistics as NetFlow records, toward at least one NetFlow collector – typically a server that does the actual traffic analysis. The NetFlow collector then processes the data to perform the traffic analysis and presentation in a user-friendly format.  NetFlow collectors can take the form of hardware based collectors or probes, or as network monitoring software collectors. SolarWinds NetFlow Traffic Analyzer (NTA) is an example of a software based NetFlow collector that collects traffic data, correlates it into a useable format, and then presents it to the user in a web based interface.

Monitoring and analyzing NetFlow will help obtain valuable information about network users and applications, peak usage times, and traffic routing.  In contrast with traditional SNMP-dependent systems, NetFlow-based traffic monitoring has the ability to characterize traffic from applications and users, understand the traffic patterns, provide a holistic view into bandwidth utilization and WAN traffic, support CBQoS validation and performance monitoring, be used for network traffic forensics, and aid in compliance reporting.

Configuring NetFlow on a Cisco router is a very straightforward and easy process.  You can use a free tool such as SolarWinds NetFlow Configurator or you can manually configure using the following steps:

To display the statistics from the NetFlow data export, including statistics for the main cache and all other enabled caches, use the show ip flow export command in user EXEC or privileged EXEC mode.  The following is sample output from the show ip flow export command:

Router# show ip flow export

Flow export is enabled
Exporting flows to 10.42.42.1 (9991) 10.0.101.254 (9991)

  Exporting using source IP address 10.0.101.203
Version 5 flow records

Export Stats for 10.42.42.1 (9991)

3 flows exported in 3 udp datagrams
0 flows failed due to lack of export packet
3 export packets were sent up to process level
0 export packets were dropped due to no fib
0 export packets were dropped due to adjacency issues
0 export packets were dropped enqueuing for the RP
0 export packets were dropped due to IPC rate limiting

Export Stats for 10.0.101.254 (9991)

7 flows exported in 7 udp datagrams
0 flows failed due to lack of export packet
6 export packets were sent up to process level
0 export packets were dropped due to no fib
0 export packets were dropped due to adjacency issues
0 export packets were dropped enqueuing for the RP
0 export packets were dropped due to IPC rate limiting

There are a number of commercially available flow analysis and network bandwidth monitor products that greatly simplify the process of enabling NetFlow and then display the raw numbers into easy-to-interpret charts and tables.

Let’s take a look at three particular use cases for using flow analysis for troubleshooting bandwidth and traffic.

Applications

SolarWinds Netflow Traffic Analyzer
Top 10 Applications View

When an application on the network begins consuming more than its fair share of network bandwidth, its use will impact the capacity available for other network services. The problem with identifying these incidents using other types of network tools is that the reporting of problems tends to focus on the network service being impacted. For example, when the problem occurs, the network administrator usually starts with knowledge that Application B “is slow today.” The job is then theirs to determine why the service is slow and what is inhibiting its desired level of performance. Using effective flow analysis tools, the administrator can easily view the traffic and usage patterns across the entire network to identify that Application A is actually the culprit. Conversely, using tools with a closer perspective may incorrectly focus the administrator’s troubleshooting on Application B, while ignoring the impact of Application A.

Protocols

A second and similar issue occurs when a specific protocol over consumes network resources. Streaming protocols are an excellent example of this type of constant and predictable network flow. When users on a network make use of streaming applications, their consumption typically occurs at a constant level over an extended period of time.

Different than transaction-based protocols, streaming protocols have the tendency to saturate available network resources due to the additive effect of multiple streams. One user making use of one stream may not be likely to cause a network problem, but 50 or 100 users employing an equal number of streams quickly begins saturating the network. Unlike packet-based tools that analyze individual pieces as they go by, flow analysis tools enable the identification of the source, destination, and protocol of streams across the network. The end result is the ability to craft effective network policies that enable streaming protocols where necessary while preventing those that negatively impact the functionality of the network.

Top Talkers

A final area for which flow analysis tools are particularly well suited is the identification of top talkers or, who is consuming the bandwidth.  The Top Talkers feature of NetFlow can be useful for analyzing and troubleshooting network traffic in any one of the following ways:  Security by viewing a list of the top talkers to see if traffic patterns are consistent with Denial of Service (DoS) attacks; Load balancing through the identification of the most heavily used parts of your network; and general traffic study and planning for your network.

Tools for Network Troubleshooting

There are many open source, free, or commercially licensed products available to monitor and troubleshoot traffic and bandwidth.  Here are a few some guidelines on picking the right tool for your needs.

• Multiple vendor device support – It would be very difficult in this day and age to find a network that consists of equipment from a single vendor.  While all vendors provide some type of tool or utility that will manage and monitor their own equipment, it is critical that you look for a tool that allows you to monitor all of your different vendors in a single pane of glass.
• Support for multiple standard protocols including: SNMP, ICMP, and Syslog for network management; RDP, WMI, and WS –Management for Windows management; and NetFlow, J-Flow, sFlow, IPFIX, and NetStream for flow based traffic monitoring.
• Real-time and historical analysis capabilities. Although most problems in network administration directly relate to how the network operates right now, the only effective way to ascertain today’s behaviors is to view them in comparison with yesterday’s or last week’s.
• Visualizations accessible from anywhere. As a network administrator, you’re not always sitting in your office. Problems and issues tend to pop up all across the network, some of which require on-site support. In these cases, having visualizations that can be accessed from anywhere—for example, using a standard Web browser—gives you the ability to take your toolset to wherever the problem exists.
• Drill-down support. With drill-down support it is possible to quickly move from the highest-level view down into specific problems as needed. Drill-down support reduces on-screen clutter, enabling a single-glimpse and high-level view during periods of nominal activity.
• Affordability. Lastly, any toolset used in troubleshooting and resolving issues must cost less than the amount of benefit it provides. Expensive solutions take longer to pay for themselves and may be more difficult to obtain in a time of shrinking IT budgets. Finding the tool that meets your needs at an acceptable cost is important to gaining the biggest return on your investment.

By Brad Hale, Product Marketing Principal for SolarWinds. SolarWinds (NYSE: SWI) provides powerful and affordable IT management software to customers worldwide.

Managing your people assets is as important as managing your IT assets

Many firms in the technology consultancy sector are facing change on several fronts. In addition to an economy that continues to leave many businesses guessing as to where it will head next; firms also have the challenge of adapting to constant technological change. As this article reminds us, CIOs across all industries are facing pressure to ensure that any IT investment pays off and all IT assets are managed as efficiently as possible. However, managing people resources across the firm can provide as positive an impact on profit margins as optimising physical assets and should not be overlooked.

For many IT consulting firms, understanding resource across the business can be a real challenge. This is frequently due to a lack of appropriate project management tools that provide a true insight into which consultants are working on which projects and the resource and skills gaps across the business. Firms that do not have this information may also find that it is more difficult to make informed decisions when balancing teams, budgets and engagements – the main cause of client under or over resourcing.

Consulting firms must ensure that their project managers have one clear view of resource and skills across the business. This will help the company balance resource and project demands and ultimately help optimise profit margins.

Troubleshooting Network Problems – Part 2: Device Configuration Issues

We will be providing a four part blog series on troubleshooting network problems.  The series will address:

• Part 1:  Network Device Performance
• Part 2:  Network Device Configurations
• Part 3:  Network Traffic and Bandwidth Consumption
• Part 4:  IP Address Issues

One of the first questions network administrators should ask themselves when troubleshooting is “did something on my network change?” More than 80% of network issues are the result of device configuration errors, many of which were unplanned, unauthorized, or not fully tested prior to deployment.

Archiving Your Configurations

Hopefully you have been keeping an archive of your device configurations so you can compare the current version to the previously archived versions.  If you haven’t been, then you need to start immediately.

For a Cisco router, the archive config command allows you to save your IOS configuration in the configuration archive using a standard location and file name prefix that is automatically appended with an incremental version number as each consecutive file is saved.

Router# configure terminal
Router(config)# archive
Router(config-archive)# path disk0:myconfig

 You then save the current running configuration in the configuration archive as follows:

Router# archive config

The show archive command displays information on the files saved in the configuration archive as shown in the following sample output:

Router# show archive

There are currently 1 archive configurations saved.
The next archive file will be named disk0:myconfig-2
Archive #  Name
0
1       disk0:myconfig-1 <- Most Recent
2

Comparing Configurations

Assuming that you have a config archive, you can perform a line-by-line comparison of any two configuration files and generate a list of the differences between them using the show archive config differences command.

show archive config differences[filename1(path)[filename2(path)][ignorecase]]

The output will display the results of the diff operation performed on the configuration files.  A plus symbol (+) indicates that the configuration line exists in filename2(path) but not in filename1(path) while a minus symbol (-) indicates that the configuration line exists in filename1(path) but not in filename2(path).  An exclamation point (!) with descriptive comments is used to identify order-sensitive configuration lines whose location is different in filename1(path) than in filename2(path).

+ip subnet-zero
+ip name-server 10.4.4.4
+voice dnis-map 1
+dnis 111
interface Ethernet1/0
+no ip address
+shutdown
+ip default-gateway 10.5.5.5
+ip classless
+access-list 110 deny ip any host 10.1.1.1
+access-list 110 deny ip any host 10.1.1.2
+access-list 110 deny ip any host 10.1.1.3
+snmp-server community private RW
-no ip subnet-zero
interface Ethernet1/0
-ip address 10.7.7.7 255.0.0.0
-no ip classless
-snmp-server community public RO

Alternatives to CLI

As opposed to relying on a cumbersome and hard to decipher CLI troubleshooting process, the network administrator may want to consider one of the many commercially available network change and configuration management tools as part of a network management system that will automate and simplify the process of managing device configurations.  SolarWinds Network Configuration Manager provides automated network configuration and compliance management and includes an easy-to-use compare and rollback functionality.

Configuration Rollback

Once you determine that a config has changed, you can replace the current running config with any saved config file using the configure replace command.  This functionality can be used to revert to a previous configuration state, effectively rolling back any configuration changes that were made since the previous configuration state was saved.

Router# configure replace disk0:myconfig

This will apply all necessary additions and deletions
to replace the current running configuration with the
contents of the specified configuration file, which is
assumed to be a complete configuration, not a partial
configuration. Enter Y if you are sure you want to proceed. ? [no]: Y

Total number of passes: 1
Rollback Done

Tools for Network Troubleshooting

There are literally many open source, free, or commercially licensed products available to manage configurations.  Below are some guidelines on picking the right tool for your needs.

• Multiple vendor device support – It would be very difficult in this day and age to find a network that consists of equipment from a single vendor.  While all vendors provide some type of tool or utility that will manage and monitor their own equipment, it is critical that you look for a tool that allows you to monitor all of your different vendors in a single pane of glass.
• Support for multiple standard protocols including: SNMP, ICMP, and Syslog for network management; RDP, WMI, and WS –Management for Windows management; and NetFlow, J-Flow, sFlow, IPFIX, and NetStream for flow based traffic monitoring.
• Real-time and historical analysis capabilities. Although most problems in network administration directly relate to how the network operates right now, the only effective way to ascertain today’s behaviors is to view them in comparison with yesterday’s or last week’s.
• Visualizations accessible from anywhere. As a network administrator, you’re not always sitting in your office. Problems and issues tend to pop up all across the network, some of which require on-site support. In these cases, having visualizations that can be accessed from anywhere—for example, using a standard Web browser—gives you the ability to take your toolset to wherever the problem exists.
• Drill-down support. With drill-down support it is possible to quickly move from the highest-level view down into specific problems as needed. Drill-down support reduces on-screen clutter, enabling a single-glimpse and high-level view during periods of nominal activity.
• Affordability. Lastly, any toolset used in troubleshooting and resolving issues must cost less than the amount of benefit it provides. Expensive solutions take longer to pay for themselves and may be more difficult to obtain in a time of shrinking IT budgets. Finding the tool that meets your needs at an acceptable cost is important to gaining the biggest return on your investment.

By Brad Hale, Product Marketing Principal for SolarWinds. SolarWinds (NYSE: SWI) provides powerful and affordable IT management software to customers worldwide.

Is Telecom Outsourcing Really the Answer?

We have all heard the term ‘outsourcing’, especially in IT.  Unfortunately, the concept has lost its appeal and value as people tend to associate outsourcing with the elimination of jobs or the replacement of resources with process.  In the world of telecommunications, a function essential to every enterprise in business today, the management of expenses, contracts, and daily tasks has immersed telecom management into chaos.  Without optimized communications and effective telecom services, your employees, customers, partners, and vendors suffer, and your reputation may tarnish due to poor customer service.  Because the very function of managing telecom seems to be a necessary evil, the concept of outsourcing it becomes a bit more viable – appealing even.   Here are 4 common beliefs that can cloud the value of outsourcing:

1. Telecom is Messy and Chaotic

The industry is full of many different service providers, carriers, agents, consultants, hardware providers, Telecom Expense Management (TEM) companies, mobility management firms, phone providers, and IT management firms.  With so many options to choose from, it becomes difficult to choose the right provider or technology for your business needs.  The result is too many vendors pointing fingers at each other when something goes wrong.

Answer: A strategic, holistic partner who will look out for your organization’s needs.  Telecom outsourcing eliminates the hassle of managing multiple carriers, multiple vendors, and having to navigate through complex technologies and solutions.

2. Telecom is Not a Core, Strategic Part of Business

While potentially confusing and certainly frustrating, telecom is a necessary aspect of your business, critical to its daily function.  However, the management and optimization of telecom is not a core part of your business. Telecom tends to fall under the umbrella of IT management, but as such does not have the luxury of qualified, devoted resources for effective daily operation.  As with other business functions in the enterprise, such as human resources, accounting, or operations, telecom should be separated out as its own entity within the enterprise.  Impact: IT teams are focusing on the day to day tasks of telecom.

Answer:  Because telecom tends to burden multiple business units with the day to day tasks of supplier and expense management, a telecom outsourcing solution becomes a highly desirable option that will get your complete organization back on track with their own departmental strategic initiatives.

3. The Amalgamation of IT and Telecom

Although IT and telecom do seem to blend together with the emergence of complex technologies such as MPLS, Ethernet, cloud computing, and disaster recovery, IT needs to focus how enterprise technology can enhance the business from a strategic perspective.  Managing carrier contracts, telecom expenses, mobility devices, and helpdesk support, while crucial to an enterprise, is not a valuable consumption of IT’s time. The result is that both telecom and IT suffer: strategic IT priorities take a back seat to the immediacy of telecom-related problems.

Answer:  Complete telecom outsourcing allows your IT organization to contribute to core aspects of their job, furthering enterprise-wide IT initiatives.

4. Effective Telecom Management Requires Dedication and Ownership

While IT and telecom go hand in hand, each discipline requires acute time dedication and ownership.  Ineffective management and outcomes result from attempting to meld two distinct businesses into one.  Precious time and budget dollars are spent on the people, processes, and technologies required to effectively manage telecom services internally.

Answer: Telecom outsourcing releases management time and reduces the total cost of telecom (TCT™) ownership and total cost of telecom per employee.

Revisit the concept of Telecom Outsourcing and see if it’s the right answer for your company. Learn more about how you can Measure Telecom in your business and alleviate hard costs, as well as soft costs, that are attached to managing your telecom environment. Re-focus and get back to what’s important to your business.

Image courtesy of FreeDigitalPhotos.net

The 7 Most Avoidable Wireless Telecommunication Expenses

A telecommunications enterprise is a complex and multi-faceted organization that is both a strategic necessity and a likely source of economic drain.  Most companies aren’t aware of the cost outflow within their wireless telecom environment, or if they are, managers don’t have the time or resources to identify and correct the source.  The lack of focus on the daily cost management of telecom can continue and snowball into significant losses to the enterprise.

Phone image via Shutterstock

Identifying and managing these areas of loss could most certainly be a full time job.  The problem with continued waste is that most enterprises do not have resources focused on the day to day management of wireless telecom expenses.  Wireless telecom cost drains can be obvious, some can be insidious, and still others need to be drudged up to the surface.   All, however, require time and a tedious level of commitment to overcome.

As the list of unmanaged wireless telecom expenses can be lengthy, a good place to start is with the lowest-hanging fruit – those problems that are the easiest to resolve and which can reap immediate savings.   In review of wireless expenses, immediacy of savings can best be realized by reviewing some key areas of your wireless expenses, move on to dig up the stealthy issues that contribute to cost outflow.

1. Plan Pooling and Optimization – Own pool optimization by specifying if and when automatic optimizations occur to improve device performance – don’t let this run unmanaged.
2. Voice/Data/Text Overages – Unless your mobile user base has an unlimited plan for voice, data and texting, monthly overages can voraciously eat away at your budget.  Overages comprise one of the most common problems amongst mid-sized organizations today.  Research and review who your mobile user is, what he/she will be utilizing the mobile device for (cellular telephone, email, presentations, web browsing, etc.) and optimize a monthly plan that will be built around users’ needs.
3. Excessive Roaming Charges – If your mobile users are racking up roaming charges then it is time to re-evaluate their plan.  Move them to a global plan that will enable them to utilize their device when and where necessary.   Roaming charges nonsensically creep into expenses but can be easily avoided.
4. Unnecessary Add-on Features and Services – Another source of escape occurs by providing an employee with too much functionality or service that goes unused.  Allowing unnecessary add-ons such as insurance, roadside assistance, or visual voicemail will only ensure that telecom cost creep will continue to erode your bottom line.   Review each mobile user contract to determine which features are absolutely necessary for him or her to perform their duties, and eliminate the rest.
5. Zero Usage Devices – One of the more common problems, the management of devices that are no longer utilized, such as those that belong to terminated employees or those that have been replaced or upgraded, requires more time and effort to optimize, but can result in modest cost savings.  Having a strictly enforced mobile user policy in place that addresses zero usage device management is essential.
6. Unused or Underutilized Data and Text Plans – Rectifying what is actually billed with what is used is an arduous task but one that can pay big dividends.  Assess those plans that are underutilized or not used at all to determine what can be eliminated or consolidated.  Get rid of those plans that are unnecessary.  Perform an audit amongst your mobile user base to determine the exact nature of plan usage on an employee by employee basis to ensure the best plan fit.
7. Mobile App/Subscription Slamming – This happens to be one of the most common areas of cost drain amongst mid-sized businesses with 500-5,000 employees.  In mobile app or subscription ‘slamming’, customers are switched to another telephone provider without their knowledge or permission.  These types of misleading sales practices among mobile service providers can lead to intentional fee increases and can also change the terms of service.  If not carefully monitored and guarded against, slamming practices can unknowingly increase costs and detrimentally alter contracted terms.

Tighter controls and regular audits can make a significant difference to the bottom line of a mid-sized company.  In auditing and assessing wireless expenses in corporate mobile telecom environments, these seven points are quite common among nearly 70% of companies we work with.

Get your Telecom Outsourcing Assessment now to see if any of these items apply to your business, and follow our blog for additional articles relating to Turnkey Telecom Management™ (TTM™) and Telecom Outsourcing.

Troubleshooting Network Problems – Part 1: Network Device Performance

We will be providing a four part blog series on troubleshooting network problems.  The series will address:

• Part 1: Network Device Performance
• Part 2: Network Device Configurations
• Part 3: Network Traffic and Bandwidth Consumption
• Part 4: IP Address Issues

No matter how carefully planned your network design is, how much redundancy you have built in, or how much you proactively monitor your network, you are bound to have a problem at some point.  And when that problem occurs, you need some steps and tools to troubleshoot the problem so you can minimize the impact to your users.

“The network is slow today” is without a doubt one of the most disliked phrases heard by network administrators. The network has become a dumping ground for problems that originate as often as not from servers and applications as from the network. Thus, one of the biggest jobs of the network administrator is to defend their network from being labeled the cause of today’s problem. Because slow environment performance is often first—and often incorrectly— attributed to the network, rapid identification and problem isolation is critical to the administrator’s workload.

But, what causes a “slow network”?  Most network performance issues can be attributed to one of four broad categories:  device performance, device configurations, traffic/bandwidth, or IP issues.  Today, I will provide you some basic tools and tips for troubleshooting network device performance issues.

Baseline Network Performance

Hopefully you have performed a baseline of your network performance so you know the normal working conditions of your network infrastructure. This baseline can then be used for comparison to catch changes that could indicate a problem, provide early indicators that application and network demands are pushing near the available capacity, and align network performance baselines with service-level agreements (SLAs).

If you haven’t established a baseline, then you will need to rely on your equipment vendors and their recommended or “best practice” thresholds.  You can also use various network equipment or monitoring forums to see what other IT professionals are doing.

Collect Network Device Performance Metrics

Network device performance metrics provide information about the system resources on each individual device. These metrics are critical in ascertaining whether a resource overuse problem is a central cause of a reduction in performance. Collecting and reporting on network devices helps the troubleshooting administrator quickly identify whether the device is a source of the problem or the problem lies elsewhere.

CPU Load & Memory Utilization from
      SolarWinds Network Performance Monitor

Device monitoring using the Simple Network Management Protocol (SNMP) provides a very device-centric view of network conditions. Using SNMP, counters on a device such as a router, switch, or firewall can be measured and forwarded to a network management system for review. This data is useful for understanding performance conditions that are specific to that device.  Performance statistics such as CPU utilization, Interface/Bandwidth Utilization, and Memory Utilization represent the majority of performance issues encountered in the day-to-day operation of network devices.  There are dozens of free and commercially available tools in the market that will allow you to monitor these device statistics.

Switch/Router CPU Utilization

Common symptoms of high CPU utilization within your switch or router include:

• High percentages in the show process cpu command output
• Input queue drops
• Slow performance
• Services such as Telnet, console response, ping response, or updates fail
• High buffer failures

If you are able to connect to the router, then you can use the show process cpu (for Cisco routers) command to check if CPU utilization is high due to interrupts or processes.

Cisco provides two great documents on Troubleshooting High CPU Utilization and Troubleshooting High CPU Utilization Caused by Interrupts.

Switch/Router Memory Utilization

Memory is a limited resource on all network devices and must be controlled and monitored to ensure that utilization is kept in check.  A memory allocation failure means either the network device has used all available memory or the memory has fragmented such that the device cannot find a usable available block.  The symptoms of memory allocation failure include, but are not limited to:

• A memory related console or log error message ( “%SYS-2-MALLOCFAIL: Memory allocation of 1028 bytes failed from 0x6015EC84, Pool Processor, alignment 0″ in the case of a Cisco router)
• Refused Telnet sessions
• The show processor memory command is displayed no matter what command you type on a console
• No output from some show commands
• “Low on memory” messages
• The console message “Unable to create EXEC – no memory or too many processes”
• Router hanging, no console response.

Possible causes of memory failure include:

• Memory Size Does not Support the Cisco IOS Software Image
• In Processor Memory (“Pool Processor” on all platforms)
• Large quantity of memory used for normal or abnormal processes
• Memory fragmentation problem or bug
• Memory allocation failure at process = <interrupt level>
• Memory leak bug

In Packet Memory

• Not enough shared memory for the interfaces
• Buffer leak bug
• Router running low on fast memory

For additional detail and troubleshooting steps for Cisco routers, see Troubleshooting Memory Problems

Interface/Bandwidth Utilization

Interface Utilization Using SolarWinds
       Free Real Time Bandwidth Monitor

Before you start digging into the gory details of your router interfaces, it is best to simply monitor the overall network bandwidth utilization to determine if you even have a problem.  Numerous open source or free tools from network management suppliers exist in the market that greatly simplify the process of gathering bandwidth utilization data and presenting it in an easy-to-consume graphical format.  SolarWinds free Real-Time Bandwidth Analyzer is an example of a commercially developed free tool that displays network device interface utilization.

If you determine that you have a problem then you will want to get detailed information about the interface on your router.  On Cisco routers, you can view the information about a particular interface using the “show interfaces” command:

If you identify an interface that has high utilization, you can take the appropriate steps to reconfigure or load balance your system.

Stay tuned for future articles where I will provide similar tips and tools for troubleshooting device configurations, traffic/bandwidth issues, and IP address issues.

By Brad Hale, Product Marketing Principal for SolarWinds.  SolarWinds (NYSE: SWI) provides powerful and affordable IT management software to customers worldwide.

Telecom: Lowering Total Cost of Telecom (TCT) vs Cost Savings

   Office image via Shutterstock

In the telecom industry you hear a lot about cost savings. Everyone out there, whether it be telecom carriers, vendors, consultants, Telecom Expense Management firms, mobility management firms, telecom agents, or other providers are all competing for the decision maker’s time to ‘reduce costs’, ‘lower costs’ and ‘reduce expenses’. While reducing costs is a common term, lowering Telecom Total Cost of Ownership (TCO) can become a more powerful phrase when speaking in terms of Telecom Outsourcing, or Total Cost of Telecom (TCT).Currently, the telecom industry is full of providers talking about ‘cost savings’; take the following claim for example:

“We will perform an audit and reduce your costs.” Or “You will realize x percentage of cost savings on a monthly or annual basis for your telecom services.”

Considering the financial impact of so many variables of telecom service, maintenance, network administrative, equipment, and network management costs, It can be hard to distinguish what they really mean and what the result will ultimately be in the end.The term “cost savings” is usually a one time event and for a specific telecom service: doing an audit of your carrier contracts for mobility and wireline services, obtaining service credits from the carriers, optimizing rates to be more market competitive, or issuing credits for services no longer used or needed. Cost savings for mobility includes negotiating better rate plans for mobility, optimization of data plans, security, policies, and applications.

When we talk about outsourcing telecom management, just as you would to outsource any other business function (IT, supply chain, accounting, HR, or payroll, etc), the term “lowering cost of telecom ownership” is a more viable outcome. Outsourcing telecom management involves lowering your total cost of ownership, which relates to lowering the cost of managing your telecom services.

The challenge companies have in managing telecom is that they require the people, processes, technology, and time to manage services. For example, processes and people may include implementing a support desk or help desk to adequately support mobility users, or for order and incident management. Technology includes having a platform and controls in place to manage the financial and inventory aspect of telecom. Reporting and metrics also fit into this category.

What does Total Cost of Telecom mean to you as an executive? Whether you are a CIO, CFO, or other executive, you may have an idea of what you spend per month/annually for telecom services related to:

• Wireline services: local voice, internet, WAN, Long distance, disaster recovery/business continuity, hosting, cloud computing, SIP, VoIP, etc.
• Mobility: individual rate plans, corporate vs individual liability, device management, applications and data plans, security, texting, etc.

AOTMP (www.aotmp.com) research indicates the top reasons why enterprises use an outside supplier for telecom management activities include:

• 73% lack tools and technology
• 41% lack internal resources
• 38% lack efficient processes & procedures
• 36% lack of internal expertise
• 20% more cost effective to outsource

The difference between lowering the “total cost of telecom” and providing “cost savings” is simple: if you manage telecom services in-house, there are costs associated with managing these services. For example, here are a few in-house costs associated with managing telecom:

1. FTE labor: Full time employees-salaries, bonuses, benefits, travel expenses.
2. PTE labor: Part time employees/consultants-hourly rates, weekly expenses, travel expenses, bonuses, any other benefits.
3. Time: How much time per week or per month do you spend managing telecom? What does this mean? How many hours do you spend on the phone providing help desk support for end users (mobility), support desk to manage mobility, help desk to address and issue trouble tickets when a location’s service is down? Mobility continues to be a growing expense with the amount of remote users. How many tickets do you support each month for mobility? How many hours per month do you provide support for mobility users? Applications? Security? Creating policies?
4. Resources: How much do you spend per month, or annually, on technology and tools to manage telecom (ie portals, expense management software, help desk software, customer service tracking software)? This also includes upfront costs and time to implement, as well as monthly user licenses
5. Downtime: If you have poor service, how much downtime is your business experiencing and how much time does it take to respond to outages and resume service? During these down times, how much is it costing your business per hour, per month when you experience service outages? How many service outages have you had in the past month?
6. Equipment & Maintenance: How much are you paying for user fees and license fees for voice services, PBX, and phone system equipment? Other licenses?
7. Network monitoring and Maintenance: Monthly fees for monitoring and maintaining the network, staff to maintain it.
8. Administrative costs: Software and portals for invoice processing (avg processing cost is $70 per invoice), asset management, expense management technology, etc

These are just a few examples related to some areas to lower the cost of ownership of managing telecom. Whether you manage telecom internally, or use a provider for telecom management, it is critical to your business to fully understand all of the costs involved regarding time, people, processes, and technology.

This may be a matter of terminology or small detail overall, but it is important to understand the concept of total cost of telecom when choosing a supplier of telecom management services.

What is your Total Cost of Telecom and how do you Measure Up?

What is Software-Defined Networking (SDN) and How Will it Impact the Network Administrator?

What is SDN?

Software-Defined Networking has become one of those industry buzzwords that quickly gets so over-used and distorted that it is hard to tell what it really means or if it really matters.  In this blog post I’ll try to go back to the basics of what is SDN and how it is likely to impact the network administrator.

SDN was defined by the Open Networking Foundation (ONF), a standards group founded in 2011 by Deutsche Telekom, Facebook, Google, Microsoft, Verizon, and Yahoo!, as an architecture that “brings direct software programmability to networks worldwide.”  ONF is pushing this goal by defining and driving an OpenFlow interface.  The ONF website describes OpenFlow as follows:

“In a classical router or switch, the fast packet forwarding (data path) and the high level routing decisions (control path) occur on the same device. An OpenFlow Switch separates these two functions. The data path portion still resides on the switch, while high-level routing decisions are moved to a separate controller, typically a standard server. The OpenFlow Switch and Controller communicate via the OpenFlow protocol, which defines messages, such as packet-received, send-packet-out, modify-forwarding-table, and get-stats.”

So the basic concept is pretty simple, move the control functions into a software layer that talks to the hardware layer which still moves the data.  The key benefits of this approach, especially if all hardware devices used the same interface protocol, is that it would be much easier to programmatically provision network resources and automate configuration tasks at a common software control layer.

This is largely what happens today in server virtualization.  A software virtualization layer interfaces with the applications and manages the underlying physical compute resources (e.g., CPU, memory, disk, etc.).  This abstraction has enabled a number of key capabilities including rapid provisioning, VM/application mobility, thin provisioning of storage, etc.

Big Vendor Positioning

The idea of abstracting the intelligence from the hardware naturally leads to the conclusion that the network virtualization layer would then be able to run on commodity networking hardware.  This is not a scenario that is likely to be very appealing to Cisco and the other big networking hardware vendors.  Looking at a parallel example in server virtualization, HP, Dell, IBM and others still sell a lot of servers even though VMware provides the most common virtualization layer.  However, it is getting harder and harder to maintain any differentiation at the hardware layer which is driving commoditization and squeezing profit margins – not an appealing vision for Cisco and the other vendors who would be happy with the status quo.  Larry Dignan’s article “Cisco makes its software defined network case: Do you buy it?” does a nice job of diving into the Cisco perspective in more depth.

In July 2012 VMware stepped into the picture in a big way with their acquisition of Nicira who has been focused on network virtualization using an SDN approach.  VMware is the dominant player in the server virtualization market so when they acquired Nicira for $1.05B it caught many people’s attention.  At VMWorld 2012 in San Francisco this past August, VMware continued to hit the theme of a Software-Defined Data Center that encompasses servers, storage and networking.  The positive side of VMware’s interest in SDN is that they have enough market presence already in the virtualization space that the other big players now have to be serious about moving forward with a SDN strategy of their own or risk losing their existing markets to VMware.  On the other side, VMware already has enough market power with their server virtualization capabilities.  It is scary to think about what they could do if they are able to successfully extend that dominance across both storage and networking as well.

So what should we expect from the big vendors like Cisco, VMware and others?  The typical approach for those vendors would be to talk a lot about their SDN strategy, claim open SDN capabilities within their products and pursue activities that have some SDN capabilities while keeping it proprietary to their product line so that they can maintain control of the customer.  With the existing VCE partnership between VMware, Cisco and EMC it will be very interesting to see how this dynamic plays out across network and storage.

How will this Impact the Network Administrator?

If history is any guide, the first challenge the Network Administrator will have to face is wading through the proprietary vendor features or capabilities marketed as SDN.  The admin will need to determine where there is real value, what is designed strictly to lock you in to a vendor, and what is just marketing hype.  In the longer term, SDN has the possibility of being a valuable capability that enables a transition to a more flexible and efficient network similar to what has happened with server virtualization.  With standardized, open SDN capabilities, the network admin would be able to programmatically automate key provisioning, configuration and management tasks.  This would allow more time to focus on business requirements and overall network optimization and less on device-level tasks.   As this transition occurs, probably over the next two to five years, the network admin will want to make sure they are at least keeping up with those changes.  By updating skills they can become the equivalent of the VMware (or Hyper-V, KVM, etc.) admin that can manage the physical server too instead of just being the legacy server admin.

Drawing another parallel, as the network virtualization layer does more, it will be harder to connect what is happening at the application layer down through the virtualization layer to the physical infrastructure.  The network administrator will want to make sure that they can monitor the network and maintain visibility from the application and virtualization layer down into the physical infrastructure.  This visibility enables both system troubleshooting and optimization.   For optimization it ensures that physical capacity aligns with the virtualization demands.   For troubleshooting you can quickly understand where network traffic monitoring or other analysis is needed to identify where the real problems are occurring and what the impacts are.  Keeping that end-to-end mapping capability from the network hardware to the network consumers will ensure that the network admin remains a critical part of the overall data center team.

While this change is likely to happen over years instead of months, it is always good to use change to your advantage as opposed to being caught napping.

By Mike Thompson, SolarWinds Networking Product Marketing Manager; on twitter @michl121

When did ‘productivity’ get lost in the BYOD discussion?

     BYOD image via Shutterstock

Employees are more mobile than ever – they can work from just about anywhere they choose, and need less and less equipment in order to be productive.  In addition to mobility, the convenience of using fewer devices to manage personal and professional tasks is fast becoming the norm for many employees, preferring one mobile device suited for all needs.  Organizations are more challenged than ever to control, track and manage the amalgamation of mobile devices, service plans, and the ever expanding ocean of apps. Because of this, wireless expense management has become far more complicated than in decades past, when there were far fewer devices and plans to manage and where it was fairly common for the enterprise to adopt a top-down approach to wireless expense management.

BYOD, or ‘bring your own device’ is a program whereby employees utilize their own mobile devices to perform their day to day duties, while granting them access to necessary corporate information.  Email, calendars, and required apps are made available to employees via their mobile devices, eliminating the need to be tied to their office or requiring them to use company-issued equipment.

Because of the recent consumerization of IT, the need for an effective BYOD policy is more evident than ever, as many employees use their own devices and laptops for work regardless of whether or not they receive support from their IT departments.  An effective and strictly enforced BYOD policy will help mitigate the security risks this type of usage evokes.  While BYOD programs have certainly enhanced mobility’s value to an organization, it has without question complicated the issue of productivity.

The arguments behind the BYOD trend suggest that employees are more productive by using their own device, cost is reduced because companies do not pay for the devices, and corporate IT departments do not have to devote resources toward the management of these devices[1].

However, when faced with the fact that more than 60% of companies do not even track the personal usage for their employees’ devices, productivity becomes objectionable[2]. Thus, employees could be claiming any number of calls or minutes as reimbursable expenses and management wouldn’t be the wiser.  If left unchecked, this lack of verification could have a cumulative, detrimental economic affect to the corporation.

In addition to the expense management issues, BYOD policies can expose a plethora of security liabilities often overlooked by management.  In the unlikely event of a lawsuit where phone records need to be subpoenaed, the possibility of acquiring them is slim if an employee is using his or her own device.  Since most companies reimburse employees for their business-related mobile usage, the likelihood of possible record falsification exists.  In addition, the process of reimbursing an employee is a soft cost that needs to be added onto the actual cost of the reimbursable expense.  According to a recent Aberdeen Group study, cellular telephone monthly reimbursement averages $70 per month, which is only $10 less than what average companies pay for wireless expenses[3].  In addition, most reimbursements occur monthly through expense report processing, which costs an organization on average $29 to process, causing the expenditure to climb over the industry average.

To ensure productivity amongst employees who adhere to BYOD, corporations require strict and specific policies.  A well thought out BYOD policy must address the issue of security.  Whether they like it or not, employees should be required to create screen-lock passwords on any personal device that will be used to access corporate data.  With just a swipe of the screen, an employee could unknowingly be exposing sensitive, confidential corporate information.

The issue of IT support for employee-owned devices can be complicated.   This is where productivity can be stalled completely if an employee has an issue that he or she cannot resolve on his or her own.  If it is a network or connectivity issue, will there be corporate IT support?  What happens in the case of damaged hardware or broken equipment issues, or if the device is lost or stolen?  Will IT support those types of problems, or is the employee required to resolve it on his own?  Can the employee count on management to provide a ‘loaner’ device while their own is being repaired, or is the employee essentially out of commission until the original device is fixed?  While these questions are debated, productivity will universally suffer.  Not only is the employee affected, the IT department and management will need to devote time to fixing the issue so the employee can get back to work as quickly as possible.

To what degree will the corporation support the device in a BYOD environment?  Since the combination of personal and professional programs and apps will exist on the device, what happens if a personal app creates a problem that interferes with the ability to access email and other company information?  An effective BYOD policy should outline what apps will be allowed and which will be banned due to any potential security or legal risks whatsoever.  What if a poorly written instant messaging client steals your organization’s address book? These are serious questions to address in a BYOD policy[5].  Who owns the apps, programs, information and the device?  Ownership becomes cumbersome in a situation where the device is lost or stolen and corporate IT needs to wipe the device clean of all information – including the employee’s personally stored information.  Additionally, if an employee leaves or is terminated, who owns the data?  If a BYOD policy is adopted, it must be clearly defined that the employee agrees to the consequences if data security is jeopardized in any way.

The question of whether or not productivity is enhanced with a BYOD policy is not easily answered.  In an environment that has a strict policy surrounding the usage of personal devices is much better poised for increased productivity than those corporations with loosely enforced guidelines.  An effective BYOD policy will address security, clearly define allowable apps, and outline support processes to keep an employee as productive as possible.