An excellent post on the science (and definitely not art) of network troubleshooting on the PacketLife blog last week, resulted in a mini-debate on whether network change andconfiguration management is a lifesaver or a time-sucking burden for network admins. The answer, it appears, is probably somewhere in the middle.
PacketLife blogger Jeremy Stretch runs through his network troubleshooting method, which includes NOT starting the process at Layer One as many do, but also involves detailed recording of problems and their solutions as well as redoing tests numerous times to confirm functionality after the fix is implemented.
The idea of redoing tests for one reader was laughable considering he has to wade through a river of paper work in order to do even one test.
“The increasing need to adhere to strict change control procedures kills the science of troubleshooting. In my world one test would require mounds of paperwork and numerous sign offs. To do my job I’m forced to do things under the table and hope I don’t break anything and call attention to my activities,” wrote the reader, who calls himself/herself PompeyChimes.
Those complaints brought on an outraged response from a reader known as HH.
“For the love of god, use proper change management procedures… Too often are problems caused by hotshot admins who think they know everything,” HH wrote.
Stretch tempers the argument with the following middle road response:
“HH makes a valid point. Change controls are great – IF they’re implemented practically. So long as they leave an engineer enough room to maneuver, they can be an excellent tool to help generate documentation during the troubleshooting process.”
We’ve written a number of pieces on the virtues of change management in virtualization and change management in storage, but much more often than not we hear of the nightmares involved in dealing with change management. The answer probably doesn’t lie in doing away with change control, but instead in implementing procedures that are realistic for the admins carrying them out daily.
Network Computing blogger Howard Marks made some good points recently about why Brocade has struggled to sell the Ethernet networking product line it acquired from Foundry back in the summer of 2008. As Marks points out, Brocade tried to sell sell Foundry products in the same way it has traditionally sold its storage networking products: via OEM agreements with big server and storage vendors like IBM and Dell. But networking pros aren’t much interested in buying networking products from server vendors. They prefer going with someone they know, such as Cisco, ProCurve or… Foundry.
Wall Street has been displeased with Brocade’s Foundry results so far. As Munjal Shah, analyst with Jefferies and Stifel Nicolaus told the Wall Street Journal:
Brocade is facing challenges in integrating the Ethernet [business] as the sales model is different and Ethernet [original equipment manufacturers sales] are slow to materialize. Brocade has solid position in data center and relative valuation is low, but we believe it will take some time to resolve the execution issues.
Brocade has responded by appointing John McHugh as its new chief marketing officer. McHugh is a veteran of HP, where he is credited with starting up the ProCurve division. More recently McHugh was the head of Nortel’s enterprise solutions business. No surprise that he’s jumping ship after the Avaya acquisition. Burnishing the Foundry business appears to be a nice challenge for him.
Marks says Brocade also got away from what made Foundry a modest success in a crowded networking market: good support from sales engineers. Brocade tried to monetize those resources by turning what used to be free support into professional services. This alienated existing customers, apparently. Now Marks says he’s hearing from internal sources that Brocade is going back to the old Foundry approach, which should help it win over some new customers and perhaps retain some existing ones.
Market research firm dell’Oro Group has published its latest quarterly market update on the wireless LAN industry. According to the firm, the market hit an all-time high in the fourth quarter of 2009. The ratification of 802.11n has really set this market on fire. Apparently IT organizations in the retail, education, healthcare and hospitality sectors are all spending a ton of money on new wireless LAN infrastructure right now.
This is driving a lot of revenue growth, but some vendors are reaping the benefits more than others. I asked dell’Oro analyst Loren Shalinsky for detials.
Cisco remains number one in the market by a huge margin, Shalinsky said. But Cisco did not have a good quarter. Its wireless LAN market share shrank by about four points he said, and revenue was down for the quarter (Shalinsky didn’t say by how much).
Motorola had an awesome quarter, growing by 40% sequentially from the third quarter, he said. The growth spurt nearly helped it overtake Aruba Networks as the number two vendor for enterprise wireless LAN. Aruba’s revenue grew by 7% in the same period. Shalinsky said total product revenue for the fourth quarter was $42 million for Aruba and $40.5 million for Motorola. Of course, Aruba would point out that it is also selling quite a few products through it’s OEM relationship with Alcatel-Lucent, which saw its revenue grow by 30%. Alcatel actually overtook Meru Networks in market share and claimed the number five position. (HP ProCurve is holding steady at number 4).
The Ponemon Institute recently surveyed 155 globally certified PCI DSS compliance auditors about how the largest retailers (Tier 1 merchants) are doing with respect to compliance with the credit card industry’s cardholder data security requirements.
Asked by Ponemon to rank the effectiveness of technologies used to protect cardholder data, auditors identified encryption of data at rest and in motion, firewalls and endpoint encryption as the best technologies. Least effective were ID & credentialing systems, intrusion protection and detection systems (IDS and IPS), and website sniffers and crawlers. Ponemon’s research didn’t explain why auditors felt this way about the various technologies. A systems administrator at a nonprofit recently told SearchNetworking.com that his organization is looking at segmenting its network with VLANs to help implement the controls it needs for compliance.
Also, the corporate network is the MOST vulnerable infrastructure element to a potential data breach, auditors said. Fifty-one percent of auditors identified corporate networks as a weak point. Corporate databases (43%) were the second most vulnerable. Only 10% considered unattended payment terminals as a vulnerability.
Ponemon also revealed that the average Tier 1 merchant spend about $225,000 on its compliance audit, but it didn’t identify how much these company’s spend on operations and technology. Auditors said that business units are the most likely (40%) part of a company to be responsible for auditing PCI compliance, but they unlikely to own responsibility for delivering that compliance (19%). IT security (30%) and the office of the CIO (10%) combine to own a plurality of compliance responsibility. This division of responsibility between compliance and auditing could create some tension between IT and business units.
As I was skimming through stories from the RSS feed of a competing publication, I came across these two sequential headlines:
“Want a job? Get a Computer Science Degree”
“Boeing prepares to cut to cut nearly 800 IT workers”
Talk about mixed messages. But that’s what this economy has been giving us for a couple years now, hasn’t it?
IBM gobbled up Intelliden this week. Intelliden bills itself as a provider of “intelligent network automation solutions.” Basically it provides automation around network-based compliance and network change and configuration management. It’s also been focusing on aligning these technologies with cloud computing, helping both enterprises and service providers automate the management of cloud networks.
IBM will likely jam Intelliden into its monstrous Tivoli IT management suite. I assume it will get folded into Tivoli’s vast armada of Change & Configuration products. Or maybe it will be absorbed into the horde of Network Management & Performance products. It’s hard to tell. Just delving into those product choices is overwhelming. Trying to figure out where Intelliden gets placed among them is a task that’s beyond me.
I’ve been covering the networking industry for about two years now, and I don’t think I’ve ever been approached by IBM PR regarding the network management capabilities of Tivoli. Come to think of it, most of the Big Four IT management companies don’t seem to have me on their radar. Only CA actively sends me news on network management technologies. On the other hand, when I wrote for SearchCIO.com I received pitches from Tivoli fairly regularly. Does that say something about IBM’s Tivoli strategy? Market and sell to CIOs, not network managers and network engineers.
Given the choice, would you buy your network management tools from one of the Big Four or from independent vendors like NetScout, SolarWinds, Fluke, etc. And what do you do when your favorite vendor gets gobbled up, whether it’s Intelliden by IBM or NetQoS by CA?
Late last month SolarWinds acquired Profiler, a storage and virtual server management software technology from fellow Texas vendor Tek-Tools for $42 million. The technology provides visibility across storage, servers and virtual server environments.
On his blog ManageEngine vice president of product marketing Girish Mathrubootham claims that SolarWinds has been on a buying spree in recent years, acquiring a handful of companies like Tek-Tools, Kiwi and ipSurveyor in order to add functionality to its flagship Orion product suite. Mathrubootham argues that this is the approach that the Big Four have taken, creating management platforms that are huge, complex and pricey. He says:
Acquiring disparate products and integrating them at a GUI level may provide short-term boosts to revenue, but it is exactly this kind of headache that customers hate when dealing with the Big 4. (And SolarWinds would know this better than anyone else )
It’s true that SolarWinds has been acquiring point tools that expand the visibility afforded by its network management and application performance monitoring platform Orion. However, is SolarWinds really hoping to become a tool of choice to server admins and storage managers? SolarWinds says it is making these acquisitions in response to what it hears from customers. The roles in the data center are changing and the tools that support these roles have to evolve, too.
Sanjay Castelino, vice president of product marketing & product management at SolarWinds says it all comes down to virtualization. The migration of virtual machines to migrate from host to host requires network managers to look beyond their organizational silos.
One of the things we seeing within the IT organization is this need for visibility across the network, the compute infrastructure, virtual machines, storage and applications in the data center. We looked at the pieces that we have in that: We have the network piece and the application monitoring piece and some server and virtual machine monitoring. Storage was an important piece to add to that mix.There is an overwhelming demand from our customers to get visibility into network, storage and virtualized environments.
You’re starting to see that these folks are starting to work more closely together. You can’t say “I’m only going to be a networking guy and I don’t have to talk to my peers in servers and storage.” That’s going away. These people have to get shared visibility so that when they make changes on one they don’t bring down pieces of the puzzle in terms of delivering applications. The networking guys own the networking piece and the storage guys own the storage piece, but all of them are wanting visibility across their domains into the other domains because they aren’t siloed anymore.
Is SolarWinds truly morphing into one of the Big Four? Will we have to start calling them the Big Five? Given that Orion is still a product that can be downloaded online, the complexity that ManageEngine is warning about seems pretty far off.
So, you heard that Cisco is cracking down on the use of IOS emulators by requiring a license registration key to activate the software on every router in its IOS v.15 update.
Angry? Want to stick it to the man? Feel a riot coming on?
Take a deep breath, put down the bricks and baseball bats, and fire up your Internet browser of choice to join the underground movement.
We spotted a Save Dynamips group pop up yesterday on Facebook, started by French networking student Benoit Goncalves in support of Dynamips, a free Cisco IOS emulator that seems to be the target of these IOS licensing changes. As I write this, it only has 18 members but sure looks like they’re rolling out the welcome mat for new recruits.
(Photo of riot police outside 2008 Republican National Convention courtesy of/politely borrowed from this flickr account)
Computer Economics has published its new IT salary report, which claims the average IT worker will see just a 1.8% raise in 2010. That’s pretty rough. But in an age where many people are seeing their salaries stay flat for the second year in a row, perhaps 1.8% can be seen as a silver lining.
According to Computer Economics, the higher up in the IT chain of command you are, the smaller your percentage pay increase will be. CIOs and directors are getting just 1.3% more money this year on average. The lower you are in the organization, the better your percentage increase might me. Computer Economics says this is part of an effort to retain talent in the trenches.
What’s the best job to have if you’re looking for a decent raise this year? Developers. They’re going to get about 2.1% more money this year, which is the highest average pay increase in the industry.
Networking pros are sort of at the median. Network and systems support personnel are going to see a 1.9% increase this year. This includes network admins and telecom analysts. How does your salary measure up? Are you getting a raise this year?
For those of you who don’t have a full-fledged network configuration and change management (NCCM) system, Solarwinds released a free tool this week that could make your lives a little easier. It’s called Network Config Generator.
Here’s a video where a couple of guys from Solarwinds showing how the product works.
[kml_flashembed movie="http://www.youtube.com/v/6E8D04zj45Q" width="425" height="350" wmode="transparent" /]
Basically, this tool works like a configuration wizard. You enter the parameters you want for the configuration of a class of switches, routers, etc. The tool takes this information and outputs a configuration template in Command Line Interface (CLI) code, which you can then cut and paste into the CLI consoles on any of your network devices.
It doesn’t scale that well. If you want to configure 100 switches, you have to cut and paste the CLI it into each one. But if you have some sort of network configuration management tool, you can probably drop these templates into that. Solarwinds, of course, is suggesting you use its Orion Network Configuration Management tool to apply these templates across your network.
Also, if you go to Solarwinds’ online community for customers, Thwack.com, you’ll find that Solarwinds customers are sharing their own configuration templates generated by Network Config Generator. You’re free to grab them and tweak them as you like with this free tool.