Up to this point I’ve resisted writing about last week’s Skype outage, simply because I found it hard to see clear security implications. I could see no solid evidence that the outage was caused by malicious activity. It simply looked like a glitch in Skype’s back-end machinery that was triggered by a botched Microsoft patch rollout.
Sure, people have been frustrated by the outage, especially in the blogosphere. But much of what I read looked more like groans of the inconvenienced than fears that bigger security issues were involved.
But in the final analysis, I think this incident does illustrate the kind of trouble we could see in the future at the hands of malicious people, so it’s time to add my two cents to the bigger discussion.
First, some background:
Skype’s immensely popular Internet phone service crashed on Aug. 16 and stayed that way for two or three days. The folks at Skype posted a message on the company’s blog blaming the outage on a botched deployment of some Microsoft patches.
“The disruption was triggered by a massive restart of our users’ computers across the globe within a very short timeframe as they re-booted after receiving a routine set of patches through Windows Update,” wrote Skype’s Villu Arak. “The high number of restarts affected Skype’s network resources. This caused a flood of log-in requests, which, combined with the lack of peer-to-peer network resources, prompted a chain reaction that had a critical impact.”
A lot of Skype users didn’t buy that, of course, and Arak came back with a blog post clarifying the official Skype position.
“We don’t blame anyone but ourselves,” he wrote. “The Microsoft Update patches were merely a catalyst — a trigger — for a series of events that led to the disruption of Skype, not the root cause of it. And Microsoft has been very helpful and supportive throughout.”
The Microsoft Security Response Center’s Christopher Budd weighed in with his own blog posting, saying that Skype did what any affected party should do when encountering patch deployment problems — contacted the Microsoft support center. “Fortunately, Skype has identified the cause,” Budd wrote. As Villu Arak notes, a previously unseen software bug within the network resource allocation algorithm was the cause, and they have corrected it.”
The second Skype posting wasn’t enough to satisfy tech blogger Ben Metcalfe, who wrote that the official explanation has too many holes for his liking.
“Having blamed Microsoft windows updates for the collapse of the Skype network, the beleaguered p2p VoIP company has spun another yarn now ‘clarifying’ that it’s not really Microsoft’s fault after all,” he wrote. Their second explanatory post contains more hot air than a dodgy data center with a broken air conditioner.”
He urged readers to inspect both Skype posts and see the contradictions he found, including:
“The disruption was triggered by a massive restart of our users’ computers across the globe within a very short timeframe as they re-booted after receiving a routine set of patches through Windows Update.”
“How come previous Microsoft update patches didn’t cause disruption? That’s because the update patches were not the cause of the disruption.”
Metcalfe said it all looked odd, given that every Microsoft update requires a restart. “There was nothing different with this latest windows update on that front,” he noted. “Thus to say that the reboots caused the outage makes no logical sense without the addition of a further factor (which they don’t appear to be disclosing).”
In the SecuriTeam blog Juha-Matti offered up eight reasons why the security community responded so skeptically to the official Skype line:
1. Microsoft has released monthly security updates since January 2004.
2. There were three critical MS patches in July, and four critical in June.
3. Only four August critical patches included a mandatory reboot.
4. Critical patch MS07-044 for the code execution issue in Excel needs no reboot.
5. Critical patch MS07-050 for the VML issue needs a reboot only if files are in use.
6. SecurityLab.ru released a public Skype Network Remote DoS Exploit on Aug 17.
7. There was a new Skype for Windows version 220.127.116.11 out on Aug 17.
8. A lot of home users go to Microsoft Update on Tuesday, not Thursday.
Ed Felten, professor of computer science and public affairs at Princeton University, offered Skype a little sympathy in his Freedom to Tinker blog:
“To deal with the ever-changing population of user computers, Skype has to use a clever self-organization algorithm that allows the machines to organize themselves without relying (more than a tiny bit) on a central authority,” he wrote. “Self-organization has two goals: (1) the system must respond quickly to changed conditions to get back into a good configuration soon, and (2) the system must maintain stability as conditions change. These two goals aren’t entirely contradictory, but they are at least in tension. Responding quickly to changes makes it difficult to maintain stability, and the system must be engineered to make this tradeoff wisely in a wide range of conditions. Getting this right in a huge P2P system like Skype is tricky.”
He cautioned against making too many broad conclusions from a single failure like this, since “large systems of all kinds, whether centralized or P2P, must fight difficult stability problems.”
Regardless of how this outage occurred, there’s a bigger lesson to be had. We’ve written many stories in the last three years about the dangers of VoIP technology, and the lesson is usually that IT shops are rushing this technology into use without considering the security implications.
Former White House cybersecurity advisor Howard Schmidt expressed those concerns when I interviewed him for a podcast earlier this year, and SANS Institute Training and Certification Director Stephen Northcutt admits organizations like his must do more to train people in the art of VoIP security.
VoIP security was also a dominant issue at the Black Hat conference in Las Vegas earlier this month.
The bottom line is that VoIP security has to be top of mind now in any corporate IT operation. The Skype outage may not have been the work of an attacker, but I’m sure the bad guys were inspired by the public outcry that ensued. That inspiration could lead to more sinister activity going forward.
About Security Blog Log: Senior News Writer Bill Brenner peruses security blogs each day to see what’s got the information security community buzzing. In this column he lists the weekly highlights. If you’d like to comment on the column or bring new security blogs to his attention, contact him at firstname.lastname@example.org.