Your <redacted> Server was found to be part of a network of compromised machines
leading a Distributed Denial-of-Service Attack (DDoS Attack) against other servers.

*******************************************************************************
IMPORTANT: In order to prevent further criminal activity from your <redacted> Server,
we have suspended access pending an investigation and resolution.
*******************************************************************************

The logs they sent me show UDP packets indicating that this could be part of a DNS amplification attack. Take a look:

Please see the firewall logs below for details:
1365103763.526228 IP xxx.xxx.111.16.44223 > xxx.xxx.149.195.80: UDP, length 1
1365103763.526232 IP xxx.xxx.111.16.44223 > xxx.xxx.149.195.80: UDP, length 1
1365103763.526234 IP xxx.xxx.111.16.44223 > xxx.xxx.149.195.80: UDP, length 1
1365103763.526236 IP xxx.xxx.111.16.44223 > xxx.xxx.149.195.80: UDP, length 1

That’s all I know for now. I have to contact the provider, open a window of time to gain access, and secure the server. I’ll keep you posted.

Source: Vistnet.com

If you have noticed a bit of sluggishness on your internet connection in the past week or so, it could be due to the most massive DDoS attack ever recorded. Here’s what’s happening according to Naked Security:

A little over a week ago a questionable internet hosting provider in The Netherlands called Cyberbunker took umbrage with SpamHaus, an non-profit organization that was founded in 1998 to take on spammers and the internet hosts who profit from their activities.
…
How big is the attack? At times it has been reported to be as large as 300 gigabits per second. Traditionally even large botnets are only able to deliver hundreds of megabits or a few gigabits per second.

Ouch! That’s huge. It seems that many primary internet backbones (“tier 1 service providers”) are being overwhelmed by the volume of traffic. That’s why you may have noticed the slowdown on the internet. I certainly did, but since it was most prevalent where I work, I didn’t think much of it. Our bandwidth is always strained when school is in session. I did find it a bit odd that my home connection seemed sluggish. It all became clear with the report of the DDoS attack.

So, if large botnets aren’t capable of delivering such a volume of traffic, what is causing it? It’s a large scale DNS amplification/reflection attack taking advantage of misconfigured DNS servers that will allow anyone to query them without any filtering or rate-throttling. It’s a huge problem as there are reportedly more than 21.7 million such servers online (Open Resolver Project). A Microsoft TechNet article provides a high-level summary of this type of attack:

A DNS amplification attack (aka DNS reflection attack) is a type of distributed denial of service (DDos) attack that takes advantage of the fact that a small DNS query can generate a much larger response. When combined with source address spoofing, an attacker can direct a large volume of network traffic to a target system by initiating relatively small DNS queries.

I’ll leave it at that for now. I plan to give a more detailed analysis in a future post.

Brian Krebs worked as a reporter for The Washington Post from 1995 to 2009, authoring more than 1,300 blog posts for the Security Fix blog, as well as hundreds of storiesfor washingtonpost.com and The Washington Post newspaper, including eight front-page stories in the dead-tree edition and a Post Magazine cover piece on botnet operators.

According to Dan Goodin at Ars Technica, Krebs is known for work that includes:

• “Exposés [that] completely shut down a California hosting service that coddled spammers and child pornographers and severely disrupted an organized crime syndicate known as Russian Business Network” and, more recently,
• “Investigative journalism that followed the money to the people who sell malware exploit kits, illicitly procured credit reports, and denial-of-service services  in underground forums.”

It’s hardly a surprise that he has made enemies in the cybercrime underworld. Last week, some of those enemies attacked him. Writing in a March 13 blog post, he described what happened:

It’s not often that one has the opportunity to be the target of a cyber and kinetic [armed -Ed.] attack at the same time. But that is exactly what’s happened to me and my Web site over the past 24 hours. On Thursday afternoon, my site was the target of a fairly massive denial of service attack. That attack was punctuated by a visit from a heavily armed local police unit that was tricked into responding to a 911 call spoofed to look like it came from my home.

Fortunately, everything turned out fine, but the incident serves to illustrate that cyber-criminals–Krebs calls them cowards–are very jealous of their turf and will retaliate against those who they believe have violated it.

If you enjoy reading about real-life attacks via cyberspace, you should check out the post here.

[kml_flashembed movie="http://www.youtube.com/v/K13nutRdlvE" width="425" height="350" wmode="transparent" /]

Various sources, including CNN and CNet, suggest that a Georgian blogger with accounts under the name “Cyxymu” (a town in the Republic of Georgia) on the services was targeted. The date of the attack coincides with the one year anniversary of the Russia-Georgia conflict.

Other sources, including The Register suggest that a JoeJob was the main source of the attack.  Joejobs are spam messages designed to induce someone to click on a link in the hopes that enough people will do so, thereby harming the site being linked to.

Still others blame a conventional DDoS attack using botnets, but Arbor Networks‘ analysis actually shows a drop in traffic volume hitting Twitter during the alleged DDoS attack, leaving doubt that this method was used.

I’ve also seen reports blaming hackers angry at Twitter for becoming more popular than IRC, a vigilante trying to point up the danger of botnets, and cyber-terrorists.

Seems no one really knows for sure at this point.

Ongoing denial-of-service attack

We are defending against a denial-of-service attack, and will update status again shortly.

Update: the site is back up, but we are continuing to defend against and recover from this attack.

Update (9:46a): As we recover, users will experience some longer load times and slowness. This includes timeouts to API clients. We’re working to get back to 100% as quickly as we can.

Update (4:14p): Site latency has continued to improve, however some web requests continue to fail. This means that some people may be unable to post or follow from the website.

As of late yesterday morning communication with the API and SMS was still down.

As usual, there always seems to be some humor in these situations. Here’s a comment by John Pescatore of SANS Institute from the SANS News Bites:

[Editor's Note (Pescatore): Wow, 2 hours without tweets! That's like a
car drive to the shore without anyone in the back seat saying "Are we
there yet? I see a rock. Is that a seagull? I like saltwater taffy.
Shaquille Oneal is really tall. Are we there yet?" the entire trip.]

The following Security Advisories are addressed in Firefox 3.0.7:

• Mozilla Foundation Security Advisory 2009-07: “Mozilla developers identified and fixed several stability bugs in the browser engine used in Firefox and other Mozilla-based products. Some of these crashes showed evidence of memory corruption under certain circumstances and we presume that with enough effort at least some of these could be exploited to run arbitrary code.”
• Mozilla Foundation Security Advisory 2009-08: “An anonymous researcher, via TippingPoint’s Zero Day Initiative program, reported a vulnerability in Mozilla’s garbage collection process. The vulnerability was caused by improper memory management of a set of cloned XUL DOM elements which were linked as a parent and child. After reloading the browser on a page with such linked elements, the browser would crash when attempting to access an object which was already destroyed. An attacker could use this crash to run arbitrary code on the victim’s computer.”
• Mozilla Foundation Security Advisory 2009-09: “Mozilla security researcher Georgi Guninski reported that a website could use nsIRDFService and a cross-domain redirect to steal arbitrary XML data from another domain, a violation of the same-origin policy. This vulnerability could be used by a malicious website to steal private data from users authenticated to the redirected website.”
• Mozilla Foundation Security Advisory 2009-10: “libpng maintainer Glenn Randers-Pehrson reported several memory safety hazards in PNG libraries used by Mozilla. These vulnerabilities could be used by a malicious website to crash a victim’s browser and potentially execute arbitrary code on their computer. libpng was upgraded to a version which contained fixes for these flaws.”
• Mozilla Foundation Security Advisory 2009-11: “Mozilla contributor Masahiro Yamada reported that certain invisible control characters were being decoded when displayed in the location bar, resulting in fewer visible characters than were present in the actual location. An attacker could use this vulnerability to spoof the location bar and display a misleading URL for their malicious web page.”

Everyone should immediately upgrade to Firefox 3.0.7 to mitigate these issues.

Steve Gibson of Gibson Research Corporation presents a good sampling of the news surrounding this issue. There’s a lot that is (and isn’t) being said. The bottom line is that it’s a nasty vulnerability. It’ll be interesting to see how this develops.