The problem with being a “geek” is that we truly love to tinker, to fix, to improve, to test….etc. So when you announce to a bunch of us that a website on the network has been broken into, there’s lots of leaping into action.
Which is exactly what you don’t want to do. At all.
While visiting a client to talk about network architecture, an engineer rushed into our room to announce that one of their websites had been hacked. We all hopped up and went out with him. (My lecture was boring, anyway.) I wanted to see what they were going to do, and if they were going to follow their own Intrusion Detection Policy. Plus, I was, like them, vastly interested.
Turns out it was a fairly generic attack, with the break-in artist simply using the website for cross-site scripting and redirection.
By the time we got there, two engineers had already been working on the web server, analyzing the code in the html, and checking other settings on the server. They took the web server offline, removed the offending code, looked at the event logs and brought it back up. All good, they said.
“Not really,” I said. “You do know that you can never trust this box again?”
“Not to be a party-pooper, but there’s no way of really knowing if a rootkit has been installed, is there? He could come back tomorrow.”
The four geeks looked pained. “What should we do?”
“Well, we can start with reformating the disk and reinstalling the OS.” I knew the minute I said that I was not going to be the most popular girl in the room. That sort of thing is awfully tedious and boring; no fun for geeks.
“But there’s ten other websites on this server!” Oops, this was going to be a LOT of work.
We segued briefly into the advantages of virtual machine backups, and then returned to the discussion of what to do.
I finally asked that deadly question: “What do your Incident Response Procedures say?”