Yottabytes: Storage and Disaster Recovery

Feb 19 2019   10:54AM GMT

Backup Lessons Learned From the VFEmail.net Hack

Sharon Fisher Sharon Fisher Profile: Sharon Fisher


Note to self: When you’re doing your backups, make sure you have them on a different place than your production network.

That’s a lesson learned the hard way by VFEmail.net, a worldwide email service provider, which recently lost not only its subscribers’ email messages, but also all its backups, because they were all on the same network.

“We have suffered catastrophic destruction at the hands of a hacker, last seen as aktv@,” noted the VFEmail.net website. “This person has destroyed all data in the US, both primary and backup systems. We are working to recover what data we can.”

The exact details are sketchy, because the people running VFEmail.net are, naturally, kind of busy trying to put it back together. Thus far they’ve found a single offline backup dating from August 2016, so, hurray, VFEmail.net users are now only missing their last two-and-a-half years of email messages.

But apparently someone hacked into the system and zapped not only the primary mail servers, but the backups as well. Speculation, and there’s plenty, is that it was either an inside job or someone – perhaps even a foreign government – thinking there was something incriminating on the server and deleting everything on it, just in case.

And the deletions were reportedly very thorough, in a way that couldn’t be recovered. “At this time, the attacker has formatted all the disks on every server. Every VM is lost. Every file server is lost, every backup server is lost.. Strangely, not all VMs shared the same authentication, but all were destroyed,” noted the VFEmail.net Twitter handle. “This was more than a multi-password via ssh exploit, and there was no ransom. Just attack and destroy.”

Ironically, VFEmail.net was originally set up in response to an email virus. “VFEmail started in 2001 by Rick Romero in direct response to the ‘ILOVEYOU’virus,” notes the company’s website. “At the time, anti-virus was not integrated into email systems. After writing a set of batch files to integrate Norton AntiVirus Corporate Edition A/V scanning into Mercury/32 on Windows, Rick turned his attention to helping regular users and local small businesses avoid email-based viruses. VFEmail started with a single FreeBSD server, and thanks to Rick’s broad and extensive IT experience, frugal purchasing, and long-term planning, VFEmail has grown into the site you see today. While other services have shut down, or been exposed as not delivering on their promises, VFEmail keeps chugging along.”

The few numbers of servers may have been part of the problem. The company offered free as well as paid email accounts, and consequently wanted to save money. “We strive to build an economical and redundant system, to provide our users with as much uptime as possible,” the website continues. “As mentioned, VFEmail started with a single machine, but over time we’ve built out, adding systems for load balancing/failover and separating services. Most recently we’ve made use of Virtual Machines in order to keep hardware acquisitions at a minimum, in those cases where it would not impact performance. By separating vital functions, upgrades, updates, and system problems can quickly and easily be isolated from the rest of the system and provide you with uninterrupted accessibility.”

Yeah, well, not so much.

It all just goes to show that simply making a single backup is not enough. The rule of thumb some people use is 3-2-1: three copies of the data, two of them onsite but one of them offline, and one of them offsite. (Not to mention, checking the backups periodically to make sure you can actually recover from them.)  While that requires a lot of hard disk drives and coordination, it at least protects against the majority of problems.

1  Comment on this Post

There was an error processing your information. Please try again later.
Thanks. We'll let you know when a new response is added.
Send me notifications when other members comment.
  • Petermcycle
    If only they also had backups on tape, stored offsite........
    10 pointsBadges:

Forgot Password

No problem! Submit your e-mail address below. We'll send you an e-mail containing your password.

Your password has been sent to:

Share this item with your network: