Hug your Backup Tapes — They May Save Your Company

As some of you may remember, I got my start in disaster recovery in 1989 just after the Loma Prieta earthquake. But that career change actually got started much earlier, around January when I asked Gayle in LOC 4 shipping what was in some large fireproof safes near the back wall. It turned out that those safes stored all of Tandem’s backup tapes.

When I mentioned to my manager that it didn’t seem all that safe to be storing backup tapes in a tilt-up a building in earthquake country, he asked me to work with business analyst Derek Riehl to develop a plan for a disaster plan. Derek and I worked on it for a few months then presented it in April, when it was shelved for being too expensive. After October 17, our plan was rescued from obscurity and I became a disaster planner.

But let’s get back to those tapes, because today, they are out of fashion. What with RAID, cloud storage, and snap copies, there seems to be no need for them, right? Tell that to email provider VFEmail, who had two decades of email wiped out by hackers. The ordeal started when the owner noticed that all of his external facing systems, of differing OS’s and remote authentication, in multiple data centers were down. A few hours later, VFEmail’s Twitter account reported the attacker “just formatted everything,” including backup servers.

Ah, “no problem for me,” you are thinking, because your systems are using NonStop RDF, ShadowBase or GoldenGate to replicate your data between data centers. You know that your data sits safely in multiple data centers, so you don’t need to worry about any one of them being taken down, right? How long does it take for a table dropped in one data center to propagate to every other data center? VFEMail had multiple data centers and it didn’t help them.

“At this time, the attacker has formatted all the disks on every server,” wrote VFEmail. “Every VM is lost. Every file server is lost, every backup server is lost. Strangely, not all VMs shared the same authentication, but all were destroyed. This was more than a multi-password via ssh exploit, and there was no ransom. Just attack and destroy.”

John Senchak, a longtime VFEmail user from Florida who also has been a loyal reader and commenter at this blog, told KrebsOnSecurity that the attack completely wiped out his inbox at the company — some 60,000 emails sent and received over more than a decade.

What about the offline backups? Nope, weren’t any. So, what lesson did we just learn? No matter how many levels of online backup you think you have, no matter how many layers of security you think you have, offline backups are still important.

Luckily, the NonStop ecosystem is full of possibilities. You older folks probably remember the Tandem WORM Jukebox. This device held a handful of large Write Once, Read Many optical disks. While HPE doesn’t seem to be selling WORM, they are selling huge, scalable tape systems, such as the HPE TFinity ExaScale Tape Library which can hold up to 1.6 Exabytes of offline tape storage. There are also third-party solutions from ETI-NET, comForte and others.

But the whole point is that you need to get your data out of the data center (even the backup data center) and to someplace safer, like Iron Mountain or an equivalent safe space which is completely away from any of your data centers.

“But what if I need my data quickly?” you may be asking. Rather than have all of your bits’in’a’basket, you should be approaching your backups with a tiered strategy. Snap backups for “oops, I didn’t mean to drop that table,” online backups for “I need to look at last month’s close,” offline but on-site backups for ransomware recovery, and finally off-site backups for natural and man-made disasters.

Yes, a tiered backup strategy costs money. And like most business continuity (and cyber) activities, it also does not show an ROI. So how do you justify it to senior management? A search of the Internet will show hundreds of blogs, white papers, and PowerPoint presentations, each with an opinion of how to best approach your management. Scare tactics usually don’t work, but a good disaster certainly helps kick off a program.

When I was at Tandem, I worked closely with the audit committee to help our senior management understand what the company was putting at risk by not taking off-site backups or testing how a new \CORP system could be brought up using backup tapes in a different building. Let me tell you that the first attempt didn’t go well. Disaster recovery and business continuity are all about assessing, planning, testing, testing again, and fixing problems before you suffer a real loss.

I am leaving you with an action item. If you are not part of the business continuity team, go have a talk with them about your backup strategy. Show them the stories about VFEMail, and have an honest discussion about how your own organization would be protected should a similar attack happen. “It will never happen to us” is not a valid answer. No company likes to think they will be the next victim, but hey, you never know when you’re next. And companies which use NonStop are in the top tier of companies that hackers want to take down.

Ron LaPedis
Managing Director
Seacliff Partners International, LLC