Recipe for Disaster Recovery
June 2, 2016
What would you do if your house caught on fire? No one really likes to think about that, but it’s likely you’ve put multiple contingencies in place—smoke alarms, a planned escape route for your family, and home insurance that covers not only rebuilding, but displacement.
Recovery Mode: Disaster Preparedness for Businesses
So, if you have a detailed disaster recovery plan for your household, you must have one for your company, right? The reality, according to a survey from the Disaster Recovery Preparedness Council (DRPC), is that as many as three out of four companies have insufficient disaster recovery plans.
Why such a large percentage? Many companies probably think they will never need to use a disaster recovery plan, don’t know where to begin in developing one— or both.
But here’s the alarming truth: As recently as 2014, more than 70 percent of companies with disaster recovery plans in place reported having to use them. Also, according to a recent Sigma study, worldwide economic losses from both natural and man-made disasters totaled $92 billion in 2015.
Disasters happen, whether you’re ready for them or not.
Knowing You’re Prepared for an Emergency
The biggest shock of all can come if you think you’re prepared, but it turns out you’re not, as the following parable demonstrates:
Once upon a time there was a company whose name you would never recognize. Why not? Because its flawed disaster recovery plan ended its livelihood.
IT was one of the most important aspects of this company’s operations—the primary way it supported, communicated with and got paid by its customers. So, the company’s CEO made the main objective of his disaster plan to protect and replicate all of the company’s IT systems and data. The company was located in an area known for frequent tornados, so the CEO built a state-of-the-art bunker under the company’s headquarters.
The bunker was secured by several biometrically-locked doors that could only be opened using the fingerprints of the CEO or the IT chief. Inside, all of the company’s above-ground IT assets were replicated and supported by a humming army of backup power generators.
The CEO wasn’t there when a massive tornado leveled the company’s headquarters. All employees were able to escape into the safety of the bunker stairwell, but the fingerprint-recognition system failed to operate. There was no automated process to initiate the transition to backup IT systems, because the plan was to enable that from inside the bunker. The tornado that leveled HQ also wiped out the company’s primary IT systems. All backups failed. The company was unable to support any revenue-generating business functions for many weeks, and its reputation was irrevocably damaged.
A Good Disaster Recovery Plan is One that Actually Works
That story demonstrates the critical importance of a comprehensive, multifaceted disaster recovery plan—and the danger in what an IT pro might overlook. The DRPC survey cites that as many as 60 percent of companies with disaster recovery plans reported that their plans “were not useful” in helping them recover from actual disasters. But that doesn’t have to be every company’s disaster recovery experience. Creating disaster recovery plans is only part of the challenge—the rest is making sure you have the right plans, and more importantly, making sure they work.
Here’s how to ensure your plan works:
- Involve the organization: No disaster recovery plan should spring from the mind of just one executive, like the CEO in our parable. Assemble a team with experience across every corporate discipline to study disaster recovery challenges and possible solutions. Let many voices be heard.
- Figure out what’s at stake: Disaster recovery experts call this a business impact analysis, but it really just means taking a good long look at what could go wrong—natural disaster or man-made, like a security breach— then figuring out which systems are most critical to supporting essential business functions and would be most harshly affected. Duplicating and backing up absolutely everything could be an expensive, unnecessary exercise.
- Determine necessary response and recovery times for maintaining business objectives: Disasters happen. Outages occur. Systems go down. Don’t focus on trying to side-step what you can’t predict. Concentrate instead on how quickly systems and processes need to be restored when disaster does hit. Set deadlines for response and recovery times, and figure out how to mobilize all available personnel and backup technology to meet these deadlines.
- Come up with a plan document: Maybe this sounds like it should be the first item, but you can only come up with the right disaster recovery blueprint after you know what needs to be done and how quickly. And again, work as a team to decide detailed step-by-step post-disaster functions. Empower multiple people to act in an organized fashion.
- Be fiscally responsible: Your disaster recovery plan needs to work, but it also shouldn’t bleed the company dry. Neither should it rely on vastly discounted supplies of questionable quality. Spend as responsibly and astutely on secondary gear, facilities and providers as you would on primary assets.
- Maintain both physical and virtual protection: The physical security perimeter and the electronic security perimeter are equally important. Don’t skimp on cybersecurity because you think a natural disaster is more likely than a security breach. The Sigma study mentioned above noted that 2015 saw a record 198 natural disasters, but more than 350 total catastrophes, the rest being man-made disasters of various types. A security breach or other operational failure can be your disaster, leveling your company from the inside out if you’re not prepared.
- Consider the cloud: Keeping all your IT assets in-house, even behind really thick, biometrically-locked doors, may not be enough. The DRPC study noted that as of 2014 only 15% of companies used the public cloud for some part of their disaster recovery plans. Barely 10% relied on a hosted or managed service provider as part of their plans, and fewer than 10% had invested in a disaster-recovery-as-a-service solution. A trusted service provider can handle a huge part of disaster recovery obligations, backing up systems and data to remote, secure locations—say, a location far from a tornado’s path that can be easily accessed via private or public online connection.
- Once complete, test your disaster recovery plan over and over again: You might find out every piece of the plan works just fine, or you might find out that your biometric locks have a bit of a glitch. In any case, why would you ever bet your entire company’s livelihood on a something you never bothered to test?
- Update your disaster recovery plan frequently: Your business is always changing—new systems, new personnel, new success and growth. A bigger, better company means there’s even more at stake.
Follow this advice, and maybe you won’t end up like that company whose disaster recovery plan failed—the company with the name no one remembers.
flickr photo by Wavlan https://www.flickr.com/photos/spleeness/8221141227/ shared under a Creative Commons (BY) license
flickr photo by Bradley P Johnson https://www.flickr.com/photos/bradleypjohnson/6366116663/ shared under a Creative Commons (BY) license
flickr photo by nssl0209 https://www.flickr.com/photos/noaaphotolib/5054532634/ shared under a Creative Commons (BY) license