Disaster Recovery at the Macro Level

But that is not where the story ends. Satisfying compliance initiatives may get you off the hook with the regulators and make you look good on paper, but what you are really interested in is staying in business for the long haul. The statistics are staggering. Eighty-five percent of companies without a disaster recovery plan go out of business within a year after a disaster. All your hard work blown away by a Katrina, washed away by a tsunami, crumbled by an earthquake, or smashed by terrorists.

After the World Trade Center disaster, statistics showed that companies with complete plans were operational within 30 days. Those with partial plans were operational within 90 days. Thirty percent of those companies went out of business in spite of their planning. Those with no plans at all – eighty to ninety-five percent went out of business altogether.

Overwhelming, isn’t it? One thing to consider is forming a separate entity from the Information Technology department to manage disaster recovery. While IT people are key partners in the disaster recovery efforts, their plates are usually full and overflowing. Disaster recovery may not get the proper focus if placed on top of the pile.

DRP and its partner Business Continuity Planning (BCP) are board level initiatives and should be treated as such. If you don’t have someone within your organization with the right expertise, go find someone who specializes. Put someone in your organization solely and fully in charge of the effort with board level authority.

The right consultant will initially do three things:
1. Perform an assessment of what is in place
2. Ask you when you last tested your existing plan
3. Review test results for successes and failures

If you never got to the point of testing your existing plan, the right consultant will help you develop a program that requires the least amount of investment initially. Good programs are implemented over time. In disaster recovery planning, you have to do the right things in the right order and there are no shortcuts. This will require your patience, but not a huge chunk immediately out of your pocketbook. If a consultant wants the big bucks up front, take your business elsewhere.

Once the program is developed and the plan completed, testing the plan is the next critical step. Business applications will have been identified, rated and ranked as “critical”, “important”, and “everything else”. However, this is where testing mistakes can have huge negative impact on the business.

Critical business applications have the most tentacles and should not be the target of the first recovery test. Pick a smaller application and test recovery outside of the production environment. Never test recovery of applications in the production environment. It won’t prove anything and will bring you more problems than it’s worth. Test recovery of one application at a time and work your way up to the most critical business applications.

After each recovery test exercise, an honest review of the successes and failures is not a step to be taken lightly. This is your opportunity to refine, refine, and refine. Every mistake or problem you address during testing will not be a problem to be solved during a real disaster.

Documenting procedures is not for the faint of heart. The procedures you created before testing will not necessarily be the same procedures after testing. Lessons learned after each test should be incorporated into the plan, procedures refined, and documented. It is important to devote a resource to documentation. IT people, while highly skilled, aren’t the best people to write down what they know and how they do it.

Disaster Recovery Planning can accomplished successfully with the right people in place. To recap, appoint a focused leader with board level authority. Find the right specialist(s) or consulting house instead of assigning the initiative to existing IT staff. Be aware that the most expensive consultants may not be the right consultants. Disaster recovery planning takes time and expertise, not necessarily all the money in the bank. Then, plan, test, test again, and document procedures. It can be done.