6 disaster recovery do’s and don’ts from Hurricane Katrina survivors
But while most disasters — natural or otherwise — can’t compare with the magnitude of Katrina, there were some hard lessons learned that could help organizations be better prepared for the next catastrophe.
IT leaders in New Orleans and nearby cities share how they maintained or resumed business operations in the wake of Katrina and what the experience taught them. Here are their disaster recovery do’s and don’ts — sage words of wisdom from the trenches.
“Have a plan, and practice it before you actually need it,” says Vernon Decossas, CEO of DirectNIC, a domain name registration and web hosting service in New Orleans. DirectNIC was fortunate to be on the 11th floor — safe from the flooding that impacted most of the city — but still in the direct path of the hurricane. A DirectNIC employee live-blogged the harrowing experience.
DeCossas says DirectNIC’s support team was displaced and spread from New Orleans to Houston in the wake of Katrina. Learning from that experience, the company has since refined its business continuity and disaster recovery plan to ensure smooth-functioning support from remote locations. “Case in point, we still have an all-hands meeting at the beginning of hurricane season so that every employee knows what actions need to be taken, and [we give] a refresher on our various policies during and after a storm should it affect any locations,” he says.
IT operations for the city of New Orleans, located on a third floor, were also hit hard during the storm. Lamar Gardere, director of IT and innovation for the city of New Orleans, learned the importance of having a written plan. “Document your equipment, document your environment’s configuration, document all vendor contracts and support agreements [including contact information], and make sure information is accessible to more than just one or two individuals,” he advises.
When it comes to business continuity and disaster recovery, the cloud is your friend. No matter where your primary office is physically located, business can continue if critical systems and data are running in the cloud. Gardere suggests that if your business doesn’t leverage the cloud for applications and data, you should at least have a secondary location — somewhere geographically separate from your primary site — where you can move to and resume operations in the event of a catastrophe.
IT leaders also suggest evaluating the aspects of your business continuity and disaster recovery plan that rely on third-party vendors and being careful to choose vendors you trust.
Ric Jones, CIO at LifeShare Blood Centers, a blood bank in Shreveport, La., cautions other IT leaders to choose backup vendors wisely. “With the explosion of data, it’s now more important than ever to have a reliable partner who can back up your IT infrastructure,” he says. “When Hurricane Katrina’s devastation struck New Orleans, several of our regional centers were closed, demand for blood product substantially increased, and donors were not able to give blood due to our site closures, interrupting the flow of business.” LifeShare Blood Centers now works with a third-party provider of donor services for backup in case of disaster or emergency.
“Despite being on the third floor, the city’s primary data center was damaged primarily due to power surges and an inconsistent supply of power,” says Gardere, of the storm’s aftermath. “It is difficult to tell how prepared the city’s IT department was, but there was no disaster recovery site available where basic IT services could be transferred in the event of an emergency.”
The city of New Orleans now performs semi-annual tests of basic backup functions and holds an annual table-top exercise simulating a hurricane to test its business continuity and disaster recovery plan. The city also refreshes documentation and reviews roles and responsibilities annually.
Kay Jones, a spokesperson for Entergy Corp., an energy company in Baton Rouge, La., says annual storm drills are now part of the company’s disaster preparations. “We use that time to talk about ‘what-ifs’ and come up with solutions to questions posed during the drills,” she says. “We use this time to get better at responding and being prepared for any situation that can arise when a storm hits our service territory.”
If the failure of any one server, application, router or individual can grind operations to a halt, you’ve got a problem, says Gardere.
He recommends using a combination of cloud, hosted and on-premises solutions to keep critical applications and data safe and accessible. That mix can ensure business operations continue whether a disaster strikes your facility or your cloud provider. It’s also a good idea to use more than one backup service or solution so you have options for recovering and restoring your data.
Also, having a generator or backup power source for critical systems is essential, as is designating a backup person for every key task in case a particular employee is unavailable.
Gardere says thousands of small electronics in the city's IT department, such as VoIP phones, computers and network switches, suffered water damage as a result of Katrina flooding. He encourages basic, commonsense precautions when housing equipment. “Elevate switches and other infrastructure components, and use UPSes on all field networking infrastructure,” advises Gardere.
LifeShare Blood Centers’ Jones offers perhaps the most important advice for organizations to heed. “Don’t wait until another disaster strikes to take action; the time to take action is before it hits,” he says.