10 years after Hurricane Katrina: IT lessons learned
Even prior to Hurricane Katrina it was a security mantra and data protection best practice to ensure at least one backup of crucial data was maintained offsite. The logic being simply that you don’t want your primary data storage and all of your backups to be destroyed in the same hurricane, fire, flood or earthquake.
Offsite backups solve only part of the problem, though, if your servers and data are maintained locally. When disaster strikes and wipes out your primary data, you’ll have to acquire the backup data, deploy and configure new hardware at some secondary location, and restore the data. You’re still looking at days of downtime in a best-case scenario.
The city of New Orleans and businesses like Entergy and DirectNIC that struggled to survive the devastation of Hurricane Katrina learned some valuable lessons. One of the primary caveats when it comes to business continuity is to mitigate risk by embracing the cloud.
Lamar Gardere, director of information technology and innovation for the City of New Orleans, admits that things were still in disarray when the current administration took office in 2010. The IT infrastructure was aging and many of the city’s critical applications were still being run on physical servers.
One of Gardere’s first tasks was to modernize onto a highly virtualized infrastructure to allow for servers to be quickly created, resized and moved from one site to another in the case of a major disaster. “We created a private cloud with the ability to leverage all the same capabilities as you might imagine are available if you were using Amazon’s cloud, for example. This flexibility is at the heart of our disaster recovery capabilities and allows us to quickly transfer/failover services to remote locations,” Gardere says. “During normal times, it also allows us to maximize our infrastructure investment, consolidate IT resources across areas of government, better manage resources remotely and respond more quickly to our customers.”
[Related: Tech apocalypse! 9 doomsday scenarios]
DirectNIC is one of a few businesses that managed to stay up and running during Katrina – partly a result of being prepared and partly a function of being safely on the 11th floor well above any flood damage. Even DirectNIC learned a thing or two from Hurricane Katrina, though. Vernon Decossas, CEO of DirectNIC explains, “We host our own operations, however, we also have the ability to move our operations onto cloud providers within the span of hours. It’s provided a peace of mind that we can keep our operations going regardless of external issues.”
Gardere also elaborates on the decision to implement a private cloud rather than simply provisioning services from one of the public cloud providers. He notes there are pros and cons to public cloud for any organization and that the city weighed those on a per-application basis to determine the best solution. “The City uses the cloud strategically and where appropriate to take advantage of its convenience while avoiding some of its problems. Perhaps most notably, the City has moved its payroll system to the cloud using ADP, ensuring that this critical but low bandwidth application is available regardless of the state of the City’s IT environment.”
Leveraging the cloud and moving critical servers and data to a cloud-based infrastructure will help organizations in New Orleans mitigate risk and maintain business continuity the next time a major natural disaster occurs, but it’s not enough by itself. Beyond the cloud, organizations also must have a clearly defined business continuity and disaster recovery plan in place and have staff that are properly trained to execute it when the time comes.
[Related: 10 tips to make sure you are ready when a disaster strikes]
“Entergy holds yearly storm drills to prepare all of our employees for what may come. We use that time to talk about ‘what ifs’ and come up with solutions to questions posed during the drills,” says Kay Jones, a spokesperson for Entergy. “We use this time to get better at responding and be prepared for any situation that can arise when a storm hits our service territory.”
Gardere stresses the importance of performing regular maintenance on backup equipment that rarely sees use and talked about how the City of New Orleans continues to strive toward more complete testing and monitoring procedures. “We perform semi-annual tests of basic back up functions and hold an annual table-top exercise simulating a hurricane to test strategy execution. We refresh documentation and review roles and responsibilities on an annual basis.”
For some companies even the best business continuity and disaster recovery plan won’t help. A local restaurant or the corner gas station can’t just continue operating from the cloud or move to an alternate location. No amount of practicing or preparing will enable such a business to remain operational while it’s literally under water.
Those businesses can still benefit from using cloud-based applications and data storage to ensure those things survive the catastrophe, though, and thankfully most businesses are not that dependent on the specific physical location. By moving critical systems and data to the cloud and practicing to smoothly implement business continuity and disaster recovery procedures organizations can mitigate the risk of the next Katrina-like event and be prepared to continue operations.