For this week’s “In the News”, research a recent article that relates how an organization was benefitted by their business continuity program, or suffered due to the lack of an adequate program? What are the key lessons learned from the article?
Reader Interactions
Comments
Leave a Reply
You must be logged in to post a comment.
Delta Downtime Causes Major Headaches for Travelers
In 2016, there was a power control module malfunctioned in the main computer’s network. This resulted in 6 hours of downtime. It is obvious this created chaos for airports and travelers across the country.
This created long lines at the check-in gates and Delta agents had no other choice but to write out boarding passes by hand with pen and paper.
Over thousands of passengers were stranded with screens showing incorrect flight times. In addition, Delta’s flight status update system failed, so passengers arriving at the airport for expected outgoing flights were also unaware.
By the time the issues were resolved, Delta had to cancel 870 flights. Afterwards, another 300 flights were being canceled the next day.
It took time before Delta was 100% back to full functioning. This also resulted in a lot of damage in the form of financial losses, compensations, customer dissatisfaction and bad public relations.
Lesson learned here is a strong business continuity plan makes all the difference. No matter how large or small the company, there are always thing that can’t be predicted. Anything from natural disasters to human error can take a company offline.
https://kjtechnology.com/4-examples-of-business-continuity-plans-that-failed/
Hi Elizabeth:
I find it very absurd to believed that a huge airline like Delta would allow its business to be crippled for three (3) days due to a critical power control module at its technology command center which eventually culminated to a loss of power.
It would appear some of the systems did not fail over completely as one would expect it to have happened. A full interruption testing and.or parallel testing of Delta’s business continuity/disaster recovery (BCDR) would have uncovered this catastrophe. As much as it comes with a cost, its still inherent to conduct these type of tests to avoid a partial success once a BCDR plan is invoked,
“Is my flight available?” its the only question you ask over and over prior to your flight. Even the most brooding worrier would figure main computer power module failure as a flight disruptor. Yet, I think you might be going too light on them. I think for a company like Delta they might attempt to find a faster solution for power in the time of a disaster. 1170 flight were canceled due to a power module. This is negligence or the most wildly acceptable risk for a company who’s main service is availability.
Data center disaster recovery plan proves successful – ConocoPhlillps
_______________________________________________________________________________________________
Highlights:
• Hurricane Harvey, a Category 4 tropical storm that hit Texas on August 25, 2017.
• ConocoPhillips, an American multinational energy corporation with its headquarters located in the Energy Corridor district of Houston, Texas in the United States was impacted by the hurricane.
• Main business complex was evacuated due to its location.
• Essential staff were left to monitor the security of the entire facility and its critical equipment
• August 29, 2017, due to worsening conditions, data center disaster recovery plan was activated, relocating the Houston data center operation to an out of state backup facility.
• IT Infrastructure & Operations Team had already discussed and rehearsed contingency plans and identifying critical points of contact.
Key Lessons:
• An existing DR plan which had been tested prior to the disaster proved contributed positively to the disaster recovery efforts.
• A key essential step was ensuring the safety of the employees.
• The modernization of the data center leading to the transitioning of the computing environment to the cloud, created additional capacity for the disaster recovery effort. The cloud allowed the business run virtually and completely from a backup facility.
• An existence of collaborative efforts within the team which was a shared responsibility enabled the DR plan to be rolled out successfully. Conoco practiced its DR plans and was well prepared for the disaster.
http://www.conocophillips.com/spiritnow/story/data-center-disaster-recovery-plan-proves-successful/
This is a great example of a plan executed properly. Its refreshing to read a positive example amongst our blog. Conoco is an example of how multiple teams operating under a BCP can work as a larger unit and succeed. As you can read they also ran into a bit of luck with their cloud usage and data center upgrades.
I am going to stay on the mostly hasn’t happened yet side of the news.
Hurricane being a very physical threat. Short of relocation if you are where hurricanes blow, plan for disaster.
https://securityboulevard.com/2019/06/hurricane-season-is-upon-us-is-your-business-continuity-and-disaster-recovery-plan-ready/
The strong use of virtualization, off site redundancy and Disaster Recovery as a Service (DraaS) all show the trend away from server rooms in brick buildings (metaphorically). In the wake of natural disaster it would be reassuring to know that a truck with a generator and a backup of your data was already ~en route~.
With so many Cloud solutions (AWS, Google, Azure, etc..), available, it is criminal if an organization isn’t using any as part of a contingency plan. Many large corporations these days are moving their core services to the Cloud, as it makes sense economically, as well as from a BCP stand-point as opposed to keeping all the services on-prem.
Good article and what frightening statistics about data loss. “close to 70% of those businesses that suffer data loss are forced to go out of business within two years of the disaster.”
I agree with Ahmed, cloud systems greatly reduce the risks for data loss due to significant disaster. Apparently 70% of such incidences result in a follow up disaster.
A crypt exchange may have lost $145 million after its CEO suddenly dies.
https://www.cnn.com/2019/02/05/tech/quadriga-gerald-cotten-cryptocurrency/index.html
This is one of my favorite business continuity / backup plan failures. Earlier this year Gerald Cotton, co-founder and CEO of Quadriga cryptocurrency exchange died. The laptop used by Cotton was the only laptop that had the encrypted information for accessing the cryptocurrency accounts for Quadriga. Cotton was the only one who knew the password, so with his death, so too went the password.
This is one of the ultimate blunders, one, and only one person holding the keys to the entire organization. Quadriga has collapsed and the company is in Canadian court for bankruptcy. How could this have been avoided? Simple, another source to hold the safeguarded password that could be accessed in the event of this type of scenario. Unfortunately Quadriga will not get to learn from it’s mistakes as the company will not be around much longer.
Scott,
This is such an interesting article. This article reminded me how companies discuss the cocnern of how there is a generation on the verge of retirement and some individuals have all the knowledge like say to systems and that knowledge that needs to be in the company is about to exit.
In this case, it is baffling that the CEO did not know any better and did not make a wise decision and provide the needed information to another indvidual. Which makes me wonder if this was an itentional decision by the CEO?
Scott,
This is a great article. It is very astonishing that only the CEO had the password to the for accessing the cryptocurrency accounts for Quadriga. A big company like Quadriga didn’t put into consideration that there should be more than one person who has the password to this information and didn’t think how this could impact the entire company and lead to its bankruptcy.
Dima, Katrina was a big learning lesson for the US. There are many skeptics and even controversies surrounding New Orleans/Katrina. Flooding on that scale was never measurable by this modern world prior. Years to follow major flooding took place in Houston Texas and while the flooding cause much damage the duration for the people stranded was nothing like New Orleans. Perhaps it was from the ocean surge. I doubt Katrina’s magnitude will be an anomaly. While it might take a perfect combination of elements to see another Katrina disaster it is comforting now to know that our government has more in place for such travesties.
https://www.infosecurity-magazine.com/news/philly-courts-still-down-after-1-1/
Philly court system seems to have been unprepared for a cyber incident. After a May 21, 2019 cyber-attack, their systems are still down and have had to contract a 3rd party to conduct an analysis of the attack and get systems back up. Unable to name the 3rd party hired for the task, based on the timeframe, they had no defined procedures/plan for these type of incident.
Hi Duy:
Interesting article that showcases the result of failing to have a BCDR process and plan in place.A clear case of anything that can go wrong is what has happened with the Philly court system. The failure to create a BCDR plan is sure going to cost the court system a lot. They will definitely learn on the go with this incident. One wonders why the name of the 3rd party firm hired to clean mess is shrouded in secrecy. I can’t help but to ask if a risk analysis was ever conducted as well. I seriously have my doubts.
https://barbadostoday.bb/2019/07/02/looting-corruption-harm-post-disaster-business-recovery/
This news article was a quick and interesting read for business continuity. It is a brief result of a Post-Disaster Business Continuity Management (BCM) workshop in Barbados. The Caribbean gets rocked every year with natural disasters. Most Caribbean destinations rely heavily on tourism.
The speaker of this workshop echos our text heavily by stating that if “our” governments cannot commit to integrity, honesty, justice, equity and democratic principles during times of regularity then looting and incompetence will continue in times of disaster.
I find it hard to believe that smaller countries in the Caribbean, like Haiti, don’t have a backup plan when they can regularly set their disasters to a calendar. Major businesses wish they could have that kind of insight into their disasters. A “when” indicator is a major leg up in continuity planning as you could reserve resources. Perhaps the lack of this kind of government planning is why such countries need distort assistance. I am sure corruption is likely a factor in why this planning is not fulfilled. The only answer to elevate these countries would be to educate the people so they could make proper choices for their government.
Scott,
I agree with you that these Caribbean countires need a BCP, especially due to all the hurricanes they experience. However, I am not suprised that they do not yet since Hati and Barbados are third-world counties. I am not saying that is an excuse for why they do not have a BCP, but I am wondering maybe that could play a factor into it. I am thinking that a BCP is not on the country’s radar. Like you said, it could be due to corruption as well. Since they are developing countries, I am not sure if they are there yet when it comes to these topics.
Lessons Learned From Hurricane Katrina: Preparing Your Institution for a Catastrophic Event
This article discusses the impact Hurricane Katrina had on financial institutions and organizations despite having business continuity and disaster recovery plans in place. The major challenges that were faced by these institutions are:
– Communication outages made it difficult for these institutions to locate missing personnel
– Transportation wasn’t always available to restricted areas
– Lack of electrical power and fuel for generators made many computer systems unavailable
– Mail service was interrupted for months in several areas
– Many ATMs and branches were underwater for weeks
The business continuity plan was well developed however it did not incorporate the possibility that damages can happen as the damages that were caused by Katrina.
The lessons learned are that business continuity plans should include all the critical functions and areas. Communication is very important when it comes to incidents and threats and we could expect that disruptions in communication services can extend for long periods of time. There is always a need for alternate facilities in case the main location is destroyed or damaged. The location of the back-up sites is very important to ensure recovery.
https://www.ffiec.gov/katrina_lessons.htm
Philly Courts System outage
This is an interesting and relevant system outage that has impacted the Philadelphia Court system for the past month or so. On May 21st, some of the Philly court computers were hit by a virus. As a precaution, the courts took down all their systems which included things like conducting title searches for real estate, filing documents, and paying bills/fees. This brought some of the business that rely on court activities to a stand-still. People were required to show up in person, causing long lines to queue, and spending many hours trying to get information that was previously at their fingertips. As of July 1st, the court systems have been restored, however, this brings to question many things.
First, how did the malware first get into the computers? Second, I understand that they brought down all the systems as a precaution, but they already knew the malware only impacted some computers. Were they truly not aware of the scope of this infection and didn’t feel confident they were going to be able to isolate the problem? It’s almost like a panic move. Finally, it didn’t seem like there was a contingency plan to continue some of the operations other than making people show up in-person for some of the functions. This wasn’t even a ransomware or a larger scale attack that brought down the systems (similar to what is happening in Baltimore) and it cause these many issues. It worries me that if we face a larger scale attack, the Philadelphia IT infrastructure would not be able to survive due to a lack of proper BCP procedures.
https://www.theverge.com/2019/6/11/18661484/philadelphia-court-system-shutdown-computer-virus-document-file-system
This story is not so recent and is many years old, it is a sad story but it was very surprising that this organizations business was able to go on and recover, after this I will go ahead and research a recent story.
Giant bond trader Cantor Fitzgerald occupied several top floors in one of the WTC (world trade center) buildings and lost its offices and perhaps 700 of its 1000 American staff.
No company could have adequately planned for a disaster of this magnitude, it was the single greatest personnel loss of any employer or institution, it accounted for more than 65 percent of the investment bank’s total workforce.
However, Cantor was almost immediately able to shift its functions to its Connecticut and London offices, and its surviving U.S. traders began settling trades by telephone. Despite its enormous losses, the company amazingly resumed operations in just two days (I think this is really very impressive), partly with the help of backup companies, software and computer systems. One reason for its rapid recovery was Recall, Cantor’s disaster recovery company. Recall had up-to-date Cantor data because it had been picking up Cantor backup tapes three to five times daily.
Another interesting one was Walmart and Hurricane Katrina. To think that Walmart was so prepared for business to continue in 2005, I wonder how great the “Business Continuity & Disaster Recovery” plan would be now- in 2019
Cantor Fitzgerald: Miracle on Wall Street
https://www.institutionalinvestor.com/article/b150q8nlv871x8/cantor-fitzgerald-miracle-on-wall-street
I was actually shocked when reading this article. The statistics that they spoke about were staging stating that 30 percent of enterprises’ data was lost and 42% of the companies experienced downtime from their cloud providers in the past year. I think this is why a disaster recover plan and business continuity is so important. It shows that it is not upcoming for something to happen and you should be prepared just in case something does happen. 11% of companies have said they had to use the cloud to recovery lost data as well. This is why I think cloud services are not only good but also bad. It can stall your company if there is an outage but it can also recover data that you might loose if you host locally and back up in the cloud.
Source: https://data-economy.com/30-of-enterprise-lost-data-due-to-data-centre-outage-in-one-year/
The Philadelphia Court is still not able to use their systems due to ransomware. The ransomware is asking for $100k to unlock file-shares and email servers. (This was not disclosed but I know from a third party)
https://www.infosecurity-magazine.com/news/philly-courts-still-down-after-1-1/
Ransomware attacks are common, so common in fact that companies usually do not pay out and instead rely on their business continuity solutions to resolve the issue. In this case the Philadelphia Court does not seem to have another option, due to lack of backups and lack of fail-over servers.
Hey Fred, I don’t believe this was a ransomware attack. I think it was just a malware that infected a few computers. Philly decided to take everything down as a precaution so they can scrub all the systems, infected or not, and make sure everything is in the clear. Also, I believe most, if not all, the systems were back online this past week. Either way, your point stands. I don’t think the courts had a legitimate BCP in place for this type of attack, which can be even more catastrophic in the future.
Ah thanks for the clarification. I was wondering why it took the court so long to remediate the issue. If they were scrubbing every system that explains the downtime.