During the recent fire at the OVHcloud data centre in Strasbourg, France, the company told their clients including the French government, the Centre Pompidou, cryptocurrency exchange Deribit, and several cyber criminal groups to activate their IT contingency plans. Their what? Isn’t the point of using a third party cloud provider that they will have IT contingency plans in place for us?
The short answer is no, everyone should have their own disaster recovery plans in place, no matter how reliable their service provider is. Your IT is too crucial to your business to rely on someone else’s IT contingency plans.
The OVHcloud fire affected approximately 2% of all .fr websites, and many organisations lost data, time, and money as both they and OVHcloud scrambled to activate contingency and disaster recovery plans enabling them to get back to business as soon as possible. Customers trying to access affected websites during the incident and in some cases for days after were met with blank screens, with no way of accessing the services they were looking for.
The fire also entirely destroyed one data centre (SBG2), damaged four out of 12 rooms in another data centre (SBG1), and rendered the data centre unusable in future. The two remaining data centres on site (SBG3 and SBG4) were shut down during the event, meaning that their services were disrupted both during the event, and for the time it took to reboot the servers and get them back online. OVHcloud will be running at reduced capacity for some time as they rehouse servers from SBG1, and replace the destroyed servers from SBG1 and SBG2.
Why is an IT contingency plan so important?
First of all, the contingency planning process requires you to understand the risks in all your IT systems, whether they are internally held, or externally. For example, many of the worst affected OVHcloud customers used bare-metal (physical server), virtual private server (VPS), or dedicated servers without backup services from OVHcloud. They may have had reasons for these agreements, but they meant that when the fire happened they lost everything – unless they had their own backup plans in place.
A full IT contingency planning process would have enabled organisations to identify whether regular data backups were in fact a part of their contract or not, enabling them to make other plans, check the suitability of the backup (for example ensuring that backups are not held on servers in the same facility as the actual server), and ultimately save themselves a big headache when a crisis does occur.
In today’s world, a company’s IT is integral to their running. Data centres such as OVHcloud are becoming increasingly important to organisations. In research carried out by Vertiv, 38% of participants said that their data centre is critical to their business, 21% of participants said their business is totally dependent on their data centre, and only 10% responded that their business could operate for a limited period without computing. Regardless of how unlikely an outage may be at a data centre, you don’t want to be left without service, or suffer a loss of data when that day does arrive.
IT contingency plans work, not just for extreme cases where servers are totally destroyed but for other IT focussed crises too. A cyber security breaches survey by the UK Government’s Department for Digital, Culture, Media and Sport in 2018 discovered that 94% of organisations who had an IT contingency plan in place found them effective when they suffered a cyber security breach. More worryingly however, only 13% of businesses actually had a plan in place.
Contingency plans that consider all possible scenarios, that are well thought out, and rehearsed so that everything is in place will help minimise disruption, safeguard data, keep systems up and running, and ensure you bounce back quickly.
What happens if you don’t have IT contingency plans in place?
For OVHcloud’s customers the impact of the fire can be seen in four outcomes. These outcomes are not unique to this crisis, rather they are common concerns for every emergency or crisis scenario you can think of.
The four common outcomes are:
Complete data loss
Data, be it customer data or internal data is a precious commodity for every organisation. Failure to plan for a crisis, for example by ensuring that you keep regular, secure, accessible, and tested backups could mean that when a crisis hits you lose everything. Data backup and recovery should be a key component of any IT contingency plan.
Business interruption
An IT failure could mean that core systems are down for an extended period of time. Time is money in these situations – the longer you are offline, the more income you lose, and the lower your employee productivity during the time out. An IT contingency plan would consider the actions that need to be taken to reduce business interruption, saving crucial time, and reducing money losses.
Loss of clients
The longer you are out of business and unable to provide services to your customers, the higher the chance that they will vote with their feet and look for a more reliable service provider elsewhere. Once again, an IT contingency plan will ensure that you minimise this risk. IT contingency plans will also enable you to demonstrate a proactive approach to this emergency, helping you save precious reputation points, and reduce customer flight.
Expensive recovery
Remember the saying, closing the stable door before the horse has bolted? It costs a lot more to recover from a crisis than it does to put plans into place that will save time and money in the event of a crisis. Many businesses simply don’t recover from a crisis, and those that do will experience a severe dent in their profits for some time to come.
Plan for the future of your business with IT contingency planning
The IT contingency plans you need to have in place are constantly evolving. Current business challenges affected IT contingency planning include the new during and post Covid-19 business landscape, new technologies, and smaller, more vulnerable data environments. As organisations rely on more third party suppliers, they need to ensure that these suppliers have adequate contingency plans in place, and that their contract is included in these plans.
Effective IT contingency plans need to consider the following:
Speed of recovery
Recovery must be carried out within as short a timeframe as possible after the attack. The contingency plan should contain steps to return to operations as quickly as possible across the entire business.
Resilience
The IT contingency plan should be accessible to all relevant parties at all times. Consider a cloud based plan that is filed in a relevant place, and can’t be changed so that every stakeholder, no matter where they are can access the same information. Given that we are talking about IT incidents which may also bring down the network, these plans should be hosted somewhere that does not rely on corporate IT networks.
Testing
Every element of your IT continuity plan should be tested to ensure that there are no gaps that could damage your incident recovery, and help every member of your response team understand what their role is during the incident, helping your incident management run smoothly even as everything is chaotic.
Learn how Crises Control can help you plan for the future of your business with our incident plan builder that will support you to create IT contingency plans that address your every scenario, and an incident management system that will keep those plans safe and ready for the day you need them.
