On Thursday 02.03.2023 between approximately 09:30 - 10:53 CET Labrador CMS experienced major service disruptions due to issues with our service provider erroneously disabling several of our physical servers.
All Labrador CMS servers are hosted by the French company OVH. Our servers are spread across three datacentres in three different countries and have a high degree of redundancy to allow for high uptime and lessen the impact of one server failing on the entire infrastructure.
Our internal monitoring systems reported the first unavailable services at 09:31 CET. Customers were at the same time reporting the inability to access their Labrador instances.
We initially suspected another issue with CephFS and the initial investigation seemed to corroborate this suspicion. Early efforts were therefore focused on trying to mitigate these errors. After a short while it became clear that the fault lay elsewhere as more servers reported being unavailable. We then discovered that the OVH dashboard reported several servers being disabled due to an error in the payment system.
After discovering the cause we quickly reversed the disabling and servers began coming online again. When all servers had come on line, our infrastructure software was restarted. All services were back to normal operation at 10:53 CET.
Services affected by this incident are specified in the table below.
Service name Minutes Time from — to Labrador CMS 83 09:30 — 10:53Following is a timeline that describes the entire incident handling process.
2023.02.03 09:30 Service outage alerts registered2023.02.03 09:50 Problem confirmed to be a critical mass of servers going offline unexpectedly2023.02.03 10:02 Server disabling confirmed to be caused by OVH systems.2023.02.03 10:20 Servers began coming back online2023.02.03 10:30 Labrador CMS became available again, with intermittent failures.2023.02.03 10:39 All servers online, restart of Labrador infrastructure software started2023.02.03 10:53 All services restarted and operationalThe root cause of the incident was an issue with OVHs payment handling and automated server disabling. Following this incident we have reached out to OVH to establish more rigid procedures around server disabling and payment handling.
In addition we are continuing our work to be able to move more infrastructure to other cloud providers such as Amazon AWS.