On Monday 16.10.2023 between 09:58 - 10:13 CEST one of our primary infrastructure providers, OVH, experienced network disruptions across multiple data centers. This resulted in a partial or complete service outage for a large part of the traffic destined to both Labrador CMS and Labrador Front.
Following this incident a subset of our customers experienced further service degradation, as some Labrador components were stuck in an unhealthy state and needed manual intervention.
At 10:42 CEST all Labrador services returned to a fully operational state for all customers.
Our internal monitoring systems reported the first unavailable services and sites at 10:03 CEST. Initial investigation revealed that a network outage was ongoing at OVH, one of our infrastructure providers, causing connectivity disruptions between our services.
Networks returned back online at 10:13 CEST, however, internal monitoring and customer reports indicated that a subset of our clients still had service degradation, either in the form of slow responses or complete time-outs. Further investigation revealed that one of our servers was stuck in an unhealthy state following the network disruptions.
The affected server was pulled from our cluster and restarted, and all Labrador services returned to normal at 10:42 CEST.
Services affected by this incident are specified in the table below.
Service name Minutes Time from — to Labrador CMS 44 09:58 — 10:42 Labrador Front 44 09:58 — 10:42Following is a timeline that describes the entire incident handling process.
2023.10.16 09:58 Initial service outage alerts registered2023.10.16 10:08 Large scale network outage confirmed2023.10.16 10:13 Network operational again, most customers back online2023.10.16 10:20 Reports that some customers still experience service degradation2023.10.16 10:35 Service degradation root cause discovered2023.10.16 10:42 Affected services restarted, fully operationalThe root cause of the incident was determined to be network disruptions at OVH, resulting in one of our servers ending up in an unhealthy state.
We are continuously working on improving and decentralizing our infrastructure so that we are less vulnerable to large scale data center network outages.
One of our current largest efforts in this regard is moving parts of the Labrador CMS and Front infrastructure to the cloud, reducing our exposure to various OVH outages and increasing our geographical presence and flexibility. This has a high priority for us, and we expect progress to be made throughout this year and the next.