On Thursday 12.01.2023 between 09:40 - 12:16 CET we experienced major service disruptions due to issues following an upgrade of our file system layer, Ceph. This resulted in a partial or complete service outage of the Labrador CMS Editor for all of our clients.
At 12:16 CET all Labrador services returned to a healthy state. Labrador Front was completely unaffected by this outage.
At Labrador CMS we use Ceph as a file system for most of our storage, including articles, front pages, images, settings, etc. A critical component of Ceph is the Metadata Server (MDS), which is responsible for handling file and directory metadata. Traditionally we have used 3 active Ceph MDS for performance and redundancy.
On Tuesday 10.01.2023 we completed maintenance of our Labrador Editor environments, upgrading Ceph from version 14 to 16. During the upgrade we experienced a similar incident, although in a much smaller scale, which at that time was deemed to be caused by the upgrade process itself. We now believe the cause to be the same CephFS bug responsible for the 12.01 incident.
Our internal monitoring systems reported the first unavailable services and sites at 09:40 CET. Initial investigation revealed the same CephFS symptoms we had encountered during the software upgrade of 10.01. File system clients were stuck in a bad state due to a fatal Ceph MDS crash, which caused the Labrador Editor to hang indefinitely. The solution was to forcefully restart the bad clients so the file system lock could be resolved.
Following the first bad client restart, a new set of clients were now reporting bad states, which were also restarted. On the third bad state client occurrence our attention shifted to the Ceph MDS. Finally, halting all file system clients, reducing the amount of active Ceph MDS to 1, restarting it, and resuming all clients resolved the issue.
Labrador Editor services returned to an operational state at 12:16 CET.
Services affected by this incident are specified in the table below.
Service name Minutes Time from — to Labrador CMS 156 09:40 — 12:16Following is a timeline that describes the entire incident handling process.
2022.01.12 09:40 Service outage alerts registered2022.01.12 09:50 Outage confirmed to be caused by CephFS crash2022.01.12 10:10 Begin partial restart of affected clients and services2022.01.12 10:30 Partial restart complete2022.01.12 10:45 Additional services affected, continue partial restart2022.01.12 11:15 Partial restart complete2022.01.12 11:30 Begin complete client and service restart2022.01.12 12:16 All services restarted and operationalThe root cause of the service outage is believed to be a software bug in version 16 of Ceph, causing the Ceph MDS to potentially crash under certain circumstances when multiple MDS are active at the same time.
The bug has been fixed, but has not yet been included in a release of Ceph v16. Until the fix has been released we will limit the amount of active MDS to 1 for Labrador Editor environments.
Labrador Front environments are not affected by this bug, as it is still running Ceph version 14.