At 1035 UTC on 23 November, the upgrade was complete and searches resumed normal operation. There was no downtime and all systems functioned as normal, except that API search requests had been returning error responses.

The issue was not identified because other API requests were returning as normal, so total API error responses were only slightly elevated and no alerts were triggered. The issue was identified after the event, after a report by a user prompted an investigation. To automatically identify such issues in future, we are evaluating ways to monitor the search API specifically. We will also be updating the Sirv WordPress plugin to show a clearly-described message in case of a failed search.

At approximately 1830 UTC on 22 November, we began an upgrade to the primary file database to a new, more powerful cluster. While the upgrade took place, API search queries were failing.