Previous incidents

More than 99% of the remaining truncated files have also now been restored. Any impacted user has been provided with an exact list of files that were impacted.

There is no residual impact remaining from this issue.

Classified as severity level 1: Critical impact.

The issue was resolved at 1031 UTC.

The cause was a server misconfiguration which made the Sydney CDN location see all requests coming from the same IP address, instead of the true client IP address. This triggered DDOS protection, to rate-limit the excessive number of requests from the same IP. 429 responses were returned for many of the requests.

The issue appeared to have been an extremely short issue when the Sydney datacentre network became unavailable. That issue lasted approximately ...

At 1035 UTC on 23 November, the upgrade was complete and searches resumed normal operation. There was no downtime and all systems functioned as normal, except that API search requests had been returning error responses.

The issue was not identified because other API requests were returning as normal, so total API error responses were only slightly elevated and no alerts were triggered. The issue was identified after the event, after a report by a user prompted an investigation. To automatica...

The new server was deployed in the evening of 14 November and gradually filled its caches. The control panel has now returned to its original fast performance. Additional servers are now being upgraded, so the control panel will load faster still over the coming 2 weeks.

Classified as severity level 4: Minimal impact.

The detailed investigation into this issue found two contributing factors.

The first was an update to AWS Route53, which appears to have been implemented by AWS on 17 October. This change was not made by Sirv to its Route53 rules but a change within the Route53 service itself. The Los Angeles POP had slightly different routing logic to other POPs and the AWS update caused routing to behave differently, with traffic becoming routed to the Washington DC fallback. This went unnoticed as he hig...