Previous incidents

February 2024
Feb 22, 2024
1 incident

Elevated 404 responses for 12 accounts


Resolved Feb 22 at 12:06pm GMT

The issue has been resolved. At 1052 UTC, about 11% of requests to a small number of accounts (12 accounts, out of 50,000+) were returning 404 responses (file not found) when they should have returned 200 responses (OK). When the Sirv team identified the issue, they quickly worked to solve it and by 1206 UTC all such requests were returning the normal 200 response.

The issue occurred due to a misconfiguration by Sirv engineers while they were configuring a new storage cluster shard. Some r...

1 previous update

January 2024
Jan 05, 2024
1 incident

Elevated errors in some CDN locations


Resolved Jan 05 at 09:07pm GMT

The issue has been resolved. It was caused by the failure of a software daemon simultaneously on multiple servers. This is a lightweight and highly efficient service manager that has never failed in the past.

During the period, between 3% and 58% of HTTPS requests returned an error, depending on which of the 25 CDN locations received the request.

We are now investigating why this happened and will be taking actions to prevent this from impacting service availability in the future.

1 previous update

December 2023
Dec 05, 2023
1 incident

Elevated rate of 415 errors


Resolved Dec 06 at 07:47pm GMT

The issue that caused the elevated rate of 415 errors has been resolved.

The truncated files were caused by a software issue that wrote some files with zero bytes. As it was software, this caused the truncated files to be replicated across all versions of the master file. Sirv triplicates every file to protect against hardware issues (server or hard drive failures), however, the updated files passed all checks and faithfully triplicated the files. It existed without detection because cach...

6 previous updates

Dec 02, 2023
1 incident

Elevated rate of 429 errors


Resolved Dec 02 at 10:31am GMT

The issue was resolved at 1031 UTC.

The cause was a server misconfiguration which made the Sydney CDN location see all requests coming from the same IP address, instead of the true client IP address. This triggered DDOS protection, to rate-limit the excessive number of requests from the same IP. 429 responses were returned for many of the requests.

The issue appeared to have been an extremely short issue when the Sydney datacentre network became unavailable. That issue lasted approximat...

1 previous update