The issue that caused the elevated rate of 415 errors has been resolved.

The truncated files were caused by a software issue that wrote some files with zero bytes. As it was software, this caused the truncated files to be replicated across all versions of the master file. Sirv triplicates every file to protect against hardware issues (server or hard drive failures), however, the updated files passed all checks and faithfully triplicated the files. It existed without detection because cached versions of the files were used for processing and delivery. All systems worked perfectly, until yesterday's infrastructure upgrade when cached files were cleared.

The root cause was solved by identifying and correcting the software issue that caused truncation. This was completed by 1500 UTC on 5 December. Then the task was to restore those files that had been truncated. Across the platform, approximately 0.2% of files had been affected. Of those files, we were able to restore between 95% and 100%, depending on the account.

Any account which had less than 100% restoration is being sent a list of zero-byte files that should be restored by uploading the files again. Please look out for our email with a list of files. Please message us via the following URL if you need any help:

https://sirv.com/help/support/#support

A thorough investigation of the issue is underway, which will be reported in detail by email. Within the report will be a list of actions being taken to prevent any possible repeat of this issue, along with improvements and new features that are already being designed.

We apologise to all customers who were impacted by this issue. It is the first time master data has been lost. We will learn every possible lesson from it and make Sirv even stronger and reliable than before.

More files have been restored. The rate of 415 errors continues to fall. You can see a list of 415 errors on this page of your account:

https://my.sirv.com/#/analytics/errors/415?from=20231206&to=20231206

You can upload files to solve any 415 errors in your files. We can provide a list of zero byte files if you need - please ask the Sirv support team via this form:

https://sirv.com/help/support/#support

We are continuing to restore as many files as possible.

We are continuing to recover the small number of remaining truncated files. This is estimated to require a further 12-24 hours. If you have any files that are not being served, you can solve this straight away by reuploading them.

We are creating a tool to generate a list of zero byte files. If you'd like a list of any zero byte files in your account, please submit a request at https://sirv.com/help/support/#support and we'll prepare a list once the tool is ready. Regardless, we will continue work to recover all files.

If you upload files using Sirv's Autofetch feature, you can simply delete the zero byte files and they'll be re-fetched from your remote location upon the next request.

Approximately 80% of truncated files have been recovered.

The number of 415 errors have fallen significantly, though they remain elevated.

To fully resolve the problem, the remaining truncated files need to be replaced with the original file. We are working on ways to recover those remaining files, though they are hard to obtain and it may take hours or days.

For the fastest possible resolution, it is recommended that you re-upload the files.

Such files will show in your account as having 0 bytes. Start at the following URL, to see a list of all files uploaded in the last 30 days:

https://my.sirv.com/#/search/?created=30d

Find any zero byte files by sorting the results by Size, Ascending.

Another update will be posted here once we have news on the remaining files.

The 415 error issue was caused by some files becoming truncated, meaning they lost part of their contents. These files show in the control panel at my.sirv.com as having 0 bytes.

Files uploaded today were more likely to become truncated, though it seems that truncation has been happening to files for a long time. This was not identified because they performed entirely as normal, due to caching of non-truncated versions.

The truncated file issue only became noticeable today, after an infrastructure upgrade. Server caches were cleared and the truncated versions became apparent.

The cause of truncation is believed to have been a software bug that has now been resolved. We are currently working to restore truncated files - a process that may take from hours to days. If you are able to reupload any files that are not currently loading, please do that now.

The issue is estimated to be affecting about ~0.2% of files.

We are continuing to investigate the cause of this issue. The nature of it is complex. It represents itself as a 415 error, from files of zero bytes. There is also an elevated though lower rate of 500 errors. The majority of requests are successful 200 responses. Our engineers are working hard to understand and resolve this.

An elevated level of 415 errors is occurring on the CDN. Our engineers are investigating the cause.