The issue has been resolved.

The issue was caused by two servers in one cluster failing simultaneously. This impacted the uploading of new files to about 20% of Sirv accounts.

Uploads are designed to continue as normal when one server is down but when two servers are down, some uploads can fail, which is what happened. They failed due to an exporter service that stopped running on one server, then the other server. Errors prevented the services from restarting. Then a second issue caused the resolution to take much longer than expected because the servers couldn't be rebooted. This was due to an outdated BIOS.

Once the BIOS had been updated, the servers were rebooted and the cause of the underlying issue was resolved.

To prevent this from happening again, we have implemented a new BIOS management process with our datacenter. We are also shortening our hardware refresh cycle, for more frequent hardware upgrades.

To reduce the chance of the exporter service failure from recurring, we have disabled a process and are monitoring the server metrics.

This issue remains in progress. The cause of the issue has been identified and we are working to resolve it as soon as possible.

Some file uploads are not completing due to a server cluster issue. The cause is being investigated. The issue is affecting a small number of accounts. If your file upload fails, please wait until this issue has been resolved.