Previous incidents

The issue has been resolved.

The issue was caused by packet loss affecting our new Los Angeles datacenter. The datacenter traced the issue to their upstream provider, Arelion, which was experiencing network instability.

The issue was resolved when Arelion was temporarily disabled as an upstream provider and traffic was rerouted through other providers.

We require that our datacenters identify and reroute traffic automatically and quickly in such scenarios. However, the resolution of this i...

The issue has been resolved.

The issue was caused by two servers in one cluster failing simultaneously. This impacted the uploading of new files to about 20% of Sirv accounts.

Uploads are designed to continue as normal when one server is down but when two servers are down, some uploads can fail, which is what happened. They failed due to an exporter service that stopped running on one server, then the other server. Errors prevented the services from restarting. Then a second issue caused th...

The issue was resolved within 3 minutes. The cause was the rebooting of multiple servers at the same time, whilst only 1 server should have been rebooted. This was an oversight in our standard operating procedures. We are investigating ways to automatically protect and prevent this from recurring.