Previous incidents

Some requests received by one of the London CDN servers failed due to a combination of local DNS resolution and pulsating network timeouts. The issue began at 1215 UTC and was resolved at 1332 UTC. During this period, 25% of requests from UK and Ireland returned a 404 or 500 error. The issue would have been solved sooner but an unrelated SSL certificate validation failure a few hours earlier blocked alerts from being received by our monitoring system.

Changes have already been deployed to...

The issue was resolved at 0755 UTC.

At 0625 UTC, some requests to the Indore CDN POP which serves India started to return a 404 response (file not found), instead of a 200 response (OK). The issue occurred 14 hours after an update to all CDN servers worldwide. All POPs behaved healthily, including the India POP but the Indore POP hadn’t updated successfully and eventually started to return errors.

Once the issue had been identified, it was resolved by re-deploying the update to the Indi...

The server became healthy again and returned to service automatically, only to become unhealthy after some time then removed from service. Server health checks became healthy, then unhealthy, alternating the server in and out of service a few times, until automatic management was overridden by manually removal at 0704 UTC.

The issue was caused by a faulty hard disk. Rather than failing, the disk had random periods of errors, then would operate normally. Sirv already employs health checks t...