Back to overview
Degraded

Some CDN requests returning 502 responses from Washington DC POP

Dec 26 at 02:32pm GMT

Resolved
Dec 26 at 08:55pm GMT

The issue has been fully resolved.

The errors were returned by the Washington CDN server after its hard drive rapidly reached maximum capacity. Such an issue has not occurred before because each CDN server automatically deletes less-frequently accessed files, to maintain space. However, unusually heavy load of CDN MISSes outpaced the rate at which other files were deleted, causing heavy filesystem load and the failure of some requests.

The 502s occurred before the server had detected the matching account, so a count of the exact number cannot be displayed in account analytics. Approximately 117,000 responses were affected, less than 1% of total requests over the period.

To prevent this from happening again, the Washington DC location is being upgraded with faster drives and updated cache clearing logic.

Created
Dec 26 at 02:32pm GMT

Some requests to the Washington DC CDN are being returned slower than normal and some are returning a 502 response. This is due to hard disk reaching its caching capacity much faster than usual. We are resolving the issue and responses will gradually return to normal.