Some CDN requests returning 502 responses from Washington DC POP
Resolved
Dec 26 at 08:55pm GMT
The issue has been fully resolved.
The errors were returned by the Washington CDN server after its hard drive rapidly reached maximum capacity. Such an issue has not occurred before because each CDN server automatically deletes less-frequently accessed files, to maintain space. However, unusually heavy load of CDN MISSes outpaced the rate at which other files were deleted, causing heavy filesystem load and the failure of some requests.
The 502s occurred before the server had detected the matching account, so a count of the exact number cannot be displayed in account analytics. Approximately 117,000 responses were affected, less than 1% of total requests over the period.
To prevent this from happening again, the Washington DC location is being upgraded with faster drives and updated cache clearing logic.
Affected services
Created
Dec 26 at 02:32pm GMT
Some requests to the Washington DC CDN are being returned slower than normal and some are returning a 502 response. This is due to hard disk reaching its caching capacity much faster than usual. We are resolving the issue and responses will gradually return to normal.
Affected services