The issue has been fully resolved.

The issue started when heavy image upscaling API requests overloaded the upload API, causing some API requests to return 504 errors. This was solved but a knock-on effect caused approximately 5% of API requests to fail with a 502 response.

To prevent this issue from happening again, improvements have been made to reduce the load of image upscaling and better distribute load. Changes are also being made to logging and alerting, for faster identification of such scenarios.

The greatest impact of this issue was on the Sirv WordPress plugin. The plugin was sending an API request on all admin pages, which blocked the page from loading until the response had been received - typically less than 300ms but up to 20 seconds if API requests timed out. Those API requests were not required, so a new WordPress plugin version v7.3.3 was released at 1805 UTC (14 November) to remove such requests and prevent this issue from recurring:

https://wordpress.org/plugins/sirv/

If you use the Sirv API, we recommend that your scripts are written to retry requests that receive a 502 response. Recommendations for handling 502 and other error responses are provided here:

https://sirv.com/help/articles/sirv-rest-api/#error-handling-guide

The API upload overload is almost fully resolved.

Changes have been made and the issue is almost resolved. The majority of upload requests are successful.

There continues to be heavy load on the API from upload requests. A contributing factor is the large number of AI image upscaling requests, which are taking longer than usual to process. The delay caused by the queue of requests is causing some API file upload requests to fail with a 504 response.

An elevated number of 504 timeouts are being returned for API file upload requests.