Some failing requests in 4 US states
Some requests from Texas, Arkansas, Louisiana and Oklahoma returned a network timeout or redirect error response, due to an outdated DNS resolver. It is estimated that this affected up to 25% of requests during the period from 13:37 to 16:10 UTC.
That DNS resolver - which is outside Sirv's control - started directing a proportion of requests to a Sirv CDN server in Dallas that had been retired 4 weeks earlier. Our engineers resolved this highly unusual issue by forcing a DNS update.
To protect against any repeat of this issue, we have updated our server retirement policy to maintain IPs for an additional 8 weeks after being removed from Sirv's DNS configuration. We are also investigating how a DNS resolver could have reverted to an outdated DNS record and will implement additional protections where possible.