Return 429 for Rate-Limited Clients
Rate limit: answer 429 for the client over the limit, not 500 for the whole site
Problem: past ~180 requests per minute from one IP, the whole site answers 500 to that IP for about a minute. A crawler or SEO audit tool that hits the limit sees a broken site and may slow down crawling or drop pages.
Proposal: return 429 with Retry-After, and exempt verified search engine bots.
Workaround today: throttle our own tools to 2 requests per second.
Completed
429 and retry after now correctly returned. Limited increased for selected bots.