URL blocked due to other 4xx issue

What this status means

According to Google's documentation (Page indexing report), the status means the server returned a 4xx error not covered by any other issue type in the report, and Google suggests debugging the page with the URL Inspection tool. Three client errors have their own reasons: 401, 403 and 404. Everything else in the 400 range lands here.

Google's documentation on HTTP status codes sets out the rules. All 4xx errors except 429 are treated the same: Google ignores the content, a previously indexed URL is removed from the index, and the codes have no effect on crawl rate. A 429 is read as a signal that the server is overloaded and handled like a server error. The HTTP status codes entry has the full list of codes.

The report does not show the code. Find it first, because a 410 on a removed product and a 429 from a rate limiter need opposite reactions.

Common causes

Rate limiting with 429

The most important case, because it hides an availability problem. A WAF, a CDN rate rule or a plugin limits requests per IP per minute, and a crawl exceeds the limit. Cloudflare rate limiting rules, nginx limit_req and WordPress security plugins all return 429 by default. The page works for humans and fails for Google only. The 429 Too Many Requests entry explains the code and its headers.

Intentional 410 on removed content

A shop removes discontinued products and returns 410 Gone. This is correct and the status is informational. Google's description of the 404 reason says Googlebot will keep trying a URL for some period of time, less and less often, and that there is no way to make it forget a URL permanently.

Broken URLs with bad characters

Links generated by a template with an unescaped space, a double-encoded %2520, a stray [ or |, or a UTF-8 path the server cannot decode. Apache and some application servers answer 400 before the request reaches the CMS. Google's guidance on URL structure recommends ASCII characters and encoding non-ASCII characters in the URL.

URLs that grew too long

Faceted navigation that appends every filter to the query string, or a redirect loop that keeps adding a parameter. The server rejects the request with 414 URI Too Long once the limit is exceeded, so the page never renders.

Method and content negotiation rules

A 405 Method Not Allowed appears when a route accepts only POST, for example a form handler linked as GET. A 406 Not Acceptable comes from ModSecurity rules and from servers that reject requests whose Accept header does not match what the application offers. Both are server rules, not content problems.

How to confirm it

1. Request the URL with headers only: curl -sI "https://example.com/page/". The first line shows the exact code.
2. Repeat with the Googlebot user agent and several times in a row: for i in 1 2 3 4 5; do curl -s -o /dev/null -w "%{http_code}\n" -A "Googlebot/2.1 (+http://www.google.com/bot.html)" "https://example.com/page/"; done. A 200 that turns into 429 after a few requests is a rate limit.
3. Run Test live URL in URL Inspection. According to Google (URL Inspection tool) the live test fetches the page in real time and reports whether the fetch succeeded, so you can confirm a fix before validating it.
4. Check the Crawl stats report. Google documents (Crawl stats report) an Other client error (4XX) row in the Crawl responses table, which dates the start of the problem and shows its share of all requests.
5. Search the logs for the affected paths, verifying the IPs with a reverse DNS lookup as described in Google's documentation on verifying Googlebot. Logs show the raw request line, which exposes bad characters invisible in the report.
6. Run the URL through the redirect checker to see whether the 4xx is on the final URL or on a hop in a chain.

How to fix it

1. Rate limit (429): raise the threshold for verified Googlebot or exempt Google's published IP ranges from the rule. If the load is real, keep the 429 but only for a short time; Google's documentation on reducing crawl rate treats 429 and 503 as acceptable for hours or 1-2 days and warns that URLs may be dropped from the index after multiple days.
2. Removed content (410): leave the status, remove the URLs from the sitemap and from internal links so Google stops finding them. Redirect only where a real replacement exists; see the Not found (404) entry for the decision rules.
3. Bad characters (400): fix the template that generates the links, encode the URL properly and redirect the broken variant if it has external links.
4. Too long (414): cap the number of parameters, block the runaway filter combinations in robots.txt and make sure redirect rules do not append parameters on every hop.
5. Method and negotiation (405, 406): allow GET on routes that are linked, or remove the links; loosen ModSecurity rules that match on headers Googlebot sends.
6. Once the live test returns 200 or the intended 404, click Validate fix in the Page indexing report.

When it is fine and when to act

The status is fine when the URLs are deliberately gone (410), when they are junk URLs that never existed, or when they are POST-only endpoints linked by mistake. Clean up the links and ignore the rest.

Act now when the code is 429 on pages that should be indexed, when the count rises after a firewall or CDN change, or when important pages appear in the list at all. A 429 points at a rule that is throttling Google on your whole site, and the other codes on real pages mean those pages are leaving the index.

How IndexRise helps

IndexRise requests every monitored page each night and stores the exact status code, not just the family. When a page that returned 200 in the baseline returns 400, 410 or 414 today, it raises a critical event and confirms it on the next check, so you see the number Search Console hides. A 429 is different: it means the site refused our check, so the URL is shown as "couldn't check" and keeps its last known state - the same as a request that never completed, a separate state from "page down". The full list of checks is on the features page.