Server error (5xx)

What this status means

According to Google's documentation (Page indexing report), the status means your server returned a 500-level error when Googlebot requested the page. The report does not say which code, so you need URL Inspection, the logs or your own request to find out.

Google treats the whole 5xx family the same way. Its documentation on HTTP status codes says that 5xx errors make crawlers slow down, that an already indexed URL is preserved for a while but eventually dropped, and that the crawl rate recovers gradually once the server returns 2xx again. The same document groups 429 with server errors.

All four land in the same bucket in Search Console; the HTTP status codes entry describes the full list.

Common causes

Application errors on the origin

A PHP fatal error after a plugin update, a theme calling a function that no longer exists, a template that throws on a missing field. The home page still works, the error sits on a category template, and nobody notices.

Database overload

At peak traffic or during a full crawl the database refuses new connections and the application returns 500 or 503. Googlebot requesting thousands of filtered listing URLs can trigger it on its own. Google's crawl budget documentation describes how the crawler backs off when the site answers with server errors, which hides the problem.

Proxy and CDN timeouts

Cloudflare, Fastly, an nginx reverse proxy or a load balancer sits in front of the origin. When the origin answers slowly, the edge returns 502 or 504. The origin may only be slower than the edge timeout.

Maintenance mode left on

A CMS plugin or deploy script turns on maintenance mode and returns 503. If the deploy fails halfway, the whole site returns 503 for days. Google's guidance on pausing an online business describes 503 as suitable for a 1-2 day shutdown and warns that a longer full disable has significant effects in Search.

Bot protection answering with 503

Some WAF and bot-management products answer suspicious traffic with 503 instead of 403 or 429. Cloudflare "Under Attack" mode and JavaScript challenges can serve 503 to clients that do not run JavaScript. If the rule hits Googlebot, every page is a server error to Google.

robots.txt returning 5xx

A more serious case. Google's specification on how Google interprets robots.txt says that when robots.txt returns a server error, Google stops crawling the site for the first 12 hours, then uses the last cached copy for up to 30 days, and after that behaves as if there were no robots.txt if the site is otherwise available. The pause guidance states it plainly: do not return 503 for robots.txt, because it blocks all crawling. Check the file with the robots.txt checker; the robots.txt entry covers the rest of the rules.

How to confirm it

1. Run Test live URL in URL Inspection. According to Google (URL Inspection tool) the live test fetches the page in real time and shows the Page fetch result, including site-wide errors such as Server connection error or Hostload exceeded.
2. Request the page yourself: curl -sI https://example.com/page/, then with the Googlebot user agent: curl -sI -A "Googlebot/2.1 (+http://www.google.com/bot.html)" https://example.com/page/. A different code on the second request points at a WAF or bot rule.
3. Check the Crawl stats report. Google documents (Crawl stats report) a Host status section with server connectivity and a Crawl responses table with a Server error (5XX) row. A spike there dates the incident.
4. Search the server and edge logs for " 50[0-4] " and filter on Googlebot. Confirm the IP with a reverse DNS lookup as described in Google's documentation on verifying Googlebot.
5. Run the URL through the redirect checker. It shows the code and headers of every hop, so you see whether the 5xx comes from the final URL or from a redirect step.

How to fix it

1. Application error: read the error log, reproduce on staging, roll back the plugin, theme or release that introduced it.
2. Database overload: cache listing and filter pages and disallow infinite filter combinations in robots.txt so the crawler does not generate the load.
3. Proxy timeout: raise the edge timeout as a stopgap, then fix the slow query or external call behind the page.
4. Maintenance mode: switch it off and confirm with curl -sI that the site returns 200. For future windows use 503 with a Retry-After header, as Google recommends.
5. Bot protection: allow verified Googlebot through the rule that returns 503, using Google's published IP ranges rather than a user-agent string.
6. robots.txt: serve it as a static file that returns 200 or 404 even when the application is down.
7. When the sample URLs return 200 in the live test, click Validate fix in the Page indexing report.

When it is fine and when to act

A handful of URLs with a 5xx on one day, no repeat on the next crawl and a green Host status is noise: a deploy, a short database restart. Google's documentation on reducing crawl rate treats a 500, 503 or 429 that lasts a couple of hours or 1-2 days as acceptable, and warns that URLs seen with those codes over several days may be dropped from the index.

Act immediately when:

• the number of affected URLs grows between report updates,
• the Crawl stats report shows a red Host status or robots.txt returns 5xx,
• the error appears only for Googlebot and not for a normal request.

Dropped pages come back after recrawling, but there is no documented timeline. Pages Google postponed because the server kept pushing back may show up as Discovered - currently not indexed rather than as a server error.

How IndexRise helps

IndexRise requests every monitored page each night and records the status code. When a page that returned 200 in the baseline returns 500, 502, 503 or 504 today, it raises a critical event right away - by default a server error is not held back for a second check, because it usually means the whole host is in trouble. The confirmation can be switched on per project if your server has frequent short blips. A timeout or blocked request is reported as "couldn't check", a separate state from "page down". The full list of checks is on the features page.