Blocked due to access forbidden (403)

What this status means

According to Google's documentation (Page indexing report), HTTP 403 means the user agent provided credentials but was not granted access. Googlebot never provides credentials, so Google's position is that a server returning 403 to it is answering incorrectly, and the page will not be indexed. The same document gives two ways out: admit non-signed-in users, or explicitly allow Googlebot requests without authentication, ideally after verifying that they really come from Google.

Google's documentation on HTTP status codes puts 403 in the same group as every other 4xx except 429: the content is ignored and a previously indexed URL is removed from the index. It also asks you not to use 401 or 403 to limit crawling, because those codes have no effect on crawl rate. The HTTP status codes entry explains the rest of the family.

The 403 differs from a 401 in intent. A 401 asks for a login and lists what kind; a 403 simply refuses. Most 403s to Googlebot come from security layers rather than from the application.

Common causes

WAF and bot protection

Cloudflare Bot Fight Mode, managed WAF rulesets, Sucuri, Imperva, AWS WAF with a bot-control rule group or ModSecurity with the OWASP core rule set. Each can score a request as automated and answer 403. Googlebot has no cookies, runs from data-center ranges and asks for many URLs quickly, which is what these products look for. A ruleset update or a tightened sensitivity level is enough to block a crawler that was fine last week.

Geo-blocking

The site serves one country and blocks the rest at the CDN or server level. Google's documentation on Googlebot notes that Googlebot crawls from IP addresses in the US, so a rule that allows only Polish or German visitors returns 403 to every crawl.

IP-based blocking and rate limits

A fail2ban jail, a server-level deny list or a plugin that bans an IP after too many requests in a minute. A crawler fetching 500 URLs in a short time trips the rule, and the ban can last hours.

User-agent filtering

An .htaccess or nginx rule that blocks "bot" or "crawler" in the user agent to stop scrapers also blocks Googlebot. Security plugins that ship with a blocklist of bad bots sometimes include patterns that match Google's crawlers.

Hotlink protection and referrer rules

Rules that refuse requests for images or PDFs without a matching Referer header. Googlebot requests resources without a referrer, so the files return 403. The HTML page is fine, but the PDF catalogs listed in the sitemap show up as forbidden.

File and directory permissions

A deploy that sets chmod 600 on files, a missing index file in a directory with listing turned off, or wrong SELinux contexts after a migration. Apache and nginx answer 403 in all of these cases, often for a whole subfolder such as /media/.

How to confirm it

1. Request the page from the command line without cookies: curl -sI https://example.com/page/. Then repeat with the Googlebot user agent: curl -sI -A "Googlebot/2.1 (+http://www.google.com/bot.html)" https://example.com/page/. A 403 only on the second request means a user-agent rule.
2. Read the headers of the 403. Server: cloudflare with cf-mitigated: challenge, or a Sucuri or Imperva signature, tells you which layer answered. A plain Apache or nginx error page points at permissions or a local rule.
3. Run Test live URL in URL Inspection. According to Google (URL Inspection tool) the live test fetches the page in real time, so it shows whether the current rules still block Google after a change.
4. Open the Crawl stats report. Google documents (Crawl stats report) a Crawl responses table with an Other client error (4XX) row, which it marks as best to fix. A jump in that row dates the rule change.
5. Look in the firewall or CDN event log for blocked requests from Google's ranges and verify the IPs with a reverse DNS lookup, as described in Google's documentation on verifying Googlebot. The same document links JSON files with Google's crawler IP ranges for an allowlist.
6. Run the URL through the redirect checker to see the code and headers of every hop, in case the 403 sits behind a redirect.

How to fix it

1. WAF or bot protection: add a skip rule for verified bots or for Google's published IP ranges, placed before the managed rules. In Cloudflare, check that Bot Fight Mode does not challenge verified bots. Do not allowlist by user-agent string alone.
2. Geo-blocking: allow Google's crawler ranges regardless of country, or replace the block with a country-specific redirect that still returns content to crawlers.
3. IP bans and rate limits: raise the threshold for verified crawlers or exclude their ranges from the jail. If the load is real, use 503 or 429 for a short time, which Google documents as the correct signal for overload.
4. User-agent filtering: remove generic patterns such as bot from deny rules and keep only named scrapers.
5. Hotlink protection: allow empty referrers, or exempt crawler ranges from the rule, so images and PDFs load for Google.
6. Permissions: set readable permissions on the affected files and directories, check for a missing index file, and compare with a working directory on the same server.
7. After the live test fetches the page successfully, click Validate fix in the Page indexing report and use Request indexing for the most valuable URLs.

When it is fine and when to act

A 403 is fine for genuinely private content and for administrative paths such as /wp-admin/ or /.git/. Those URLs should not be in your sitemap, and a robots.txt disallow keeps the report quiet; the robots.txt entry covers the syntax.

Act at once when the affected URLs include pages that were indexed, when the number of 403s rises after a security or CDN change, or when the status covers resources such as CSS and images that Google needs to render your pages. A security product that blocks Googlebot removes your site from Search one recrawl at a time, and nothing looks wrong to users.

How IndexRise helps

IndexRise requests every monitored page each night from a fixed set of addresses and records the status code. When a page that returned 200 yesterday returns 403 today, it raises a critical event and confirms it on the next check, which catches a WAF or CDN rule change the morning after it ships. A request that was blocked before it reached the server is reported as "couldn't check" rather than "page down", so you know whether to look at the firewall or at the application. The full list of checks is on the features page.