Blocked due to unauthorized request (401)

What this status means

According to Google's documentation (Page indexing report), the status appears when the page was blocked to Googlebot by a request for authorization, which is a 401 response. Google's advice in the same document has two branches: remove the authorization requirement for the page, or allow Googlebot to access it after verifying its identity. The document also notes you can reproduce the error by visiting the page in incognito mode.

A 401 is the server saying "who are you?" and expecting a WWW-Authenticate challenge to be answered. Googlebot does not answer it. Google's HTTP status codes documentation says all 4xx codes except 429 are treated the same: Google ignores the content and removes a previously indexed URL from the index. The same document asks you not to use 401 or 403 to limit crawling, because they have no effect on crawl rate. The HTTP status codes glossary entry lists the rest of the family.

In practice the status is almost always a configuration leak, not a decision.

Common causes

HTTP basic auth copied from staging

The most common case. Staging is protected with .htaccess and .htpasswd or an nginx auth_basic block. During a deploy the config is copied to production, or a shared virtual host template includes the directive for both. The whole site, or one directory such as /blog/, returns 401.

Protected directories that hold public assets

Auth applied to a directory that also serves content linked from public pages: a /downloads/ folder with PDFs, a /media/ path with product images or an old /wp-admin/ rule that also covers admin-ajax.php used by the front end. Google discovers the URLs through links and reports them as 401.

Members-only sections and paywalls

A forum, a customer portal, course content or a B2B price list that requires login by design. These URLs are linked from the public site or listed in the sitemap, so Google tries them and gets a 401. If the content is meant to be private, the status is correct and the only cleanup is to stop advertising the URLs to Google.

CDN or reverse-proxy access rules

Cloudflare Access, AWS CloudFront with a Lambda@Edge authorizer or an nginx proxy in front of a headless CMS can return 401 before the request reaches the origin. The origin looks fine when tested directly, so the rule stays unnoticed.

Redirects into a protected host

A production URL redirects to staging.example.com or to an internal tool that requires login. Google follows the redirect and the final URL returns 401. The report shows the original URL, so the cause is hidden one hop away.

Misconfigured 404 or error templates

A custom error handler that lives under a protected path. The page the user wanted is gone, the server tries to render the 404 template from /private/errors/, and the auth rule on that directory turns the answer into a 401. Every missing page then shows up as unauthorized instead of not found.

How to confirm it

1. Open the URL in an incognito window. A browser login dialog or a 401 page confirms the challenge comes from the server, not from your session.
2. Request the page without cookies: curl -sI https://example.com/page/. Look for HTTP/1.1 401 and a WWW-Authenticate header. Then test with the Googlebot user agent: curl -sI -A "Googlebot/2.1 (+http://www.google.com/bot.html)" https://example.com/page/. If only the first request gets a 401, a rule is keyed on the user agent.
3. Run Test live URL in URL Inspection. According to Google (URL Inspection tool) the live test fetches the page in real time and reports whether the page fetch succeeded, so you can check whether the challenge is still in place after a change.
4. Check the Crawl stats report. Google documents (Crawl stats report) a Crawl responses row named Unauthorized (401/407), which shows how large a share of all crawl requests hit the block and when it started.
5. Walk the chain with the redirect checker. It shows the status code and headers of every hop, so a redirect into a protected host is visible at once.
6. Check the logs for 401 entries from Googlebot and verify the IP with a reverse DNS lookup, as described in Google's documentation on verifying Googlebot.

How to fix it

1. Staging auth on production: remove the auth_basic block or the .htaccess require directives from the production virtual host. Keep the auth on staging and move it to a host-level config so a deploy cannot copy it.
2. Protected directory with public assets: narrow the rule to the admin paths only, or move the public files to a separate directory without auth.
3. Members-only content by design: keep the 401, but remove those URLs from the sitemap and from public internal links, and consider a robots.txt disallow so Google stops requesting them. The indexability entry explains which signals keep a page out of the index.
4. CDN or proxy rule: add an exception for the public paths, or allow verified Googlebot using Google's published IP ranges. Do not allowlist by user-agent string alone, because it is easy to spoof.
5. Redirect into a protected host: point the redirect to the public URL or remove it, and check the page with redirect entries in the same report for the same pattern.
6. Error template under auth: move the error template outside the protected path and make sure a missing page returns a plain 404.
7. When the live test fetches the page successfully, click Validate fix in the Page indexing report.

When it is fine and when to act

A 401 is correct when the content is private: staging, admin panels, customer accounts, paid content without a free preview. Nothing needs fixing, though removing these URLs from sitemaps and public links keeps the report clean.

Act when the affected URLs include pages that earn traffic, when the number of 401 URLs jumps after a deploy, or when the Crawl stats row for 401/407 appears where it used to be empty. A 401 on an indexed page removes it from Search after the next crawl, so the cost grows daily. The related Blocked due to access forbidden (403) status comes from the same family of access rules and often shows up alongside this one.

How IndexRise helps

IndexRise fetches every monitored page each night without cookies or credentials, the way Googlebot does, and records the status code. When a page that returned 200 yesterday returns 401 today, it raises a critical event and confirms it on the next check. A request that could not be completed at all is reported as "couldn't check" rather than "page down", so a network problem does not look like an auth rule. The full list of checks is on the features page.