Blocked by robots.txt

What this status means

Google knows the URL, but robots.txt forbids crawling it. The crawler doesn't fetch the page, so it doesn't know its content or directives — including noindex or canonical.

When it's fine

• cart, checkout, customer account,
• internal search results (/search, ?q=),
• filter and sort combinations that multiply low-value URLs,
• technical resources that don't matter for search.

When to act

Typical causes of an accidental block:

• Disallow: / copied from staging to production,
• a rule that's too short, such as Disallow: /p blocking /products/ and /posts/,
• a wildcard rule matching more than intended,
• a blocked directory that, after a structure change, now contains important pages.

How to fix it

1. Check the URL in the robots.txt checker — you'll see the rule that blocks it.
2. Fix or remove the rule; to block only part of a directory, add a longer Allow rule.
3. If you want the page out of results, don't block it in robots.txt — use noindex.
4. Submit the most important unblocked pages in URL Inspection.

How to keep an eye on it

IndexRise fetches robots.txt every night and checks every monitored page against the Googlebot rules. When a new rule blocks a page, you get a critical "Page is no longer indexable" event naming the robots.txt line.