Indexed, though blocked by robots.txt
Search Console status: Indexed, though blocked by robots.txt
Google can't crawl the page because robots.txt blocks it, but it indexed the URL anyway — usually because links point to it. It then appears in results without a description. The fix depends on your goal: if the page should rank, unblock it; if not, unblock it and add noindex.
What this status means
The page is blocked in robots.txt, so Google doesn't fetch it, but the URL still made it into the index, usually thanks to links from other pages. In results it shows a title taken from links and a note that no description is available.
Where it comes from
Most often someone wanted pages out of Google and blocked them in robots.txt instead of using noindex. That doesn't work: the block stops crawling, and URLs that were already indexed stay.
How to fix it
| Goal | What to do |
|---|---|
| the page should rank | remove the Disallow rule that blocks it |
| the page should disappear from results | unblock it in robots.txt and add noindex |
| a file (PDF, image) should disappear | unblock it and set an X-Robots-Tag: noindex header |
Once the pages have dropped out of the index (after Google sees the noindex), you can block them in robots.txt again if you want to save crawl effort.
How to check the rule
The robots.txt checker shows for each URL whether Googlebot may crawl it and which line decides.
How to keep an eye on it
IndexRise fetches robots.txt every night and records every change with the lines that changed. When a new rule blocks a monitored page, you get a critical event with the cause.
Frequently asked questions
Why does Google index a page it can't visit?
Because robots.txt blocks crawling, not indexing. Google knows the URL from links and can show it in results based on link text, even without seeing the content.
Why doesn't noindex work on a blocked page?
Google has to fetch the page to see the noindex. If robots.txt doesn't allow it, the directive stays invisible.
Do I need to fix this warning?
If these pages shouldn't be in results — yes, because they look bad there (no description). If they should — also yes, because Google can't see their content.
Updated: 2026-10-20