robots.txt checker
Paste up to 10 URLs and see whether a search engine crawler may visit each one — with the exact rule from robots.txt that decides. Rules are matched the way Google does it.
How this robots.txt checker works
We fetch robots.txt from each host in your list and pick the group of rules for the crawler you chose: the Googlebot group if there is one, otherwise the group for all crawlers (*).
Then we compare each URL's path and query with every Allow and Disallow rule. The longest matching rule wins; if an Allow and a Disallow are equally long, Allow wins. The * wildcard matches any text and $ marks the end of the URL — the same rules Google documents.
What the robots.txt status means
200: the rules apply. 404 or another 4xx: there is no file, so nothing is blocked. 5xx or no response: Google treats the site as temporarily off-limits and stops crawling it — a server error on robots.txt can hurt more than a wrong rule.
Blocked doesn't always mean deindexed
Disallow stops crawling, not indexing. A blocked URL can still appear in results without a description if other pages link to it. To keep a page out of the index, allow crawling and use noindex — Google has to see the noindex to obey it.
Questions about robots.txt
How do I check if a URL is blocked by robots.txt?
Paste the URL into the checker above. You'll see whether it's allowed or blocked for the crawler you chose and which line in robots.txt decides.
Does robots.txt Disallow remove a page from Google?
No. Disallow only stops crawling. A blocked page can stay in the index without a description. Use noindex on a crawlable page to remove it.
Which rule wins when Allow and Disallow both match?
The more specific one — the rule with the longer path. If both are the same length, Allow wins.
What happens if robots.txt returns a server error?
Google stops crawling the site until it can read the file again, and after a longer outage falls back to the last version it saw. That's why a 5xx on robots.txt is a critical alert in IndexRise.
How do I monitor robots.txt changes?
Add your site to IndexRise. We fetch robots.txt every night with your URLs and alert you when a change blocks monitored pages or the file becomes unavailable.