How do I check whether a URL is blocked by robots.txt?

Use the robots.txt checker: enter the URL, choose the crawler, and you'll see whether it may crawl the page and which rule decides.

Does Disallow remove a page from Google?

No. Disallow stops crawling. The page can stay in the index without a description if other pages link to it. To remove it from the index, use noindex.

What if robots.txt returns a 404?

For Google, no file means no restrictions — it may crawl everything. A 5xx error is more dangerous: Google then pauses crawling the site.

Robots.txt

Robots.txt is a text file at the root of a domain (/robots.txt) that tells search engine crawlers which URLs not to crawl. Disallow and Allow rules control crawling, not indexing — a blocked page can still appear in results. A mistake in robots.txt can block an entire site with one line.

What robots.txt looks like

User-agent: *

Disallow: /cart/

Disallow: /search

Allow: /cart/terms

Sitemap: https://shop.com/sitemap.xml

Each group starts with User-agent (a crawler name, or * for all), followed by rules. The Sitemap line points to your XML sitemap.

How Google reads the rules

Googlebot uses the Googlebot group, and if there isn't one, the * group,
a rule matches the start of the path: Disallow: /cart also blocks /cart-deals,
* matches any text, $ marks the end of the URL: Disallow: /*.pdf$,
when several rules match, the longest wins; on a tie, Allow wins,
an empty Disallow: blocks nothing.

What the robots.txt status code means

Status	How Google treats it
200	applies the rules in the file
404, 410 and other 4xx	no restrictions
5xx, 429, no response	pauses crawling the site, later falls back to the last known version

Common mistakes

Disallow: / copied from staging to production — the whole site blocked,
blocking CSS and JavaScript directories — Google can't render pages properly,
blocking pages that have noindex — the crawler never sees the noindex,
blocking the sitemap or its directory,
a rule that's too broad, such as Disallow: /p blocking both /products/ and /promotions/.

How to test robots.txt

The robots.txt checker fetches the file, matches rules the way Google does and shows, for each URL, whether it's allowed and which line decides. In Search Console, the robots.txt report (Settings) shows which version of the file Google saw.

Monitoring robots.txt

One line in robots.txt can block a whole section of a site — while nothing changes on the pages themselves.

IndexRise fetches each host's robots.txt every night. When a new rule blocks monitored pages, each one gets a "Page is no longer indexable" event naming the rule. An unavailable robots.txt (5xx) is a separate critical event, and every change to the file is recorded with the lines that changed.