What's the difference between noindex and Disallow in robots.txt?

Noindex removes the page from results but lets the crawler visit it. Disallow stops visits, but the page can stay in results without a description if other pages link to it.

Does “noindex, follow” make sense?

Yes, for pages that shouldn't rank but link to important content, such as paginated filter pages. With long-term noindex, though, Google may eventually follow links from such pages less often.

How fast does noindex work?

After the crawler's next visit — from a few days to a few weeks, depending on how often Google crawls the page.

Noindex (meta robots)

Noindex is a directive that tells search engines: don't show this page in results. It goes in a <meta name="robots" content="noindex"> tag or in the X-Robots-Tag: noindex HTTP header. Google has to fetch the page to see the noindex — so the page must not be blocked in robots.txt at the same time.

Where noindex goes

in the HTML head: <meta name="robots" content="noindex">,
for Google only: <meta name="googlebot" content="noindex">,
in the HTTP response header: X-Robots-Tag: noindex — which also works for PDFs and images (more on X-Robots-Tag).

Directives can be combined: noindex, nofollow, noindex, follow.

When to use noindex

cart, checkout, customer account, thank-you pages,
internal search results and low-value filter combinations,
test pages and drafts,
duplicate content you can't consolidate with a redirect or canonical.

Noindex vs robots.txt

	noindex	Disallow in robots.txt
Does the crawler visit the page	yes	no
Can the page appear in results	no	yes, without a description
What it's for	removing from the index	saving crawl effort

The most common mistake: adding noindex while also blocking the page in robots.txt. The crawler never visits, so it never sees the noindex.

How to check whether a page has noindex

Search the page source for noindex and check the response headers — the header isn't visible in the source. The redirect checker shows both for the final page. In Search Console, such pages appear under "Excluded by ‘noindex’ tag".

Monitoring noindex

A noindex carried over from staging is one of the most common causes of a sudden traffic drop.

IndexRise checks meta robots and X-Robots-Tag on monitored pages every night. A noindex that wasn't in the baseline is a critical "Page is no longer indexable" event — confirmed on the next check so a momentary glitch doesn't wake anyone up.