What is the default when there is no meta robots tag?

Index and follow. Google's documentation lists "all" as the default value, which means no restrictions; writing it explicitly has no effect.

Does the meta robots tag have to be in the head?

Google says it does not enforce placement and will respect a robots meta tag in the body as well. Keep it in the head anyway, so other crawlers and tools find it.

Does meta robots noindex work if the page is blocked in robots.txt?

No. The directives can only be read if crawlers are allowed to fetch the page. A URL disallowed in robots.txt may still be indexed without its content.

Is noarchive still a thing?

Google's specification lists noarchive among historical rules it no longer uses, because the cached link feature no longer exists. It is ignored.

What is the difference between name=robots and name=googlebot?

name=robots addresses all crawlers; name=googlebot addresses only Google's text results crawler. Google supports two tokens, googlebot and googlebot-news, and ignores other names.

Can meta robots stop AI Overviews from using my page?

nosnippet and max-snippet limit what Google may use as a direct input for AI Overviews and AI Mode, according to the specification. They do not remove the page from the index.

Meta robots tag (robots meta tag)

The meta robots tag is an HTML element in the head of a page that tells search engines how to index and display that page. Its most used values are noindex and nofollow, but Google documents a dozen directives, from snippet length to an expiry date. The same rules can be sent as an X-Robots-Tag HTTP header, and when they conflict, the more restrictive one wins.

The meta robots tag sets page-level indexing and snippet rules for search engines in one HTML element. Google supports noindex, nofollow, none, nosnippet, max-snippet, max-image-preview, max-video-preview, notranslate, noimageindex, unavailable_after and indexifembedded.

How it looks

The tag sits in the <head> and takes a comma-separated list of rules in the content attribute:

<meta name="robots" content="noindex, nofollow">

<meta name="robots" content="max-snippet:160, max-image-preview:large">

<meta name="googlebot" content="notranslate">

<meta name="robots" content="unavailable_after: 2026-12-31">

Per Google's robots meta tag specification, both the name and content attributes are case-insensitive, and rules can be combined in one tag or split across several tags. To address Google alone, replace robots with the user agent token googlebot (all text results) or googlebot-news (news results); the specification says other values are ignored.

Directives Google supports

The specification lists these valid indexing and serving rules:

Rule	What it does
`all`	No restrictions; the default, so listing it has no effect
`noindex`	Do not show the page, media or resource in search results
`nofollow`	Do not follow the links on the page
`none`	Equivalent to `noindex, nofollow`
`nosnippet`	No text snippet or video preview; also blocks use as direct input for AI Overviews and AI Mode
`indexifembedded`	Allow indexing of content embedded in another page via iframe despite `noindex`; only works together with `noindex`
`max-snippet:[number]`	Maximum characters in the text snippet; `0` equals `nosnippet`, `-1` leaves the length to Google
`max-image-preview:[setting]`	Image preview size: `none`, `standard` or `large`
`max-video-preview:[number]`	Maximum seconds of video preview; `0` allows at most a static image, `-1` means no limit
`notranslate`	Do not offer translation of the page in results
`noimageindex`	Do not index images on the page
`unavailable_after: [date/time]`	Stop showing the page after the given date; Googlebot then lowers the crawl rate of the URL considerably

Three rules are documented as historical and ignored by Google Search: noarchive (the cached link no longer exists), nocache and nositelinkssearchbox. If a CMS still emits them, they do nothing.

How Google handles it

Google can only follow the tag if it can fetch the page. The specification says these settings are read and followed only if crawlers are allowed to access the pages that include them, which is why a noindex behind a robots.txt disallow does not work. Google's page on blocking indexing with noindex repeats this: the page must not be blocked by robots.txt for the tag to be seen.

Conflicts resolve toward the stricter rule. The specification gives two examples: with both max-snippet:50 and nosnippet on a page, nosnippet applies; and with <meta name="robots" content="nofollow"> plus <meta name="googlebot" content="noindex">, Googlebot reads the page as noindex, nofollow, because the engine uses the sum of the negative rules.

The tag applies to search crawlers. To block a non-search crawler such as AdsBot-Google, the specification says you may need a rule addressed to that crawler, for example <meta name="AdsBot-Google" content="noindex">.

Meta robots vs X-Robots-Tag

Any rule from the table can be sent as an HTTP header instead: X-Robots-Tag: noindex, nofollow. The header works for PDFs, images and other files that have no head, and can be set once for a whole directory. Google reads both and applies the stricter combination. The header variant is described in the entry on X-Robots-Tag; the semantics of the key directive are in the entry on noindex.

When to use it

Thin or internal pages that must stay reachable but out of results: noindex.
Campaign pages with an end date: unavailable_after.
Content you do not want quoted in snippets or AI answers: nosnippet or max-snippet.
Image galleries with licensing restrictions: noimageindex.

Common mistakes

Mistake	Effect
`noindex` on a page also disallowed in robots.txt	Google never sees the tag; the URL can be indexed without content
Staging `noindex` deployed to production	Pages drop out of the index after the next crawl
Two robots tags with different values	The stricter one wins; a stray `noindex` beats an intended `index`
`nofollow` used to save crawl budget on internal links	Links may still be discovered elsewhere; robots.txt is the documented tool
`max-snippet` without a number	Rule is ignored as unparseable
Relying on `noarchive`	Ignored by Google since the cached link was removed

How to check it

View the page source and search for name="robots" and name="googlebot". Then check the header, because a page can carry both:

curl -sI https://example.com/page/ | grep -i "x-robots-tag"

In devtools, the Elements panel shows the rendered head, which matters when JavaScript injects or rewrites the tag.

In Search Console, inspect the URL. A page excluded by the tag appears in the Pages report as Excluded by 'noindex' tag, and the URL Inspection tool reports whether indexing is allowed and why not.

The indexability checker reads meta robots and X-Robots-Tag together with the status code, canonical and robots.txt rules for up to 10 URLs and names the directive that blocks indexing.

How IndexRise monitors it

IndexRise reads meta robots and the X-Robots-Tag header on every monitored URL each night. A noindex that was not in the baseline raises a critical "Page is no longer indexable" event with the cause stated, either "noindex (meta robots)" or "noindex (X-Robots-Tag)". A second robots tag on a page is reported as a duplicated element. A noindex present since the first check is recorded as the existing state, not as a change, so a deliberate exclusion does not generate daily noise.

One line in a template can switch an entire site to noindex. Check the tag in the source and in the headers after every release, or let a nightly diff do it.