X-Robots-Tag

X-Robots-Tag is an HTTP response header that carries the same directives as meta robots — such as noindex or nofollow. It's set by the server, not the page code, so it also works for PDFs, images and other non-HTML files. It's invisible in the page source, which makes it easy to overlook.

What it looks like

X-Robots-Tag: noindex

X-Robots-Tag: noindex, nofollow

X-Robots-Tag: googlebot: noindex — a directive for one crawler only.

When it's useful

keeping PDFs, DOCX files and images, which have no HTML, out of the index,
setting directives for a whole directory or file type with one server rule,
staging environments: one header for the whole domain instead of template changes.

How to set it

In Apache (.htaccess) for all PDF files:

<FilesMatch "\.pdf$"> Header set X-Robots-Tag "noindex" </FilesMatch>

In nginx: add_header X-Robots-Tag "noindex"; in the relevant location block. A CDN can also set the header, for example with a Cloudflare transform rule.

The biggest risk

A rule adding X-Robots-Tag, left over from staging work or scoped too broadly, can drop an entire site from the index — while nothing changes in the page source.

That's why, after a traffic drop following a release, always check response headers, not just the source. The redirect checker lists the headers of every hop and warns when the final page has noindex.

Monitoring

IndexRise reads X-Robots-Tag on every nightly check, together with meta robots. A noindex in the header that wasn't in the baseline is a critical "Page is no longer indexable" event with the cause "noindex (X-Robots-Tag)".

Check it on your siteRedirect checkerThe full chain with status codes, targets and headers — up to 10 URLs at once.

Frequently asked questions

How is X-Robots-Tag different from meta robots?

They work the same way, but meta robots lives in the HTML and X-Robots-Tag in the response header. The header also works for PDFs and images, which have no head section.

How do I check whether a page sends X-Robots-Tag?

Look at the response headers in your browser's developer tools (Network tab) or in the redirect checker, which lists the headers of every hop.

What if meta robots and X-Robots-Tag disagree?

Google considers both and applies the more restrictive directive. A noindex in either one is enough to drop the page from the index.

Updated: 2026-10-20