AI-generated illustration
Use robots meta tags correctly: noindex, nofollow, nosnippet and Co.Meta-robots directives only develop their benefits with clean demarcation and consistent technology
Robots meta tags and X robots tags control the handling of retrieved content. They must be distinguished from robots.txt rules for retrieval and from real access protection.
First, determine what a URL should do. After that, check all robots specifications that are actually shipped, including HTTP headers.
HTML meta tag or HTTP header?
For an HTML page, the default can be in the head. For other response types, such as a PDF, a matching HTTP header is a possible output location. Application, server and CDN can be different sources of such a specification.
A source code without noindex therefore does not prove that the answer does not contain an exclusion. Always read the headers of a normal call if the message and the visible setting do not match.
Conscious exclusions
Both examples presuppose that the content can be retrieved. A simultaneous crawl lock can prevent the default from being read. Confidential files require access control regardless.
<!-- Beispiel: interne Suchergebnisse -->
<meta name="robots" content="noindex,follow">
# Beispiel: nicht für die Suche vorgesehene Datei
HTTP/2 200
Content-Type: application/pdf
X-Robots-Tag: noindexFinding an Unintentional Noindex
- Publication intent and page type: performance page or deliberately excluded user function?
- Check meta-robots and X-robots tag of the public URL.
- Limit the difference between page field, template and server rule.
- Change the responsible source and take into account upstream caches.
- Retrieve response; observe actual later indexing separately.
Link control is not a substitute for indexing control
An internally linked shopping cart may consciously carry noindex. Removing the link would damage its user task. Similarly, nofollow is not a reliable way to remove a landing page from search.
In the case of several specifications, the last line found is not simply decisive. Avoid conflicting sources and check the rules applicable to the respective crawler. The technical correction does not guarantee the immediate inclusion of a URL.
Frequent questions
Does noindex prevent access to a page?
No. It is not password protection. Anyone who has to keep content confidential needs a suitable access rule.
Can robots.txt replace a noindex?
No. A crawl lock and an indexation ban are different mechanisms. In particular, a lock can prevent reading of the noindex.