noindextags

Noindextags, or noindex directives, are mechanisms used on web pages to request that search engines exclude the page from their index. They rely on crawlers observing the directive when visiting the resource.

Common forms include the meta robots tag placed in the page head, such as: <meta name="robots" content="noindex"> or <meta name="robots" content="noindex, nofollow">. Some engines also recognize engine-specific variants like <meta name="googlebot" content="noindex">. In addition, the HTTP header X-Robots-Tag can convey a noindex instruction, for example: X-Robots-Tag: noindex. All of these require the page to be reachable by the crawler to be effective.

When a crawler encounters a noindex directive, it generally will not include the page in the search index. Depending on the engine and prior indexing, the page’s URL may still appear in search results for a time, sometimes with limited or no snippet, until the engine recrawls and processes the change. Noindextags do not guarantee instant removal and may take days or weeks to fully propagate.

Robots.txt, which governs crawling, operates differently. If a page is disallowed by robots.txt, crawlers cannot fetch the page to read noindex directives, so noindex cannot be applied via the latter method. Therefore, for removal from indexing, it is typically necessary to place a noindex directive on the page itself or use the corresponding HTTP header.

Use cases for noindextags include removing outdated or duplicate content, protecting sensitive information, or signaling content that should not appear in search results while allowing access for users or other systems.