Test robots.txt changes before publication

Making the consequences of a rule change visible

An additional disallow can sensibly block an internal area, but also hit important product or category pages. The simulator evaluates the same URL list with the previous and planned files. It shows newly blocked, newly released and unchanged URLs as well as the relevant rule with row number.

Select appropriate test quantity

Use a list of important URLs or import a sitemap. A sitemap index is tracked only to a limited extent; unread files and truncated URL sets are named. The simulation is therefore always a statement about the selected sample, not about all conceivable URLs of your website. Entries of other origins are not incorrectly assessed as allowed.

Example: block a catalogue, release products

For Disallow: /catalog/ and Allow: /catalog/products/, a suitable product path is allowed if the more specific rule applies. The comparison also shows whether a new user-agent group displaces the previous star group for Googlebot. Query parameters, upper case and encoded paths should be tested with realistic examples.

Differentiating Crawling and Indexing

A robots.txt lock is not a noindex. Locked URLs may still appear in search results, while Google may no longer be able to retrieve the page content. The local rule test does not replace checking HTTP status, meta-robots or canonicals. Basis: Google's robots.txt rules.

Frequent questions