Free · without registration
Which URLs really blocks your robots.txt?
Insert your robots.txt and check specific URLs against the rules. The tester shows which bot group and which rule are the decisive factor.
Processing directly in your browser · no file upload · no account required
Allow and Disallow
The tester selects the most specific matching user-agent group. Several equally specific groups are evaluated together; a suitable specific group does not adopt rules from the general *-group. If there is no suitable lock, crawling is allowed according to these rules.
With several suitable rules, the longest rule decides. At the same length, Allow wins. Paths distinguish capitalization and lower case. * stands for any sign and $ for the end of the URL. Query parameters belong to the matching; a fragment behind # not. The result range indicates the decisive rule with its row.
Crawling Permit Is Not an Indexing Commitment
This tool only evaluates the inserted text. It does not retrieve your real robots.txt and does not simulate HTTP status, Google cache or accessibility. A permitted URL can be noindex be excluded. A blocked URL can in turn appear as a known URL in the index, although Google is not allowed to retrieve its content.
noindex and crawl-delay in a robots.txt are not evaluated for this Google rule test. Such information shall be indicative. Use the robots.txt checker for the published file and the meta-robots checker for indexing signals of individual pages. Technical basis: Google specification for robots.txt.
Frequent questions
Will my published robots.txt be changed?
No. The rule test works with the inserted text. It does not upload a change and also does not call your website.
Why is a PDF with query parameters allowed?
A rule like Disallow: /*. pdf$ only applies if the URL including Query ends with .pdf. /file.pdf? version=2 does not meet this end condition.
Is a permitted URL automatically indexable?
No. The tester evaluates the crawling rules from robots.txt. HTTP status, noindex, canonicals, content and other requirements are not checked.