robots.txt blocks important URLs: check and fix
When crawling rules exclude important pages or resources.
When crawling rules exclude important pages or resources, crawling and indexing errors are especially serious if important pages are not found, understood, or indexed as a result.
Finding the Effective Rule for the Right Crawler
A disallow rule for /shop can also capture important shop pages if it has been worded too broadly. Check URL path, user agent group, and competing Allow rules together. A line in the document alone does not always explain the effective outcome.
Example: Recognizing the Result
Illustrative example: A rule for /offers/ also captures a published service page that should appear in the search.
Limit cause and implement correction
After the correction, test both a URL that is to be shared and a URL that is still to be excluded. A crawl ban does not reliably remove an already known address from the index. If a noindex is to be read, the crawler must be able to reach the corresponding answer. Access to private content must remain protected regardless.
What makes you recognize a suitable correction
Check the actually effective user-agent group and suitable path rule. Test with a permitted and a still blocked control URL.
Classification and boundaries
When is this indication not a clear error?
An intentional crawl exclusion is not automatically an error. Confidential content requires additional access control.
robots.txt blocks important URL after CMS and cause
Many audit messages can be resolved faster if they are grouped by system, page type and technical cause. CMS indentations are only linked where robots.txt blocks important URLs with a clear cause or fixed route in the respective system.
Frequent causes of this error
Related SEO Errors
Appropriate Seitenreport checks
This error should not be checked in isolation. The following tools and clusters help to narrow down the cause faster and to cleanly prioritize the affected URL groups.