robots.txt find and fix syntax errors
Faulty rules can unintentionally release or block crawling.
Faulty rules can unintentionally release crawling or block crawling and indexing errors are especially serious if important pages are not found, understood or indexed as a result.
Test rules with positive and negative examples
A spelling error in a user agent or an incorrectly assumed pattern syntax can cause a rule to not work as intended. Compare the file with the documented robots.txt syntax. Also check if an editor or forwarder is delivering an HTML page instead of plain text.
Example: Recognizing the Result
Illustrative example: A block for a directory was written to the wrong user-agent group. The intended crawler thus receives a different rule than expected.
Limit cause and implement correction
For each rule changed, set at least one URL that should remain allowed and one that should be blocked. Control the public file after deployment and consider cached versions. A successful retrieval of the file only proves accessibility, not the intended effect of its rules.
What makes you recognize a suitable correction
Read file in groups and test specific permitted and blocked URLs. Do not judge changes solely on the basis of a successfully stored text field.
Classification and boundaries
When is this indication not a clear error?
Robots rules do not use any server RegEx. Syntax and supported patterns must match the robots.txt evaluation.
robots.txt syntax error after CMS and cause
Many audit messages can be resolved faster if they are grouped by system, page type and technical cause. CMS indentations are only linked where robots.txt syntax errors have a clear cause or fixed route in the respective system.
For this error, there is currently no own CMS deepening, because the correction is usually not sufficiently system-specific. Take advantage of the general causes and checks on this page.
Frequent causes of this error
Related SEO Errors
Appropriate Seitenreport checks
This error should not be checked in isolation. The following tools and clusters help to narrow down the cause faster and to cleanly prioritize the affected URL groups.