robots.txt missing: deliberately controlling crawling rules
A missing robots.txt is not always critical, but should be deliberately decided.
A missing robots.txt is not always critical, but should be deliberately decided crawling and indexing errors are especially serious if important pages are not found, understood or indexed as a result.
Consciously without locks is something other than a defective response
A small public website doesn’t necessarily need restrictions on crawlers. A missing robots.txt should therefore be assessed differently than a file that temporarily responds with server errors. First, check the actual HTTP status under the domain root.
Example: Recognizing the Result
Illustrative example: The root address /robots.txt provides 404. The website wants to allow public crawling in principle and does not require any special rules.
Limit cause and implement correction
If rules are needed, document their purpose and test typical URL examples. An empty or simple file can express a conscious decision; additional disallow lines just to eliminate a warning can cause damage. Confidential paths belong behind an access control and are not secret by a robots.txt lock.
What makes you recognize a suitable correction
First decide whether a rule file is needed at all. If you create one, it must be accessible under the correct root URL as readable text.
Classification and boundaries
When is this indication not a clear error?
A missing robots.txt is not a blanket SEO error. A 5xx response or unwanted lock file should be evaluated differently than a missing document.
robots.txt missing after CMS and cause
Many audit messages can be resolved faster if they are grouped by system, page type and technical cause. CMS indentations are only linked where robots.txt lacks a clearly own cause or fixed route in the respective system.
For this error, there is currently no own CMS deepening, because the correction is usually not sufficiently system-specific. Take advantage of the general causes and checks on this page.
Frequent causes of this error
Related SEO Errors
Appropriate Seitenreport checks
This error should not be checked in isolation. The following tools and clusters help to narrow down the cause faster and to cleanly prioritize the affected URL groups.