Robots.txt Generator: Creating Rules and Sitemap Notices
Free generator · without registration
Create a robots.txt from a template and your own path rules. Add sitemap URLs, check the preview and copy or download the finished text file.
- Create basic rules for new websites or relaunches
- Cleanly lock out internal areas such as login, search or admin
- Deposit sitemaps directly in the robots.txt
- Quickly prepare WordPress, TYPO3 or shop base
- copy or download finished robots.txt immediately
# Hier erscheint die generierte robots.txt
- Only really unwanted areas are blocked.
- Do not accidentally block the entire website
- Validate with the checker after the upload
| # | Type | Value |
|---|
Create, review and publish robots.txt
Combine template and own rules
Choose Base, WordPress, TYPO3, Shop or the stricter template. The generator adds your input to the rules of the selected template; an empty input field does not remove their default rules. Therefore, check the full edition. Especially with individual CMS paths or shops, blanket rules can block important pages or required resources.
Differentiate Allow, Disallow and Sitemap
Disallow describes paths that a cooperating crawler should not retrieve. Allow can release certain paths. A sitemap line refers to an XML sitemap and is not a release for blocked pages. For rules, enter the actual paths of your website and control sitemap URLs for accessibility. CSS, JavaScript and images necessary to understand your pages should not be blocked accidentally.
Preventing Crawling Is Not the Same as Noindex
A URL blocked via robots.txt can still appear in search results. If an accessible page is to disappear from the index, a crawler must be able to read its noindex statement. Simultaneous disallow can prevent this. For confidential content, a registration or other access control is necessary; the robots.txt is public and not password protection.
Deploy file in main directory
Upload the verified file to /robots.txt of the respective host. Then check the delivered version and test important permitted and blocked URLs. A CMS plugin or virtual route can deliver your own robots.txt instead of your uploaded file. For multiple subdomains, you control each affected host separately.
What the additional bot lock does
The option complements named user-agent groups with Disallow: /. It does not block IP addresses and does not guarantee that every service complies with your specifications. Whether a specific provider takes a rule into account and what its crawler is used for, you have to check against its current documentation.
Exclude internal search results from crawling
Practical example · for illustration
Starting position
Example rule: User-agent: * followed by Disallow: /search/. No block is provided for /products/ in this example.
Classify the findings
Disallow concerns the retrieval of the path. Already known URLs can still remain in the index without read content.
Specific amendment
Matching rules to the path actually used. Protect confidential areas with login, not just with robots.txt.
Control and border
With the robots.txt tester /such/? Simulate q=test and /products/ separately; then retrieve the file at the domain root.
Google: Purpose and Limitations of robots.txt
Questions about use and boundaries
Is the Robots.txt Generator free?
Yes. Templates, preview, copy and download are available without registration. The tool creates a file; it does not automatically change your website.
Are the WordPress and TYPO3 templates immediately suitable?
They are starting points. Check actual paths, plugins, assets, and desired crawl destinations. Own rules are merged with the template.
Where do I upload robots.txt?
Under /robots.txt directly on the respective host, such as https://example.org/robots.txt., then check in the browser which file is actually delivered there.
Can I specify multiple XML sitemaps?
Yes. Enter a sitemap address per line. Check the generated URLs and whether the files are publicly accessible.
Does Disallow prevent indexing?
Not reliable. Disallow controls the call. For noindex, the crawler must be able to read the corresponding instruction; a simultaneous crawl lock can prevent this.