AI-generated illustration
Lock out web crawlers that ignore the robots.txtUpdated for 2026: Protection with Rate Limits, WAF and Clean Access Control
robots.txt formulates rules for cooperating crawlers. If a bot ignores these rules, you need a controlled access restriction at the server, CDN or application level – with a check of the side effects.
First occupy accesses, then specifically limit them. User-agent names are easy to imitate; a blanket bot lock can hit desired search engines and test tools.
What robots.txt can do and cannot do
A robots.txt file is not access control. It can tell cooperating crawlers which paths they should not retrieve. Confidential content requires an actual access restriction, such as appropriate authentication.
An indexing decision must also be distinguished from this: If a crawler is not allowed to retrieve a page, it cannot read a noindex statement stored there. Use access protection, crawl control, and indexing rules each for their actual task.
Prove the accesses first
| Log observations | Question to be clarified |
|---|---|
| Many retrievals of the same expensive search | Does the same client repeat identical requests? |
| High number from an IP address | Is this a bot, proxy or shared legitimate access? |
| Well-known bot name in the user agent | Can the origin be confirmed according to the provider’s procedure? |
| Retrieval despite robots.txt | Was the correct file delivered successfully and could a new rule already be adopted? |
Introducing a limited safeguard measure
- Record time period, paths, status codes, request rates and noticeable effects from the logs.
- Desired services are distinguished from mere names based on their documented identity verification.
- First, choose tightly limited rules for affected paths or conspicuous behavior.
- In the case of a rate limitation, align thresholds and time period to real legitimate accesses.
- After introduction, check search engine retrievals, customer channels and server load; enable a take-back.
Do not adopt a blanket block list
IP areas and infrastructures are changing. An old list can meet uninvolved users or no longer reach the actual bot. Even a fake Googlebot name must not automatically lead to an exception.
Test rules first in a suitable environment or observation phase. Unnecessary HTTP 403 or 429 status on important pages can significantly affect accessibility.
Frequent questions
Can I recognize a bot safely by name alone?
No. The user agent is a self-disclosure. For important providers, use their documented verification procedures.
Does robots.txt protect private files?
No. Private content requires an actual access restriction.