Discovery · Local browser tool
Robots.txt and AI Crawler Analyzer
Inspect pasted robots.txt rules for search and AI crawler access, keeping search and training controls distinct.
Use the tool
Enter only data you are authorised to inspect. The result is a focused technical check, not a certification or full vulnerability assessment.
What it checks
Clear scope. Reproducible output.
- Wildcard and named crawler groups
- OAI-SearchBot, GPTBot and ChatGPT-User
- ClaudeBot, PerplexityBot and Google-Extended
Questions
Does Robots.txt and AI Crawler Analyzer upload my input?
No. The working tool runs in the current browser tab. Avoid browser extensions or shared devices when handling sensitive material.
Does a passing result prove my system is secure?
No. It proves only the checks described on this page against the input you supplied. Server behaviour, business logic and runtime vulnerabilities require separate authorised testing.
Why is the result different from a live scanner?
This tool analyses pasted or local data. A live scanner observes network responses, DNS, TLS and redirects, so its evidence and scope are different.