Robots.txt for AI crawlers: allow discovery without exposing private paths.
A practical robots.txt policy allows documented search and citation crawlers to access public pages you want discovered, while disallowing private, authenticated or budget-consuming routes. Review provider names and controls in current official documentation, because search retrieval and model-training controls may differ.
Researched and reviewed by Eli · 9 August 2026
The question this page answers
Configure robots.txt for AI search crawlers
Separate public content from application surfaces
List the public product, resource, documentation and evidence paths that should be crawlable. Keep dashboards, authentication, account data, internal APIs and on-demand scans out of crawl queues. Robots rules are public guidance, not an authorization layer, so sensitive content still needs real access control.
Use documented crawler controls
OpenAI and Perplexity publish specific crawler information. Check current names, purposes and verification guidance before deploying a rule copied from an old template. Avoid a blanket block added by infrastructure defaults when the business expects AI search discovery.
- Public allow paths
- Private disallow paths
- Current user agents
- Sitemap location
Test the live response
Fetch `/robots.txt` from production, confirm the content type and inspect server logs after deployment. Test representative public and private URLs. A correct robots file does not guarantee crawling or citation, but an accidental disallow can make the intended page unavailable to a compliant crawler.
Questions buyers ask next
Does robots.txt secure private data?
No. It is crawler guidance. Use authentication and authorization to protect private content.
Should every AI crawler be allowed?
Choose based on documented purpose, security policy and whether you want the related search or citation eligibility.
Can Cloudflare or another platform change bot behavior?
Yes. Review edge bot controls and managed defaults together with the origin robots file.
Primary sources
Find the buyer question you are losing
Run the free scan to see the observed competitors and evidence behind your first AI search gap. No ranking is promised or invented.
Run your free scan