Pick a preset (or customize) → generate a robots.txt snippet you can paste into your existing robots.txt.
Tip: “Recommended” keeps AI visibility while opting out of common training/usage tokens.
If you already have sitemap lines in robots.txt, leave this blank.
One path per line. Always start with /. Example: /private/
Non-standard. Some bots support it (e.g., Anthropic mentions support).
Important: robots.txt relies on crawler compliance; it’s not a security feature. For truly private content, use authentication or server-side blocking.
User-initiated fetchers may ignore robots.txt depending on provider policy.
Search indexing for Claude web search (recommended: Allow).
Signals training opt-out (recommended: Disallow).
Blocking may reduce user-directed retrieval visibility.
Surfaces links in Perplexity search (recommended: Allow).
Docs note user-requested fetchers may ignore robots.txt.
Doesn’t impact Google Search inclusion/rankings (token control).
Doesn’t affect Apple search results (token control).
robots.txt snippet (paste into your existing robots.txt)
Where to paste: robots.txt lives at https://visualhull.com/robots.txt. If you already have rules, paste this snippet below them.
Want us to implement + test? We can configure robots.txt, validate in Search Console, and verify bot activity in server logs / WAF.
Contact Visual Hull.
Need Help?
We’ll Implement AEO + GEO For You
Use the free tools — or hire us to implement strategy, pages, and tracking properly.
AEO
Answer Engine Optimization
Direct answers, FAQs, internal linking, and structured data that improves extractability.
