AI Bot Blocking Guide
Control which AI bots can access your content. Allow search bots for visibility, block training bots if you don't want your content used for model training.
The Key Distinction
Not all AI bots are the same. Training bots scrape your content to build AI models — blocking these is about intellectual property. Search/retrieval bots index your content for AI-powered search — blocking these makes you invisible in AI answers. The recommended strategy: allow retrieval, block training.
Interactive robots.txt Builder
Toggle each bot to allow or block it. Your custom configuration will be generated below.
Your Configuration
Add this to your existing robots.txt file. These directives work alongside your existing rules.
Important Notes
robots.txt is an honor system
Major AI companies generally respect robots.txt, but it is not enforceable. Malicious scrapers may ignore it entirely.
Blocking training ≠ blocking search
Blocking GPTBot does NOT affect your Google search rankings. It only prevents OpenAI from using your content for training.
User agents can be spoofed
For stronger protection, verify bot identity using published IP ranges. Consider WAF rules for high-value content.
New bots appear regularly
The AI landscape evolves fast. Check back periodically for new user agents to manage.