AI robots.txt generator

Why blocking “AI bots” usually backfires

The most common AI robots.txt mistake is blocking the crawler that would have cited you while leaving the one that trainson you switched on. They are different user-agents with different jobs, and the names give you no hint which is which — OpenAI’s GPTBot is the training crawler, while OAI-SearchBot is the one that puts you in ChatGPT’s cited answers. Block the wrong one and you get the worst of both: no traffic, and your content still in the corpus via someone else’s copy.

What each AI crawler actually does

Every token below is a real robots.txt user-agent, grouped by what blocking it costs you, and linked to the operator’s own documentation. A control tokenis not a crawler at all — it is a name that only exists in robots.txt to govern how content the operator already fetched may be used. Tokens with no primary source (and dead strings like anthropic-ai, which Anthropic no longer documents) are deliberately left out — a token we can’t verify would tell you you’re protected when you may not be.

Is robots.txt enough to stop AI training?

No — it is a stated preference, not a lock. The major operators honour it, but robots.txt cannot stop your content reaching a model through a scraper, a third-party mirror, or a public dataset someone else compiled. Treat it as the polite front door: it handles the well-behaved majority and puts your policy on the record. If you need enforcement, that belongs at your CDN or WAF.

What about Content-Signal?

Per-crawler rules go stale the moment a new bot ships. Content Signals — the convention feeding the IETF aipref working group — instead declares how your content may be used (search, ai-input, ai-train), so it applies to crawlers that don’t exist yet. It’s young and not universally honoured, so the generator emits it alongside per-crawler rules rather than instead of them.

After you publish it

A permissive robots.txt only gets an agent to your door — it still has to be able to read what’s inside. Once the file is live, run a full agent-readability scan to confirm the crawlers you allowed can actually parse your content, and generate an llms.txt to give the agents you welcomed a curated map of your site.

Frequently asked questions

Does blocking GPTBot remove my site from ChatGPT?

Not from ChatGPT's answers — only from OpenAI's training data. GPTBot collects text to train models. A different crawler, OAI-SearchBot, builds the index ChatGPT search cites at answer time, and a third, ChatGPT-User, fetches a page live when someone asks ChatGPT about that specific URL. They are separate tokens in robots.txt. If you block all three you vanish from ChatGPT entirely; if you block only GPTBot you opt out of training while staying citable.

Does Google-Extended opt me out of AI Overviews?

No — and this is the most misunderstood token in robots.txt. Google-Extended controls whether your content trains and grounds Gemini. AI Overviews and AI Mode are built from the ordinary Googlebot search index, so Google-Extended has no effect on them. Historically the only robots.txt lever was blocking Googlebot, which also removes you from Google Search entirely. In June 2026 Google added a Search generative AI control in Search Console that opts a site out of AI Overviews and AI Mode without affecting ranking (UK-first, still a limited rollout). That is a Search Console setting, not a robots.txt directive, so no generator — including this one — can emit it for you.

What's the difference between a training crawler and a search crawler?

A training crawler collects text to build an offline corpus that a model is trained on; your page contributes to the model's weights but is never linked back to. A search crawler builds a retrieval index the assistant queries at answer time and cites with a link. Blocking a training crawler costs you nothing in visibility. Blocking a search crawler removes you from AI answers — it is the same decision as removing yourself from a search engine.

Does robots.txt actually stop AI crawlers?

It is a request, not an enforcement mechanism. Crawlers from OpenAI, Anthropic, Google, and Apple honour it. Some do not: Perplexity's and OpenAI's user-triggered fetchers state in their own docs that robots.txt may not apply to them because a human asked for the page, Bytespider is widely reported to crawl Disallow-ed URLs, and Cloudflare reported in August 2025 that Perplexity used undeclared stealth crawlers to reach content that had blocked its declared bots. robots.txt also cannot stop your content reaching a model through a scraper, a mirror, or a public dataset someone else compiled. If you need enforcement rather than a stated preference, block at your CDN or WAF — robots.txt is the polite front door, not a lock.

Why isn't Googlebot in the list?

Because it is not an AI crawler, and putting a one-click block next to it would be dangerous. Googlebot powers Google Search; blocking it removes you from Google altogether (and, as a side-effect, from AI Overviews, which are built on that same index). The same is true of Bingbot and Applebot. This generator covers AI-specific crawlers and the AI-specific control tokens — Google-Extended and Applebot-Extended — which are the knobs that actually let you opt out of AI training without touching your search visibility.

What is the Content-Signal directive?

Content-Signal is a robots.txt extension (from contentsignals.org, feeding the IETF aipref work) that states how your content may be used — search=yes|no, ai-input=yes|no, ai-train=yes|no — rather than just which bot may fetch it. It expresses intent independently of crawler names, so it keeps working as new bots appear. It is a young convention and not yet universally honoured, so it complements per-crawler rules rather than replacing them.

Where do I put the generated file?

Save it as robots.txt at the root of your domain, so it is served at https://yourdomain.com/robots.txt as plain text. It must be at the root — a robots.txt in a subdirectory is ignored. If you already have one, use the merge option on this page so your existing rules, comments, and Sitemap lines are preserved and only the AI-crawler groups are replaced.