---
title: AI robots.txt generator — block AI training, keep AI citations
description: "Free AI robots.txt generator: choose which AI crawlers can read your site. Training, search, and user-triggered bots are different — opt out of model training without disappearing from AI answers."
last_updated: 2026-07-13
canonical_url: https://agent-ready.dev/ai-robots-txt-generator
---

# AI robots.txt generator

> Free AI robots.txt generator: choose which AI crawlers can read your site. Training, search, and user-triggered bots are different — opt out of model training without disappearing from AI answers.

## What it does

"Block AI" is not one decision. Pick a policy per crawler and the generator emits a ready-to-paste robots.txt, with optional `Content-Signal:` directives. It warns you when a policy is self-defeating — most commonly, blocking the crawler that would have cited you while leaving the one that trains on you switched on. Generation runs in your browser; the only network call is the optional merge with your site's existing robots.txt.

## Why blocking "AI bots" usually backfires

The names give you no hint which bot does what. OpenAI's `GPTBot` is the **training** crawler; `OAI-SearchBot` is the one that puts you in ChatGPT's **cited** answers. Block the wrong one and you get the worst of both outcomes: no referral traffic, and your content still in the training corpus via a third-party copy you don't control.

The three categories:

- **Training** — collects text for offline model training. Your page shapes the model's weights but is never linked back to. Blocking costs you **no visibility**.
- **Search** — builds the retrieval index an assistant queries at answer time and **cites with a link**. Blocking removes you from AI answers; it is the same decision as removing yourself from a search engine.
- **User-triggered** — fetches a page live because a human asked the assistant about that specific URL. Blocking means an assistant can't read a link your reader pasted.

## What each AI crawler actually does

### Search crawlers — blocking these costs you AI citations

| User-agent | Operator | If you block it |
|---|---|---|
| [`OAI-SearchBot`](https://developers.openai.com/api/docs/bots) | OpenAI | You will not appear in ChatGPT search answers. OpenAI is explicit about this — it's the single most costly block on this page. |
| [`Claude-SearchBot`](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) | Anthropic | You lose citations in Claude's web-search results. |
| [`PerplexityBot`](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) | Perplexity | Your site won't appear in Perplexity's search results. _Cloudflare reported in August 2025 that Perplexity used undeclared stealth crawlers to reach content that had blocked its declared bots, and de-listed it as a verified bot. robots.txt is not a reliable control here._ |
| [`Meta-WebIndexer`](https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers) | Meta | You lose visibility in Meta AI's search results. |
| [`Amzn-SearchBot`](https://developer.amazon.com/amazonbot) | Amazon | You lose eligibility in Alexa and Amazon's search surfaces. Amazon states it is not used for generative-AI training. |
| [`MistralAI-Index`](https://docs.mistral.ai/robots) | Mistral | You lose citations in Mistral's assistant. Mistral states this index is not used for training. |
| [`DuckAssistBot`](https://duckduckgo.com/duckduckgo-help-pages/results/duckassistbot) | DuckDuckGo | You're excluded as a source for DuckAssist AI answers. DuckDuckGo confirms this does not affect organic search rankings. |
| [`YouBot`](https://docs.you.com/youbot) | You.com | You lose visibility in You.com's search and assistant. |

### User-triggered fetchers — blocking these stops assistants reading a link your reader pasted

| User-agent | Operator | If you block it |
|---|---|---|
| [`ChatGPT-User`](https://developers.openai.com/api/docs/bots) | OpenAI | ChatGPT can't open a link one of your readers pasted or clicked. _OpenAI notes that because these fetches are user-initiated, robots.txt rules may not apply to them._ |
| [`Claude-User`](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) | Anthropic | Claude can't read a link one of your readers pasted. _Unlike OpenAI's and Perplexity's user fetchers, Anthropic states this one does honour robots.txt._ |
| [`Perplexity-User`](https://docs.perplexity.ai/docs/resources/perplexity-crawlers) | Perplexity | Perplexity can't fetch a page a user asked it about. _Perplexity's own docs say this fetcher generally ignores robots.txt because a user requested it._ |
| [`Meta-ExternalFetcher`](https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers) | Meta | Meta AI can't fetch a link a user asked it about. _Meta's docs note this fetcher may bypass robots.txt._ |
| [`MistralAI-User`](https://docs.mistral.ai/robots) | Mistral | Mistral's assistant can't fetch a page a user asked about. |
| [`Amzn-User`](https://developer.amazon.com/amazonbot) | Amazon | Alexa can't fetch live information for a user's question. |

### Training crawlers — blocking these costs you nothing in visibility

| User-agent | Operator | If you block it |
|---|---|---|
| [`GPTBot`](https://developers.openai.com/api/docs/bots) | OpenAI | Your content is no longer used to train OpenAI's models. This does not affect ChatGPT citations — that's OAI-SearchBot. |
| [`ClaudeBot`](https://support.claude.com/en/articles/8896518-does-anthropic-crawl-data-from-the-web-and-how-can-site-owners-block-the-crawler) | Anthropic | Your content is no longer collected to train Claude. Claude can still cite you via Claude-SearchBot. |
| [`Google-Extended`](https://developers.google.com/search/docs/crawling-indexing/google-common-crawlers) | Google | Your content is not used to train or ground Gemini. It has no effect on Google Search ranking — and none on AI Overviews either. _Not a crawler: Google fetches with Googlebot and reads this token only as a usage control._ |
| [`Applebot-Extended`](https://support.apple.com/en-us/119829) | Apple | Apple won't use content it already crawled to train its foundation models. You stay in Siri and Spotlight search. _Not a crawler: Apple fetches with Applebot and reads this token only as a usage control._ |
| [`CCBot`](https://commoncrawl.org/ccbot) | Common Crawl | You're excluded from the Common Crawl corpus — an upstream training source for many labs, so this one block propagates widely. |
| [`meta-externalagent`](https://developers.facebook.com/documentation/sharing/webmasters/web-crawlers) | Meta | Your content is not used to train Meta's AI models. |
| [`Amazonbot`](https://developer.amazon.com/amazonbot) | Amazon | Amazon says content it crawls may be used to train its AI models; blocking opts you out of that. |
| [`AI2Bot`](https://allenai.org/crawler) | Allen Institute for AI | You're excluded from Ai2's open datasets (Dolma), which train open-source models. |
| [`Diffbot`](https://docs.diffbot.com/docs/does-crawl-respect-robotstxt) | Diffbot | You're excluded from Diffbot's Knowledge Graph, which is resold into training pipelines. No citation cost — it isn't an assistant. |
| [`Bytespider`](https://darkvisitors.com/agents/bytespider) | ByteDance | Intended to exclude you from ByteDance/Doubao training data. _ByteDance publishes no crawler docs and Bytespider is widely reported to crawl Disallow-ed URLs anyway. Treat this as best-effort and enforce at your WAF if it matters._ |

A **control token** (`Google-Extended`, `Applebot-Extended`) is not a crawler at all — it never fetches anything. It exists only in robots.txt to govern how content the operator already fetched with a different user-agent may be used.

## Why Googlebot, Bingbot, and Applebot are not listed

They are general search crawlers, not AI crawlers, and a one-click block next to them would be dangerous. Blocking Googlebot removes you from Google Search altogether — and, as a side-effect, from AI Overviews, which are built on the same index. This generator covers AI-specific crawlers plus the AI-specific control tokens (`Google-Extended`, `Applebot-Extended`), which are the knobs that let you opt out of AI training without touching search visibility.

## Is robots.txt enough to stop AI training?

No — it is a stated preference, not a lock. The major operators honour it, but some do not, and robots.txt cannot stop your content reaching a model through a scraper, a mirror, or a public dataset someone else compiled. If you need enforcement, that belongs at your CDN or WAF.

## Content-Signal

Per-crawler rules go stale the moment a new bot ships. Content Signals (<https://contentsignals.org/>), the convention feeding the IETF aipref working group, instead declares how your content may be *used* — `search`, `ai-input`, `ai-train` — so it covers crawlers that don't exist yet. It is young and not universally honoured, so the generator emits it alongside per-crawler rules rather than instead of them.

## Frequently asked questions

### Does blocking GPTBot remove my site from ChatGPT?

Not from ChatGPT's answers — only from OpenAI's training data. GPTBot collects text to train models. A different crawler, OAI-SearchBot, builds the index ChatGPT search cites at answer time, and a third, ChatGPT-User, fetches a page live when someone asks ChatGPT about that specific URL. They are separate tokens in robots.txt. If you block all three you vanish from ChatGPT entirely; if you block only GPTBot you opt out of training while staying citable.

### Does Google-Extended opt me out of AI Overviews?

No — and this is the most misunderstood token in robots.txt. Google-Extended controls whether your content trains and grounds Gemini. AI Overviews and AI Mode are built from the ordinary Googlebot search index, so Google-Extended has no effect on them. Historically the only robots.txt lever was blocking Googlebot, which also removes you from Google Search entirely. In June 2026 Google added a Search generative AI control in Search Console that opts a site out of AI Overviews and AI Mode without affecting ranking (UK-first, still a limited rollout). That is a Search Console setting, not a robots.txt directive, so no generator — including this one — can emit it for you.

### What's the difference between a training crawler and a search crawler?

A training crawler collects text to build an offline corpus that a model is trained on; your page contributes to the model's weights but is never linked back to. A search crawler builds a retrieval index the assistant queries at answer time and cites with a link. Blocking a training crawler costs you nothing in visibility. Blocking a search crawler removes you from AI answers — it is the same decision as removing yourself from a search engine.

### Does robots.txt actually stop AI crawlers?

It is a request, not an enforcement mechanism. Crawlers from OpenAI, Anthropic, Google, and Apple honour it. Some do not: Perplexity's and OpenAI's user-triggered fetchers state in their own docs that robots.txt may not apply to them because a human asked for the page, Bytespider is widely reported to crawl Disallow-ed URLs, and Cloudflare reported in August 2025 that Perplexity used undeclared stealth crawlers to reach content that had blocked its declared bots. robots.txt also cannot stop your content reaching a model through a scraper, a mirror, or a public dataset someone else compiled. If you need enforcement rather than a stated preference, block at your CDN or WAF — robots.txt is the polite front door, not a lock.

### Why isn't Googlebot in the list?

Because it is not an AI crawler, and putting a one-click block next to it would be dangerous. Googlebot powers Google Search; blocking it removes you from Google altogether (and, as a side-effect, from AI Overviews, which are built on that same index). The same is true of Bingbot and Applebot. This generator covers AI-specific crawlers and the AI-specific control tokens — Google-Extended and Applebot-Extended — which are the knobs that actually let you opt out of AI training without touching your search visibility.

### What is the Content-Signal directive?

Content-Signal is a robots.txt extension (from contentsignals.org, feeding the IETF aipref work) that states how your content may be used — search=yes|no, ai-input=yes|no, ai-train=yes|no — rather than just which bot may fetch it. It expresses intent independently of crawler names, so it keeps working as new bots appear. It is a young convention and not yet universally honoured, so it complements per-crawler rules rather than replacing them.

### Where do I put the generated file?

Save it as robots.txt at the root of your domain, so it is served at https://yourdomain.com/robots.txt as plain text. It must be at the root — a robots.txt in a subdirectory is ignored. If you already have one, use the merge option on this page so your existing rules, comments, and Sitemap lines are preserved and only the AI-crawler groups are replaced.

---

Generate your AI robots.txt on the web: <https://agent-ready.dev/ai-robots-txt-generator>

## Sitemap

See the full [sitemap](https://agent-ready.dev/sitemap.md) for all pages on agent-ready.dev.