AI robots.txt generator.
Fifteen named AI crawlers are reading the web right now — some send you customers, some just take. Pick a policy, flip individual bots, and get paste-ready robots.txt rules for GPTBot, ClaudeBot, Google-Extended, PerplexityBot and the rest. No signup, nothing stored.
Allow assistants and AI search, block bulk scrapers. The default we recommend for most businesses.
Feed foundation-model training. Blocking removes future-model exposure, not search.
- GPTBotOpenAI
- ClaudeBotAnthropic
- Google-ExtendedGoogle
- Applebot-ExtendedApple
- Meta-ExternalAgentMeta
Put you in AI search results with citations. Blocking these costs you referrals.
- OAI-SearchBotOpenAI
- Claude-SearchBotAnthropic
- PerplexityBotPerplexity
Fetch a page because a human asked about it, like a browser does.
- ChatGPT-UserOpenAI
- Claude-UserAnthropic
- Perplexity-UserPerplexity
- Meta-ExternalFetcherMeta
General-purpose or dataset scrapers with no direct referral value to you.
- CCBotCommon Crawl
- AmazonbotAmazon
- BytespiderByteDance
# AI crawler rules — generated with techpotions.com/tools/ai-robots-txt-generator # Merge into your existing robots.txt; rules for other crawlers are unaffected. # AI training crawlers User-agent: GPTBot Allow: / User-agent: ClaudeBot Allow: / User-agent: Google-Extended Allow: / User-agent: Applebot-Extended Allow: / User-agent: Meta-ExternalAgent Allow: / # AI search indexers User-agent: OAI-SearchBot Allow: / User-agent: Claude-SearchBot Allow: / User-agent: PerplexityBot Allow: / # User-triggered fetchers User-agent: ChatGPT-User Allow: / User-agent: Claude-User Allow: / # Note: Perplexity documents that user-triggered fetches generally ignore robots.txt. User-agent: Perplexity-User Allow: / # Note: Meta documents that this fetcher may bypass robots.txt rules. User-agent: Meta-ExternalFetcher Allow: / # Bulk / dataset crawlers User-agent: CCBot Disallow: / User-agent: Amazonbot Disallow: / # Note: Widely reported to ignore robots.txt; blocking it here is a signal, not a wall. User-agent: Bytespider Disallow: /
Paste into the robots.txt at your site root, e.g. https://yoursite.com/robots.txt. Keep your existing rules; these blocks only address the named AI agents.
Every AI crawler that matters in 2026.
This is the complete AI crawler list this generator manages, grouped by what each agent actually does. Every description below comes from the operator’s own documentation, linked at the end of this section.
| User agent | Operator | What it does |
|---|---|---|
| GPTBot | OpenAI | Crawls the open web for OpenAI foundation-model training. |
| ClaudeBot | Anthropic | Crawls the web for Anthropic model training. |
| Google-Extended | Robots token controlling Gemini training use of your content. Does not affect Google Search crawling or ranking. | |
| Applebot-Extended | Apple | Opt-out token for Apple foundation-model training. Regular Applebot search crawling is unaffected. |
| Meta-ExternalAgent | Meta | Crawls for training Meta AI models and indexing content into Meta products. |
| OAI-SearchBot | OpenAI | Indexes sites so they can appear and be linked in ChatGPT search answers. |
| Claude-SearchBot | Anthropic | Indexes sites to improve search citations inside Claude. |
| PerplexityBot | Perplexity | Surfaces and links sites in Perplexity results. Not used for foundation-model training. |
| ChatGPT-User | OpenAI | Fetches a page live when a ChatGPT user asks about it. |
| Claude-User | Anthropic | Fetches a page live when a Claude user asks about it. |
| Perplexity-User | Perplexity | Visits pages on behalf of a Perplexity user asking a question.Perplexity documents that user-triggered fetches generally ignore robots.txt. |
| Meta-ExternalFetcher | Meta | Fetches individual links at a user’s request for Meta AI features.Meta documents that this fetcher may bypass robots.txt rules. |
| CCBot | Common Crawl | Builds the open Common Crawl dataset, a corpus many AI labs train on. |
| Amazonbot | Amazon | Amazon’s general web crawler, used to improve Amazon products and services. |
| Bytespider | ByteDance | ByteDance’s crawler. No official documentation.Widely reported to ignore robots.txt; blocking it here is a signal, not a wall. |
Grouped another way: 5 training crawlers, 3 AI search indexers, 4 user-triggered fetchers, and 3 bulk crawlers. Put you in AI search results with citations. Blocking these costs you referrals. Feed foundation-model training. Blocking removes future-model exposure, not search.
Sources: OpenAI — overview of OpenAI crawlers · Anthropic — does Anthropic crawl data from the web · Google — Google common crawlers (Google-Extended) · Apple — about Applebot (Applebot-Extended) · Perplexity — Perplexity crawlers · Meta — Meta web crawlers · Common Crawl — CCBot · Amazon — Amazonbot
Should you block AI crawlers?
Blocking everything feels safe and costs you quietly. AI assistants now answer a meaningful share of buying questions — “best agency for X”, “what does Y cost” — and they answer from the sites they can read. Block the search and assistant agents and you are invisible in that channel; your competitors who stayed readable become the answer instead.
Blocking nothing is not obviously right either. Training crawlers and dataset scrapers take your content to build products that never send a visitor back. If your content is your moat — original research, paid courses, proprietary data — opting out of training while staying in AI search is a coherent position, and it is exactly what the “Protect content” preset above produces.
One honest caveat: robots.txt is a request, not a firewall. The major labs document compliance, but user-triggered fetchers often bypass it by design, and Bytespider is widely reported to ignore it. If you need enforcement rather than a signal, do it at the CDN with bot management rules.
gptbot robots.txt, the surgical version.
The most common real-world policy is not “block OpenAI” — it is “stay out of training, stay in the answers.” That takes six lines:
# Block OpenAI model training only User-agent: GPTBot Disallow: / # Keep ChatGPT search citations + live visits working User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: /
In a Next.js app, these rules belong in your app/robots.ts route rather than a static file, so they deploy with the site. And if you want AI systems to read you well rather than just read you, pair your robots.txt with an llms.txt — our free llms.txt generator builds one in the same paste-ready way.
Common questions.
01How do I block ChatGPT from crawling my site?
Add two lines to your robots.txt: "User-agent: GPTBot" followed by "Disallow: /". That stops OpenAI’s training crawler. Note that OpenAI runs three separate agents: GPTBot (training), OAI-SearchBot (ChatGPT search results) and ChatGPT-User (live fetches when a user asks about your site). Blocking one does not block the others, which is exactly why this generator lists them separately.
02Does blocking GPTBot remove my site from ChatGPT?
No. Blocking GPTBot only keeps your content out of future model training. ChatGPT can still cite you in search answers via OAI-SearchBot and still open your pages live via ChatGPT-User. If you want to disappear from ChatGPT entirely you would need to block all three OpenAI agents — and most businesses should not, because those citations are free referrals.
03What is Google-Extended and does blocking it hurt my SEO?
Google-Extended is not a crawler you will see in logs — it is a robots.txt token that tells Google not to use your content for Gemini model training and grounding. Google documents that it has no effect on Google Search crawling, indexing or ranking. Blocking it is one of the few genuinely free protective moves: you keep your search traffic and opt out of AI training.
04Do AI crawlers actually respect robots.txt?
The documented crawlers from OpenAI, Anthropic, Google and Apple state they honor robots.txt, and their published behavior generally matches. The honest exceptions: Perplexity documents that its user-triggered fetcher generally ignores robots.txt, Meta documents that Meta-ExternalFetcher may bypass it, and ByteDance’s Bytespider is widely reported to ignore it entirely. robots.txt is a protocol built on good faith — for hard enforcement you need a WAF or bot management on your CDN.
05What is the full list of AI crawlers in 2026?
The agents that matter: GPTBot, OAI-SearchBot and ChatGPT-User (OpenAI); ClaudeBot, Claude-SearchBot and Claude-User (Anthropic); Google-Extended (Google); Applebot-Extended (Apple); PerplexityBot and Perplexity-User (Perplexity); Meta-ExternalAgent and Meta-ExternalFetcher (Meta); CCBot (Common Crawl); Amazonbot (Amazon); and Bytespider (ByteDance). The table on this page describes what each one does and links every official source.
06Should my business block AI crawlers?
For most businesses, no — at least not the search and assistant agents. AI assistants increasingly answer buying questions, and a site they cannot read is a site they cannot recommend. The pragmatic middle ground, and this tool’s default: allow the agents that cite you and send traffic, block the bulk scrapers that offer nothing back, and decide on training crawlers based on how much your content is your moat.
- llms.txt generatorGenerate a spec-correct llms.txt file for your site, so AI assistants read you right.
- App development cost calculatorScope a web or mobile app and get an honest 2026 cost and timeline range.
- AI receptionist cost calculatorCompare an AI receptionist against in-house staff and answering services, in dollars.
- LLM API pricing calculatorCompare Claude, GPT, Gemini and DeepSeek token costs on your real usage.
Want to be the site AI recommends, not just the one it reads?
Crawler policy is the defensive half. We build the offensive half: sites AI assistants can parse, cite and recommend — schema, GEO, llms.txt and the engineering underneath. techpotions is a small studio; you talk to the people who build.