Skip to content

AI Bot User-Agent Enumeration in robots.txt: The Complete 2026 List

| 6 min read
aao ai-search robots-txt ai-bots technical-seo discovery-gate content-optimization b2b-finance
A reference card on a navy field listing eleven AI bot user agents. Each entry is a gold-bordered row with the bot name in monospace, an icon representing whether it is training-class or citation-class, and a small note about its parent engine. The visual is purpose-built as a quick-reference table for robots.txt configuration.

Eleven Bots, Real Consequences

Eleven AI bot user agents matter for a B2B finance site in mid-2026. The list is stable enough to publish; it shifts perhaps twice a year as engines launch new crawlers or rename existing ones. Each user agent has a specific role (training data extraction, live retrieval, brand monitoring) and warrants its own allow versus disallow decision.

The default for most YMYL sites is to allow all eleven. The asymmetry favors visibility: blocking a citation-class bot to protect a small competitive advantage costs more in AI search citation than it gains in proprietary content protection. There are exceptions; this post covers them.

This is the complete reference. See the Discovery Gate post for why enumeration matters and the llms.txt patterns post for how robots.txt composes with the broader Discovery surface.

The Eleven User Agents

The 2026 list, with parent engine and role per bot:

  1. GPTBot (OpenAI, training-class). Used by OpenAI for training data extraction. Honors robots.txt explicitly. Blocking does not prevent training on already-collected data; it prevents new extraction.

  2. OAI-SearchBot (OpenAI, citation-class). Used by ChatGPT search and SearchGPT for live retrieval. Citations on user queries flow through this bot. Blocking it removes the site from ChatGPT search citation eligibility.

  3. ChatGPT-User (OpenAI, citation-class). Used when a ChatGPT user explicitly browses a URL the user has typed or pasted. Direct user-initiated retrieval rather than discovery.

  4. ClaudeBot (Anthropic, training-class). Used by Anthropic for training data extraction. Honors robots.txt.

  5. Claude-Web (Anthropic, citation-class). Used by Claude with web access for live retrieval per user query.

  6. PerplexityBot (Perplexity, citation-class). Used by Perplexity for live retrieval. Perplexity’s citation behavior is heavily dependent on this bot’s access.

  7. Perplexity-User (Perplexity, citation-class). Used when a Perplexity user explicitly browses a URL. Direct user-initiated retrieval.

  8. Google-Extended (Google, training-class). Used by Google for training on content for Gemini and other AI products. Does not affect Google Search ranking or indexing. Blocking is a clean way to opt out of Google AI training without affecting SEO.

  9. Applebot-Extended (Apple, training-class). Used by Apple for AI training, separate from Applebot which handles standard search. Same opt-out logic as Google-Extended.

  10. Amazonbot (Amazon, mixed). Used for Amazon’s broader crawling including for Rufus and Alexa AI. Honors robots.txt.

  11. CCBot (Common Crawl, training-class). Common Crawl provides web archives that many AI models train on. Blocking reduces the training surface available to a wide set of downstream engines.

The Allow Versus Disallow Decision Tree

Run each user agent through three questions:

  1. Is the bot citation-class or training-class? Citation-class bots directly affect AI search visibility per user query. Training-class bots affect what models learn from your content over time but do not directly cite per query.

  2. Does my content have competitive value that training-class extraction would erode? Proprietary research with limited public exposure has more to lose from training-class extraction than commodity content does. Most B2B finance blog content is not proprietary in a way that training-class extraction harms.

  3. Do I have a specific reason to block this engine versus all engines? Blocking only OpenAI but allowing Anthropic, for example, is a coherent stance if the firm has a competitive concern with OpenAI specifically. Blocking all training-class bots while allowing all citation-class bots is a coherent stance for firms with proprietary research they want cited but not used as training data.

The default for B2B finance YMYL sites is allow all. The asymmetry favors citation visibility over training-data protection.

Selective Blocking Patterns

Four patterns worth knowing:

  1. Block training-class, allow citation-class. Allows real-time citation behavior while preventing training data extraction. The bots blocked: GPTBot, ClaudeBot, Google-Extended, Applebot-Extended, CCBot. The bots allowed: OAI-SearchBot, ChatGPT-User, Claude-Web, PerplexityBot, Perplexity-User, Amazonbot. This is the most defensible split for firms concerned about training-data extraction.

  2. Allow citation-class, block specific engines entirely. A firm with a competitive concern with OpenAI specifically might block both GPTBot and OAI-SearchBot while allowing Anthropic and Perplexity. The cost is loss of ChatGPT search citation; the benefit is full opt-out of one ecosystem.

  3. Allow all bots, monitor server logs. The fully-open posture with observability. Monitor which bots actually crawl, which pages they visit, and which queries surface in downstream AI citations. Adjust as patterns emerge. This is the recommended starting posture for most sites.

  4. Block all AI bots, accept the visibility cost. Rare. Some firms in regulated industries (defense, certain financial services) have explicit policies against AI training of their content. The robots.txt achieves the policy at the cost of full AI search invisibility.

The Production robots.txt Pattern

For most B2B finance YMYL sites in 2026, the production robots.txt explicitly enumerates all eleven bots with Allow directives:

User-agent: *
Allow: /

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Claude-Web
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Applebot-Extended
Allow: /

User-agent: Amazonbot
Allow: /

User-agent: CCBot
Allow: /

Sitemap: https://example.com/sitemap-index.xml

The explicit enumeration is the key element. The wildcard alone (User-agent: *) is technically sufficient, but the explicit enumeration removes ambiguity and signals to bots that respect explicit allow patterns.

Worked Example

A wealth management firm had a default WordPress robots.txt with Disallow: /wp-admin/ and nothing else. The firm assumed AI bots were following the implicit allow.

Audit: server log analysis showed that PerplexityBot was crawling at one-tenth the frequency of Googlebot. ClaudeBot was barely visible. The implicit allow was being honored but with reduced crawl priority compared to explicitly-allowed sites.

Fix: replaced the robots.txt with the explicit enumeration pattern above. Six weeks later, server log analysis showed PerplexityBot crawl frequency had increased fourfold. ClaudeBot frequency increased threefold. AI citation traffic followed the crawl improvement.

Frequently Asked Questions

What happens if a new AI bot launches and is not in my robots.txt?

The wildcard User-agent: * rule applies as the fallback. The new bot reads the wildcard and inherits its policy. Adding the new bot explicitly when it launches is a quarterly maintenance task.

Do these bots actually respect robots.txt?

The major bots in this list (operated by OpenAI, Anthropic, Google, Apple, Amazon, Common Crawl, Perplexity) all publicly commit to honoring robots.txt. Smaller AI scrapers may not. The robots.txt is a public policy declaration that affects responsible actors.

Should I block AI bots from specific subdirectories like /pricing/ or /clients/?

Possibly. Pages that contain sensitive competitive information (detailed pricing, named clients without consent, internal-facing pages) can be blocked with bot-specific disallow rules. The pattern is the same as for traditional crawler blocking.

Does Google-Extended affect Google Search ranking?

No. Google-Extended controls only AI training. Standard Google Search uses Googlebot, which is independent. Blocking Google-Extended does not affect organic SEO.

How does this interact with the Infrastructure Gate?

robots.txt enumeration is one of the Infrastructure Gate’s five components. The others (server-side rendering, edge deployment, crawl budget signaling, observability) compose with the bot enumeration to produce the full Infrastructure passing posture.

Next In Series

The next Tuesday post covers citation worthiness: the five signals LLMs use to decide which source to cite on a query, beyond entity recognition and content quality.

About the Author

Andrés Plashal

Author of the Assistive Agent Optimization (AAO) framework. Twenty years building search and measurement systems for B2B and SEC-regulated firms. Google Partner since 2017.

Credentials: UIUC Gies College of Business (Behavioral Science), Columbia College Chicago (Interactive Arts & Media). Member: American Marketing Association, GAABS, Paid Search Association. Published researcher (SCTE/NCTA).

About → LinkedIn → Contact →