AI Crawler Management (GPTBot, ClaudeBot, PerplexityBot)

AI crawler management is the practice of controlling which AI crawlers can access a website, which sections they can reach and how much load they may generate. It sits between SEO, infrastructure and AI visibility. A line in robots.txt decides whether a brand can appear in ChatGPT, Claude, Perplexity and Google AI Overviews at all.

Every major AI platform uses named user agents. The most important as of 2026:

• GPTBot: OpenAI's crawler for content that may be used to train its models. Independent of OAI-SearchBot.
• OAI-SearchBot: OpenAI's crawler for ChatGPT search retrieval.
• ChatGPT-User: user-initiated fetches when a ChatGPT user asks the model to read a page.
• Google-Extended: a control for training and grounding in some Google systems. Googlebot controls Google Search, including AI Overviews and AI Mode.
• ClaudeBot / anthropic-ai: Anthropic's crawler for Claude. Cloudflare measured a very high crawl-to-referral ratio for it in Q1 2026. It reads far more than it cites.
• PerplexityBot: Perplexity's crawler.
• Applebot-Extended: Apple's AI-specific variant of Applebot.
• Bytespider: ByteDance's crawler, often blocked over IP concerns.

Manage search access and training permissions separately. You can allow OAI-SearchBot for ChatGPT search while disallowing GPTBot for training. Choose access and load limits for each documented use.

Cloudflare's Q1 2026 analysis of robots.txt across its network found GPTBot the most-blocked AI crawler. A meaningful share of those blocks look unintentional, inherited from stock plugins, security-oriented WAF rules or earlier decisions about scraping that nobody revisited.

OpenAI documents independent controls for GPTBot and OAI-SearchBot. Blocking GPTBot opts out of training use, not ChatGPT search. Sites that block OAI-SearchBot will not appear in ChatGPT search answers, though they can still appear as navigational links. Check search access separately from training permissions.

A 2026 parse.gl analysis of Anthropic crawlers across commonly-used hosts found ClaudeBot implicitly blocked on a meaningful share of sites whose owners had no intention of blocking it. The implicit block was usually from a stock robots.txt template that listed "bot" as a disallowed pattern, inadvertently catching every named crawler.

For a retailer the fix is a standing check: confirm each quarter that the robots.txt actually served allows the AI crawlers on the product and category paths, and that CDN-level WAF rules are not overriding it with IP or user-agent blocks. It is also the first thing a Paz expert checks when a store's products are missing from AI answers, because no amount of product information work helps if the crawler cannot get in.

This example allows both search and training crawlers. Allowing GPTBot is a separate training choice, not a requirement for ChatGPT search. Specific user-agent groups need their own path restrictions, and robots.txt works as a request to crawlers, with access control enforced elsewhere:

# Allow all well-behaved crawlers on public content
User-agent: *
Disallow: /cart/
Disallow: /checkout/
Disallow: /account/
Disallow: /api/

# Explicitly allow AI crawlers
User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: anthropic-ai
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Applebot-Extended
Allow: /

Sitemap: https://www.example.com/sitemap.xml

Three things that are easy to miss. First, robots.txt is not the only layer: WAF rules and rate limits at the CDN often override it, so check both. Second, llms.txt complements robots.txt. One says what crawlers can reach, the other says what is authoritative. Third, the explicit Allow: / lines are there because stock templates sometimes include broad Disallow patterns that catch AI crawlers by accident.