Should you block AI crawlers?

robots.txt is a request, not a fence. The Robots Exclusion Protocol defines how a site states which user-agents may fetch which paths, and compliant crawlers are expected to read and honour it (IETF, 2022). The major named AI crawlers from OpenAI, Google, Anthropic and Perplexity document that they follow it.

What robots.txt does not do is enforce anything. It cannot claw back content that was already collected, it does not bind third parties who assembled datasets from the open web, and it has no effect on a crawler that chooses to ignore the file. So the honest framing is narrow: a disallow rule tells the operators who play by the rules to stay out. That is worth something, but it is a policy signal, not protection in the security sense.

The bots worth naming fall into two groups by purpose.

Training and dataset crawlers collect pages to build or improve models. OpenAI's GPTBot fetches content for model training (OpenAI, 2025). Anthropic's ClaudeBot plays a similar role. Common Crawl's CCBot feeds a public dataset that many others train on in turn. Blocking these affects whether your content becomes part of a model's training material.

Answer and search crawlers fetch pages so an engine can retrieve and cite them when it responds to a live question. OpenAI runs OAI-SearchBot to surface sites in ChatGPT's search results, and ChatGPT-User for fetches a user triggers directly (OpenAI, 2025). Perplexity operates its own crawler for the same reason. Blocking these affects whether you can appear as a named source in the answer itself.

The two groups feel similar because they are all "AI bots", but the cost of blocking them is not the same. That is the whole point.

Google-Extended is the cleanest example, because Google documents it explicitly. It is not a crawler with its own user-agent that fetches pages; it is a robots.txt token that governs whether content Google already crawled may be used for generative-model training and grounding. Google states that using it does not affect a site's inclusion or ranking in Google Search (Google Search Central, 2025).

That distinction matters more than it first appears. AI Overviews are a feature inside Google Search, and they draw on the same index that ordinary results do. So blocking Google-Extended keeps your content out of Gemini's training pipeline while leaving you fully eligible to appear in AI Overviews. The training switch and the answer-visibility switch are genuinely separate, and Google-Extended only touches the first.

The lesson generalises. Before you disallow a bot, ask which of the two things you are trying to control. If the goal is to stay out of training data, the training token or the training crawler is the right target. If you reach for a blanket block instead, you often give up answer visibility you never meant to trade away.

An answer engine can only quote a source it is allowed to retrieve. If you disallow the crawler that grounds live answers, you remove yourself from the candidate set for those answers. The engine will compose its response from whatever it can reach, and that will be your competitors.

This is the part that catches teams out. They block AI crawlers to protect content, watch their competitors get named in ChatGPT and Perplexity answers for their own category, and only then discover that the blanket rule they pasted included the search crawler, not just the training one. The protection was real for training and accidental for reach.

For training-only crawlers the cost is subtler and depends on your position. Being in a model's training data can mean the model has some latent familiarity with your brand, which is not nothing. But training is a slow, indirect channel, and it is the channel most content-protection policies are actually aimed at. Blocking there is a defensible choice with a modest and mostly invisible cost.

Start from what you want, in plain terms, then translate it. A policy that says "we do not want our content training models, but we do want to be citable in AI answers" maps onto specific directives: disallow the training crawlers and the training token, allow the search and answer crawlers.

Name the user-agents individually. The Robots Exclusion Protocol matches rules by user-agent string, so a group targeting GPTBot is separate from one targeting OAI-SearchBot, even though both are OpenAI (IETF, 2022). A single wildcard group that disallows everything is the blunt instrument that causes the accidental blocks described above.

Keep the file honest about paths. If your private application lives under a path you already disallow for every crawler, you do not need a special AI rule for it; the existing block covers compliant bots regardless of who they belong to. Reserve the AI-specific groups for the genuine policy question, which is training versus answer visibility on your public content.

Then verify what actually reaches the page. robots.txt is one layer; a content delivery network or firewall in front of the site can block a crawler the file allows, silently, so the AI bot never even reaches the disallow it would have honoured. Check that your edge configuration and your robots.txt agree, because a mismatch between them is where intended policy and real behaviour drift apart (Osoro Solutions, 2026).

There is no single correct answer, because the right policy depends on what you are protecting and what you are trying to earn. A publisher licensing its archive has a real reason to restrict training use. A brand that wants to be recommended inside AI answers has an equally real reason to stay open to the crawlers that make those recommendations possible. The two are not in conflict once you separate them.

What is almost never correct is the reflexive blanket block, because it quietly couples a decision you meant to make to one you did not. Decide the training question and the visibility question separately, write the robots.txt to match, and confirm your edge rules do not override it. That is the whole discipline, and it is worth an hour to get right.