What is AI Crawler?
A bot an AI company uses to fetch web pages - GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Google-Extended and peers.
What AI crawlers are
AI crawlers are the bots AI companies use to fetch web pages, identified by user-agent strings like GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, Google-Extended and Bytespider. Some fetch for model training, some for grounding live answers with sources, and several companies run separate bots for each purpose.
The distinction matters for policy: blocking a training bot keeps your content out of future model weights, while blocking a search/grounding bot removes you from the engine's live answers - and with it, any chance of being cited there.
What AI crawlers can and cannot read
Most AI crawlers do not execute JavaScript. A page whose body text only exists after client-side hydration is effectively blank to GPTBot, ClaudeBot and PerplexityBot, however good it looks in a browser. Server-rendered HTML is the single most consequential technical decision for AI readability.
Access is governed by robots.txt plus any CDN or firewall rules in front of the site. Edge bot protection is a common silent failure: Cloudflare's default AI-bot settings can block these crawlers without anything visible breaking, which is why an AI-crawler access check is part of every technical audit.
Also known as: GPTBot, ClaudeBot, PerplexityBot, AI bot.
Frequently asked questions
- Should I block AI crawlers?
- Separate the two purposes. Blocking training-only bots (Google-Extended, or GPTBot where OpenAI honours it for training) is a legitimate content-policy choice with little visibility cost. Blocking answer-engine bots (OAI-SearchBot, PerplexityBot, ClaudeBot) removes you from those engines' answers entirely - for most commercial sites that trade is a poor one.
- How do I check which AI crawlers can reach my site?
- Check robots.txt for each bot's user-agent, then verify at the edge: CDN bot-management rules can block crawlers robots.txt allows. Fetching key pages with each bot's user-agent string, from outside your network, shows what the crawler actually receives - the free GEO scan runs this access check as part of the Technical pillar.
Every Osoro GEO scan measures this and the rest of the CITE framework on your own site - free, in about 30 seconds. How the CITE framework works →