What is Crawlability?
Whether a bot can reach, fetch and read your pages at all.
What crawlability is
Crawlability is whether the crawlers you want on your site can actually reach it. That includes classic search crawlers (Googlebot, Bingbot) and the newer AI answer-engine crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Perplexity-User, Google-Extended, Applebot-Extended).
It is not one setting. It is a chain: DNS resolves, the edge (Cloudflare, Vercel, WAF) does not block the user agent or IP, robots.txt allows the path, the origin returns 200 with real HTML, the response is small enough to fit within the crawler's fetch budget, and canonicals + redirects behave. Any weak link cuts the chain.
The most common crawlability failure in 2026
Since July 2025 Cloudflare has blocked AI crawlers by default on new zones. The result: a site with a perfect robots.txt welcoming GPTBot and ClaudeBot silently 403s them at the edge, and nothing in the site's own configuration explains why. Cloudflare AI Crawl Control on the zone dashboard is the setting to check first.
The second most common: React or Vue applications that render body content only after client-side hydration. GPTBot, OAI-SearchBot, ClaudeBot and PerplexityBot do NOT execute JavaScript. If the body is not in the initial HTML, those crawlers see an empty shell and never extract the content that would have been quoted.
Frequently asked questions
- Do AI answer-engine crawlers execute JavaScript?
- Mostly no. GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Bytespider do not execute JavaScript - they read the initial HTML response only. Google's Googlebot does render JavaScript (via Chrome/WRS) and AI Overviews / Gemini ground in Google's index, so JavaScript-rendered content still surfaces there. For every non-Google AI surface, treat SSR / prerendering as required.
- How do I verify crawlability without deploying?
- Use curl with the crawler's user agent from a network that is not the office network (edge IPs behave differently on corporate IPs than they will for a real crawler). Example: `curl -A 'OAI-SearchBot' -sS https://your-site/`. The body must contain the visible content in the response. If it returns 403 or a mostly-empty shell, the chain is broken somewhere - see docs/ai-crawler-access.md for the fire drill.
Every Osoro GEO scan measures this and the rest of the CITE framework on your own site - free, in about 30 seconds. How the CITE framework works →