There are two completely different types of AI bots visiting your website right now. One type trains AI models on your content — you get nothing back. The other type cites your content in live AI answers, sending you traffic and visibility. Most websites are blocking both with the same robots.txt rules. That’s the mistake.
A September 2026 analysis of robots.txt configurations across major publishers found that 73% of sites have technical barriers that block AI crawlers — including the answering bots that would otherwise cite their content in ChatGPT, Perplexity, and Google AI Overviews. The sites aren’t trying to disappear from AI search. They set up blanket rules to block training scrapers and caught the citation crawlers in the same net.
Here’s what the data shows about which bots to block, which to allow, and what else needs to change once your crawl access is fixed.
Training Bots vs. Answering Bots: A Distinction That Determines Whether You Get Cited
The robots.txt blocking data from August 2026 (covering major news publishers, e-commerce sites, and SaaS companies) reveals a clear pattern in how sophisticated publishers are configuring access:
- GPTBot (OpenAI’s training crawler): blocked 2.33:1 vs. allowed
- ClaudeBot (Anthropic’s training crawler): blocked 2.39:1 vs. allowed
- Bytespider (ByteDance/TikTok training): blocked 4.30:1 vs. allowed
- OAI-SearchBot (OpenAI’s live citation bot): allowed 0.94:1 — roughly equal, slight allow edge
- ChatGPT-User (ChatGPT real-time web search): allowed 1.12:1 — net positive allowance
This split is intentional. Publishers who understand the distinction are blocking the bots that extract training data with no return benefit, while explicitly allowing the bots that generate live citations. The problem is that most websites copied boilerplate User-agent: * Disallow rules or added GPTBot blocks in 2023 and never revisited them — before the citation-specific bots existed or were well-documented.
The Full List of AI Bots and What They Actually Do
Before updating robots.txt, you need to know which bot does what. The 2026 AI crawler taxonomy breaks down into three categories:
Training-only crawlers (generally safe to block):
GPTBot— OpenAI training data collectionClaudeBot— Anthropic training data collectionBytespider— ByteDance/TikTok model trainingCCBot— Common Crawl (used by many model trainers)Google-Extended— Google’s opt-out token for Gemini training (separate from Google Search)
Answering/citation crawlers (allow these to appear in AI answers):
OAI-SearchBot— ChatGPT Search live web retrievalChatGPT-User— ChatGPT browsing during active conversationsPerplexityBot— Perplexity’s real-time search crawlerGooglebot— Still needed for Google AI Overviews and AI ModeClaude-SearchBot— Anthropic’s web search for live Claude queriesBingBot— Microsoft Copilot’s primary web source
Dual-purpose (judgment call):
FacebookBot— Meta AI product development; citation use case emergingAmazonbot— Alexa AI and potentially Rufus product search
The Correct robots.txt Configuration for AI Visibility
Here’s the configuration that blocks training bots while preserving full access for citation crawlers:
# Block training-only crawlers
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
# Allow citation and answering crawlers (explicit allows)
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
# Standard search bots (already allowed by default, but explicit is safer)
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /
# Default: allow everything not explicitly blocked
User-agent: *
Allow: /One critical note: if you’re running a CDN or WAF (Cloudflare, Fastly, Akamai), robots.txt rules alone are insufficient. Bot management rules at the infrastructure layer can block crawlers before they ever read your robots.txt. Check your Cloudflare bot management settings specifically — the “managed challenge” or “block” rules for “AI crawlers” categories introduced in 2025 may be catching citation bots alongside training bots.
Beyond robots.txt: What Citation Crawlers Need to Actually Extract Your Content
Access is necessary but not sufficient. Once PerplexityBot or OAI-SearchBot can reach your pages, they still need to extract citable content. Three barriers block extraction even when crawl access is open:
JavaScript rendering: Perplexity’s crawler does not fully execute JavaScript. If your key content loads via React, Vue, or client-side API calls, Perplexity sees an empty shell. Server-side rendering (SSR) or static generation is required for content to be extractable. Check this by viewing your page with JavaScript disabled — that’s roughly what Perplexity indexes.
Paywall signals: Pages with content-visibility: hidden CSS, login redirects on crawl, or noindex meta tags are explicitly excluded from AI citation pools. ChatGPT Search will not cite pages it can’t fully render. If you have a freemium model, ensure the free-tier content is fully accessible without cookies or session tokens.
Missing schema markup: Sites with valid JSON-LD schema — specifically Article, FAQPage, HowTo, or QAPage types — receive 20–40% more citations in AI-generated answers than equivalent pages without schema, according to a 2026 structured data impact analysis. The schema tells citation crawlers what type of content they’re reading and whether the source has identifiable authorship and publication context.
Content Placement Affects Citation Rate More Than Most Tactics
A SparkToro analysis of 1+ million AI citations published in January 2026 found that 44.2% of all AI citations reference content from the first 30% of the source page. AI citation crawlers weight early content more heavily — both because they parse top-down and because answer synthesis extracts the clearest, most direct statement of the key claim.
The practical implication: if your page buries its main conclusion in section four after two paragraphs of introduction and a background section, citation crawlers may cite a competitor whose page leads with the answer. Restructure your most citable content — the specific data point, the named statistic, the direct recommendation — to appear within the first 200 words of body text.
This is different from traditional SEO writing, which often delays the main claim to build context and engagement time. AI citation writing front-loads the claim, then supports it with context. The same SparkToro data found that reference-grade content — content with explicit sourcing, named experts, and specific figures — gets 3–5x more citations than equivalent content without those signals.
The Community Citation Problem Most Brand Sites Can’t Solve Alone
Even with perfect technical access and well-structured content, there’s a distribution problem: community platforms capture 52.5% of all AI citations across ChatGPT, Perplexity, and Google AI Overviews, compared to 47.5% for brand domains. Reddit is the single most cited domain across all major AI platforms. Perplexity draws 16.9% of its citations from Reddit alone.
This means a brand page explaining a concept competes against Reddit threads discussing the same concept — and the thread often wins, because it represents multiple independent perspectives that AI systems interpret as more authoritative than a single brand’s self-description.
The practical response is to participate in the communities where AI systems are looking: relevant subreddits, Quora topics, Stack Overflow threads, industry forums. Not for link-building — the citation isn’t a backlink. But when a Perplexity query on your core topic surfaces community content where your product or perspective appears, that’s a citation path no amount of on-site optimization can replicate.
Verification: How to Check Whether AI Bots Are Actually Reaching Your Site
Before and after updating your configuration, verify that the changes are working:
Server logs: Filter access logs for User-agent strings containing PerplexityBot, OAI-SearchBot, ChatGPT-User, and Claude-SearchBot. If these don’t appear in your logs within 2 weeks of opening access, something upstream (CDN, WAF, or hosting firewall) is still blocking them.
Google Search Console: The Crawl Stats report in GSC shows Googlebot crawl volume, which correlates with Google AI Overviews access. A drop in Googlebot crawl frequency often precedes a drop in AI Overview appearances.
AI answer monitoring: Run your core queries in ChatGPT Search and Perplexity directly. If competitors appear and you don’t, the access problem is likely compounded by a content structure problem — both need to be addressed together.
The Correct Sequence for Agent-Ready Technical Setup
Most site owners work on content before fixing technical access, which produces no results and leads to the conclusion that “GEO doesn’t work.” The correct order:
- Audit robots.txt — separate training bots from citation bots, update rules
- Check CDN/WAF bot management — ensure answering bots aren’t blocked at infrastructure layer
- Verify SSR or static rendering — ensure key content is visible without JavaScript execution
- Add JSON-LD schema — Article or BlogPosting at minimum; FAQPage for Q&A content
- Restructure content — move the citable claim to the first 200 words
- Monitor citation crawlers in logs — confirm access is working before optimizing further
Steps 1–3 take an hour if you have direct access to hosting configuration. Steps 4–5 take a day. Step 6 is ongoing. But none of the content-side work produces results until technical access is clean — and 73% of sites haven’t done step 1 yet.
Conclusion: Fix Access Before You Optimize Content
The most common GEO mistake in 2026 is spending time on content structure, statistics, and schema while inadvertently blocking the bots that would cite the improved content. If your robots.txt was set up before 2025 or hasn’t been reviewed since adding general AI-bot blocking rules, there’s a meaningful chance you’re stopping OAI-SearchBot or PerplexityBot from crawling your best pages.
The fix is specific and takes under an hour: audit which bot names you’re blocking, separate training crawlers from citation crawlers, and explicitly allow the answering bots by name. Then check your CDN. Then verify in logs. Content optimization built on top of clean technical access compounds. Content optimization on top of blocked access produces nothing.
Before you invest more time in content changes, find out which AI systems are currently citing your pages — and which are blocked. The LLMagnet AI Visibility Scan shows your current citation status across ChatGPT, Perplexity, and Google AI Overviews, and flags technical access issues that may be preventing crawlers from reaching your content. Run a free scan first — then you’ll know exactly where to start.