There is a technical problem affecting a significant portion of modern websites that nobody is talking about in the context of GEO: AI crawlers cannot execute JavaScript. Not poorly — not inconsistently — not at all. An analysis of over 500 million GPTBot fetches found zero evidence of JavaScript execution. If your site is a single-page application built with React, Vue, Angular, or any client-side framework that renders content dynamically, the AI systems your customers use to discover products and services are seeing an empty shell.
This matters more than it did six months ago. GPTBot traffic grew 305% year-over-year. AI bots now represent 4.2% of all HTML page requests. And pages with fast First Contentful Paint — a proxy for server-rendered content — earn an average of 6.7 ChatGPT citations compared to 2.1 for slower, dynamically-rendered pages. That’s a 3x citation gap driven entirely by a rendering architecture decision most teams made years ago without thinking about AI crawlers.
What AI Crawlers Actually Receive From Your Site
When GPTBot, ClaudeBot, PerplexityBot, or OAI-SearchBot visits your site, it makes a standard HTTP GET request and processes the HTML response. That’s it. There is no headless browser, no JavaScript engine, no waiting for API calls to resolve. The crawler captures whatever is in the raw HTML bytes returned by your server.
Compare this to Googlebot, which runs a full headless Chrome instance with JavaScript execution enabled. Googlebot waits for dynamic content to load. AI crawlers do not. The comparison looks like this:
| Crawler | Executes JavaScript | Sees Dynamic Content |
|---|---|---|
| Googlebot | Yes (headless Chrome) | Yes |
| Bingbot | Limited | Partial |
| GPTBot (OpenAI) | No | No |
| OAI-SearchBot | No | No |
| ClaudeBot (Anthropic) | No | No |
| PerplexityBot | No | No |
| Meta-ExternalAgent | No | No |
For a traditional server-rendered or statically-generated site, this is fine — the HTML response contains all the content. For a React SPA, the HTML response looks something like this:
<div id="root"></div>
<script src="/static/js/main.chunk.js"></script>The crawler sees an empty div. Your product descriptions, FAQ sections, pricing tables, case studies, and service descriptions — everything that would help an AI understand what your business does — is loaded by JavaScript after the page renders in a real browser. From the AI crawler’s perspective, the page is blank.
How to Test What AI Crawlers See on Your Site
You can replicate what an AI crawler receives from any page with a single curl command:
curl -A "Mozilla/5.0 (compatible; GPTBot/1.0; +https://openai.com/gptbot)" \
--no-javascript \
https://yoursite.com/your-page/If the HTML response contains your actual page content — headings, body text, product information — you’re fine. If it returns a minimal shell with a JavaScript bundle reference and an empty container div, AI crawlers are getting nothing useful from that page.
A faster check: view-source your URL in a browser (Ctrl+U or Cmd+U). View-source shows the raw server response before JavaScript runs. If your key content doesn’t appear in view-source, it won’t appear to AI crawlers.
Which Sites Are Most Affected
The rendering gap creates very different risk profiles by site type:
High risk — client-side SPAs: Create React App, Vite + React, Vue 3 without SSR, Angular with client-side rendering only. These return empty HTML to AI crawlers. If your marketing site, SaaS product pages, or documentation was built as a pure client-side app, assume AI crawlers are not reading your content.
Medium risk — hybrid apps with partial server rendering: Some Next.js deployments use client-side navigation for most routes while only server-rendering the homepage. AI crawlers will index the server-rendered entry points but miss dynamically-loaded sections. Use the curl test on individual product pages and FAQ pages, not just your homepage.
Low risk — traditional or statically generated sites: WordPress, Webflow, traditional CMS platforms, Next.js with full SSR or static generation (SSG), Astro, Jekyll, Hugo. These return complete HTML to every request. AI crawlers receive the full content.
The Fix: Server-Side Rendering and Static Generation
There are three approaches, in order of implementation complexity:
1. Static Site Generation (SSG) — lowest effort, best performance. Generate HTML files at build time rather than at request time or in the browser. Every page becomes a static file. AI crawlers receive complete HTML immediately. Next.js (`getStaticProps`), Astro, Gatsby, and Hugo all support this. For content that doesn’t change frequently — your product pages, feature descriptions, pricing page, FAQ — SSG is the right approach. Build times are typically under 5 minutes for most marketing sites.
2. Server-Side Rendering (SSR) — best for dynamic content. Render each page on the server at request time, send complete HTML to the crawler (and the user). Next.js (`getServerSideProps`), Nuxt (universal mode), and SvelteKit all support SSR. The tradeoff is higher server load versus SSG, but crawlers and users receive fully-rendered HTML. For pages with user-specific content or real-time data that still needs to appear in AI search, SSR is the path.
3. Prerendering services — lowest code change, acceptable for simpler sites. Services like Prerender.io or Rendertron detect bot user agents and serve a pre-rendered HTML snapshot instead of the JavaScript bundle. This works for AI crawlers that identify themselves via user agent. The risk: not all AI crawlers consistently identify themselves, and pre-rendered snapshots can become stale if not refreshed frequently. Consider this a bridge solution rather than a permanent fix.
What to Prioritize First
Not every page needs to be server-rendered. Focus SSR or SSG effort on the pages that drive AI citations — the pages AI systems are most likely to quote when answering questions about your category:
- FAQ and Q&A pages — the format AI crawlers extract most readily
- Feature and product description pages — what your product does, in clear sentences
- Comparison pages — how you compare to alternatives (AI engines quote these heavily)
- Use case and customer story pages — specific, named examples with outcomes
- Pricing pages — AI search increasingly surfaces pricing for commercial queries
Your checkout flow, user dashboard, and account settings don’t need SSR for GEO purposes. They’re not the content AI systems cite. Prioritize the pages that answer the questions your target customers ask AI systems.
Verify AI Crawlers Can Access Your Site
After implementing SSR or SSG, verify three things:
robots.txt allows the main AI crawlers. Check that your robots.txt doesn’t block GPTBot, ClaudeBot, PerplexityBot, or OAI-SearchBot. Some security plugins and CDN configurations add blanket bot blocks that capture AI crawlers alongside malicious bots. The relevant robots.txt entries to verify are:
User-agent: GPTBot
Disallow:
User-agent: ClaudeBot
Disallow:
User-agent: PerplexityBot
Disallow:
User-agent: OAI-SearchBot
Disallow:An empty Disallow means allowed. Many sites that haven’t reviewed their robots.txt since 2023 are inadvertently blocking AI crawlers.
Server logs show AI crawler visits. Once your pages are server-rendered, check your access logs or analytics for GPTBot and PerplexityBot traffic. The 47.95% figure — AI bots reaching nearly half of all requests after prerendering was enabled in one case study — is an upper bound, but seeing AI crawler traffic in logs confirms they’re reaching your pages.
Curl test returns full content after deployment. Re-run the curl command above on your key pages after migrating to SSR or SSG. You should see headings, body text, and structured content in the raw HTML response.
The Connection to Citation Rates
The 3x citation gap between fast-FCP pages (6.7 citations) and slow ones (2.1 citations) isn’t explained entirely by rendering — page speed is a correlated signal. But the underlying mechanism is the same: AI crawlers retrieve and process content that’s available in the initial HTML response. Pages where content is available immediately and completely get indexed more reliably and cited more often.
This creates a compounding advantage. A site with SSR-enabled product pages that loads in under 2 seconds will be crawled more successfully by AI bots than a client-side SPA that forces crawlers to abandon after receiving an empty shell. Over months, the citation gap between server-rendered and client-rendered sites in the same category will widen — the server-rendered sites accumulate AI training signal while the SPAs remain invisible.
Conclusion: A Technical Problem With a Technical Fix
The AI crawler rendering gap is one of the few GEO problems with a clear, measurable fix. Unlike content quality improvements (which take months to show up in AI citations) or third-party link building (which depends on other publishers), switching from client-side rendering to SSR or SSG produces results the next time AI crawlers index your site — typically within 2–4 weeks for ChatGPT and Perplexity.
Run the curl test on your most important pages this week. If your content isn’t in the raw HTML response, AI systems are not seeing it. The fix is established, well-documented, and within the reach of any engineering team that’s already running a JavaScript framework.
Want to see how your site looks to AI systems right now? Run a free scan at ai-visibility.llmagnet.com — it checks AI crawler accessibility alongside 11 other readiness signals and returns results in under 5 minutes.