Why AI Crawlers Cannot See Your JavaScript Site

The rendering gap explained

When a modern JavaScript framework (React, Next.js in SPA mode, Vue, Angular, Svelte in client-only mode) serves a page, the initial HTTP response is often a thin HTML shell: a DOCTYPE declaration, a head section, and a body containing only a root div and a script tag. The actual content, the text your users read, is injected into the DOM by JavaScript after the browser downloads, parses, and executes the bundle.

Human browsers handle this seamlessly: the browser executes JavaScript automatically and the user sees the rendered page within milliseconds. Web crawlers face a fundamentally different situation. Most crawlers make a single HTTP request, receive the HTML response, parse the static text in that response, and move on. They do not execute JavaScript.

Which AI crawlers render JavaScript and which do not

Googlebot is the most sophisticated crawler: it runs a headless version of Chromium and executes JavaScript. Even Googlebot, however, processes JavaScript on a second pass that may lag the initial crawl by days or weeks, and its rendering queue has limited capacity.

OAI-SearchBot (OpenAI's crawler for ChatGPT), PerplexityBot, and Claude-SearchBot are primarily HTTP-only crawlers at the time of writing. They make a standard HTTP request and parse the static HTML. If your page content is client-rendered, these crawlers receive an empty shell and have nothing to cite.

Bingbot processes some JavaScript but is less consistent than Googlebot. Microsoft's documentation notes that 'JavaScript-rendered content may take additional time to be indexed.'

The practical implication: if your site serves JavaScript-rendered content, ChatGPT and Perplexity likely see nothing. Google may see it eventually. Bing may see it inconsistently.

How to check whether your site has a rendering problem

The simplest check is a curl request:

• Run: curl -s https://yourdomain.com | grep -c '<p'
• If the paragraph count is near zero, your page is likely client-rendered
• Compare this to the rendered paragraph count in your browser's DevTools
• A large gap between the two numbers confirms a rendering problem
• SnagTrace's grader measures this as the server-rendered text ratio signal

The fix options, ordered by effort

Option 1 (easiest): Enable server-side rendering (SSR) or static site generation (SSG) in your existing framework. If you are using Next.js, this is the default for the App Router. If you are using Create React App or another SPA setup, migrating to Next.js or Vite with SSR is the path forward. Next.js generates full HTML on the server, so the initial HTTP response contains all your content.

Option 2 (medium effort): Use a pre-rendering service such as Prerender.io or Rendertron. These services sit between your web server and crawlers, detect bot user agents, render the page in a headless browser, and return the fully rendered HTML. This approach works without changing your application code but adds latency and operational complexity.

Option 3 (highest effort): Rebuild the affected pages as static HTML or use an edge-side include approach where critical content is embedded in the initial response. This is the most durable fix but requires the most engineering work.

Next.js specifically: App Router vs Pages Router

Next.js with the App Router (introduced in version 13) defaults to server-side rendering for Server Components. If your Next.js site uses the App Router correctly, with content in server components rather than in 'use client' components that fetch data after mount, your pages will be server-rendered by default.

The most common mistake in Next.js App Router sites is placing content fetching inside useEffect in a 'use client' component. This pattern causes the content to be absent from the initial HTML response, exactly as if you were using a traditional SPA. The fix is to move data fetching to the server component level, where it executes during the render on the server and the result is embedded in the HTML before the response is sent.

What the rendering gap costs you

Sites with a server-rendered text ratio below 60% typically receive a D or F on SnagTrace's readability sub-score. The practical effect is that AI crawlers fetch these pages and have no content to quote. When a user asks ChatGPT a question your site should answer, ChatGPT has no text from your site to cite even if it retrieved your URL as a candidate.

Fixing the rendering gap is the highest-ROI technical change most JavaScript-heavy sites can make for AI visibility. It also improves traditional SEO, page load performance (because users see content sooner), and accessibility. Grade your site free at snagtrace.com to check your server-rendered text ratio.