Make JavaScript Sites Accessible to AI Crawlers

Mostly no. In Vercel’s analysis of AI crawler traffic across its network (published December 2024), OpenAI’s crawlers (GPTBot, OAI-SearchBot, ChatGPT-User), Anthropic’s ClaudeBot, PerplexityBot, Meta-ExternalAgent and Bytespider did not execute JavaScript. Only Google’s Gemini, which uses Googlebot’s infrastructure, and AppleBot rendered it. If your text appears only after client-side JavaScript runs, most AI crawlers receive an empty shell, and the fix is to put the content into the HTML the server sends.

Below: which crawlers render JavaScript, how to see your page the way GPTBot sees it, which rendering strategy to pick, and what must be in the initial HTML.

Which AI crawlers render JavaScript?

The table combines Vercel’s measurement with the crawler roles from each provider’s documentation. Vercel also observed that ChatGPT and Claude do download JavaScript files (11.50% and 23.84% of their requests on Vercel’s network), but they do not run them. Fetching a script is not the same as executing it.

JavaScript rendering by crawler (Vercel measurement, December 2024)

Crawler	Operator	Used for	Executes JavaScript?
GPTBot	OpenAI	Content that may be used for training models	No
OAI-SearchBot	OpenAI	Surfacing websites in ChatGPT search	No
ChatGPT-User	OpenAI	Visits pages when a user asks ChatGPT	No
ClaudeBot	Anthropic	Crawler for Claude	No
PerplexityBot	Perplexity	Crawler for Perplexity	No
Meta-ExternalAgent	Meta	Crawler for Meta AI	No
Bytespider	ByteDance	Crawler for ByteDance	No
Googlebot (also used for Gemini)	Google	Google Search, AI Overviews, Gemini	Yes, headless Chromium
Applebot	Apple	Apple search and AI features	Yes, browser-based

The roles of the three OpenAI agents come from OpenAI’s crawler documentation: GPTBot collects content that may be used for training, OAI-SearchBot surfaces sites in ChatGPT search, and ChatGPT-User acts on behalf of a user. The documentation says nothing about JavaScript rendering, which is why the independent measurement matters. Treat it as the best public data point, not a permanent guarantee: crawler capabilities change, so re-test your own pages after major releases.

Why Google is the exception

Google processes JavaScript in three phases: crawling, rendering and indexing. According to Google’s JavaScript SEO documentation, a headless Chromium renders the page “once Google’s resources allow”, and a page “may stay on this queue for a few seconds, but it can take longer than that”. The same page adds that server-side or pre-rendering “is still a great idea because it makes your website faster for users and crawlers, and not all bots can run JavaScript.” Being visible in Google Search therefore proves nothing about ChatGPT, Claude or Perplexity.

How to see your page the way GPTBot sees it

You do not need a special tool for the first check. Pick one sentence from the main content of an important page and look for it in the raw HTML.

1. Fetch the raw HTML with a crawler user agent

curl -s -A "Mozilla/5.0 (compatible; GPTBot/1.0; +https://openai.com/gptbot)" https://www.example.com/product | grep -c "a sentence from your main content"

A result of 0 means the sentence is not in the HTML the server sends. The browser shows it because JavaScript adds it later, and a non-rendering crawler will never see it. Repeat the check for your title, meta description, H1, internal links and JSON-LD.

2. Compare the source with the rendered DOM

In Chrome, view-source: shows what the server delivered; the Elements panel in DevTools shows the DOM after JavaScript. Everything that appears only in the Elements panel is invisible to most AI crawlers. A quicker visual check: disable JavaScript in DevTools (Command menu, “Disable JavaScript”) and reload. A spinner or empty layout is the page GPTBot gets.

3. Check Google separately

The URL Inspection tool in Google Search Console shows the rendered HTML Googlebot produced. Use it for Google, and the curl test for everyone else.

4. Make sure the crawlers are allowed in

Rendering only matters if the crawler may fetch the page at all. Paste your robots.txt into the AI crawler check to see which AI bots it blocks, and use the robots.txt generator if you need to change the rules per bot. Our guide to AI crawler management covers the trade-offs between training and search crawlers.

Rendering strategies compared

The goal is the same for every option: the first HTML response contains the complete, meaningful content. The choice depends on how often content changes and how much of the existing codebase you can change.

Rendering strategies and AI crawler visibility

Strategy	How it works	Best for	AI crawler visibility
Server-side rendering (SSR)	Server returns full HTML per request	Product pages, content that changes often	Very good
Static generation (SSG)	HTML built at deploy time	Blogs, docs, marketing pages	Very good
SSR or SSG with hydration	Server HTML first, JavaScript adds interactivity	Most modern sites	Very good
Dynamic rendering	Crawlers get a pre-rendered snapshot, users get the SPA	Legacy SPAs as a temporary bridge	Good, if snapshots stay fresh
Client-side rendering only	Browser builds the page from JavaScript	Logged-in apps and dashboards	Poor for AI crawlers

Server-side rendering and static generation

These are the durable fixes. In Next.js with the App Router, components are Server Components by default, so content fetched in them lands in the HTML; with the Pages Router, getServerSideProps or getStaticProps do the same job. Nuxt renders on the server by default, and Angular offers server-side rendering as well. The common mistake is moving data fetching into a client component with useEffect or a client-only fetch: the page then ships a skeleton, and AI crawlers index the skeleton.

Hydration: keep interactivity, ship the text

Server-render the public, indexable content and let JavaScript add interactivity afterwards. Personalised parts such as a greeting, cart or recommendations can stay client-side because they do not belong in an AI answer anyway.

Dynamic rendering: a bridge, not a destination

Dynamic rendering detects crawler user agents and serves them a pre-rendered snapshot, for example generated with a headless browser. It helps large legacy SPAs that cannot move to SSR quickly. But Google calls it “a workaround and not a recommended solution, because it creates additional complexities and resource requirements”. If you use it, add every AI crawler user agent you care about to the detection list, refresh snapshots when content changes, and plan the move to SSR or static generation.

What must be in the initial HTML

Run this checklist against the raw HTML of each page template, not against the browser view:

• Main content: headings, body text, product data, prices you publish, FAQ answers.
• Title and meta description: correct for the requested URL, not set later by a client-side head manager.
• Heading structure: one H1 and a logical H2/H3 order.
• Internal links: real <a href> elements; click handlers without an href are not followed.
• Structured data: JSON-LD in the server response. See our guide to JSON-LD for AI search.
• Canonical and hreflang: in the HTML head, not injected by scripts.
• Content behind tabs and accordions: present in the HTML and only hidden with CSS, not loaded on click.
• Status codes: real 404 and 301 responses from the server, not a client-side error view with status 200.

If the checklist turns up more than a few templates with missing content, the fix is usually architectural rather than page by page. Our GEO agency team can review the rendering setup with your developers.

FAQ: JavaScript and AI crawlers

Can AI crawlers execute JavaScript?

Most cannot. In Vercel's analysis of crawler traffic on its network, published in December 2024, GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Meta-ExternalAgent, Bytespider and PerplexityBot did not render JavaScript. Google's Gemini uses Googlebot's rendering infrastructure and AppleBot renders with a browser-based crawler.

Can ChatGPT crawl JavaScript content?

ChatGPT's crawlers fetch the HTML a server returns but do not execute JavaScript, according to Vercel's measurement. Content that only appears after client-side JavaScript runs is therefore invisible to GPTBot, OAI-SearchBot and ChatGPT-User. Server-side rendering or static generation fixes that.

Does Google see content rendered by JavaScript?

Yes, with a delay. Google crawls a page, puts it in a render queue and renders it with a headless Chromium once resources allow. Google's own documentation still calls server-side or pre-rendering a great idea because not all bots can run JavaScript.

Is dynamic rendering still a good solution?

Only as a stopgap. Google describes dynamic rendering as a workaround and not a recommended solution because it adds complexity and resource requirements. Google recommends server-side rendering, static rendering or hydration instead.

How can I test what an AI crawler sees on my page?

Request the page with curl and a crawler user agent, then search the raw response for a sentence from your main content. If the sentence is missing from the raw HTML but visible in the browser, it is rendered by JavaScript and most AI crawlers will not see it.

If I block GPTBot, do I disappear from ChatGPT search?

Not necessarily. OpenAI documents separate crawlers: GPTBot collects content that may be used for model training, OAI-SearchBot surfaces websites in ChatGPT search, and ChatGPT-User visits pages when a user asks. Each can be addressed separately in robots.txt.

To place this measure in the wider context of technical access, content and measurement, read our generative engine optimization guide.

To create an optional Markdown overview of your key pages, use our free llms.txt generator.