Most JavaScript-Heavy Websites Are Invisible to ChatGPT

Most JavaScript-heavy websites are invisible to ChatGPT. Not partially visible. Not difficult to parse. Invisible.

GPT, Claude, Perplexity, Gemini — none of them execute JavaScript. They read what the server sends in the initial HTML response, then leave. If your content only exists after a bundle hydrates, a fetch resolves, or a client-side router renders — it doesn't exist for any AI crawling your site.

Four days ago, our own site scored 44 on the AI Readability Score (ARS). This morning: 96.

The fixes were server-side. Platform-agnostic. They persist.

Why AI crawlers skip JavaScript entirely

Traditional search crawlers like Googlebot have invested years in JavaScript rendering infrastructure. They queue pages for delayed rendering, maintain headless browser pools, and retry content extraction. That infrastructure is expensive to build and expensive to run.

AI language model crawlers have a different priority: volume. GPTBot, ClaudeBot, PerplexityBot, and Google's AI crawlers are indexing billions of pages to train models and populate real-time retrieval systems. The economics of JavaScript rendering at that scale don't work. So they skip it. They hit your URL, read the HTML payload, extract what's there, and move on.

If your homepage renders its headline in a React component that mounts on the client, that headline does not exist in the HTML payload. The crawler sees an empty <div id="root"></div> and a collection of script tags. It leaves with nothing.

The implication: your brand's presence in AI-generated answers is determined almost entirely by what your server sends in the initial response — before any JavaScript runs.

The AI Readability Score: what 44 meant

The ARS framework evaluates six components:

1. Crawler access — Is the page reachable by named AI bots? Are they blocked in robots.txt?
2. JavaScript rendering dependency — What percentage of meaningful content requires JS execution?
3. Structured data — Is there JSON-LD schema markup? Is it valid and complete?
4. Content quality — Is the content specific, substantive, and citable?
5. Content size — Is there enough content for an AI to extract meaningful signal?
6. LLM accessibility — Is there an llms.txt file? Are canonical signals clean?

A score of 44 meant we were passing on crawler access and LLM accessibility, but failing on the middle four. The majority of our content lived inside React components that only existed after hydration. Our structured data was present but incomplete. Our pages were technically reachable but practically empty from an AI crawler's perspective.

That 44 explained something we'd been watching for months: our brand appeared in AI answers, but inconsistently, and usually in summarised form rather than cited directly. AI engines had partial information about us. They were guessing the rest.

What server-side rendering actually means for AI visibility

SSR is not a new concept. But the reason to adopt it has shifted. The old argument was performance — Time to First Byte, Core Web Vitals, SEO ranking. Those remain valid. The new argument is AI citability: if an AI crawler can't read your content, your content doesn't exist in the AI knowledge layer.

Server-side rendering means your server produces complete HTML — headlines, body copy, navigation, metadata, structured data — before the response leaves your infrastructure. The client receives a document that reads correctly without executing a single line of JavaScript. AI crawlers, which behave like very fast, very literal readers who refuse to run scripts, get the full picture.

The distinction matters for every framework in common use:

• Next.js: Pages using getServerSideProps or generateStaticParams with static export are AI-readable. Pages using client-side data fetching inside useEffect are not.
• Nuxt / SvelteKit / Astro: Default SSR or SSG outputs are AI-readable. Islands using client-only directives render invisible content.
• Plain React SPA: Essentially invisible. The entire application lives in JavaScript.

The fix is not always a full framework migration. In many cases, moving data fetching from client to server, adding a static export, or pre-rendering critical pages is sufficient to lift the score significantly.

The four changes that moved us from 44 to 96

We made four targeted changes over approximately seventy-two hours. None required a framework migration. All four are permanent — they don't degrade over time, don't require ongoing maintenance, and work identically across every AI platform that indexes the web.

1. Server-rendered content for all critical pages

We audited every page against a simple test: view-source. If view-source on a page returned empty containers or minimal content, that page was failing AI crawlers. We moved content generation for our homepage, product pages, and blog index to the server layer. The HTML payload now contains the full text of every headline, description, and body section before any JavaScript runs.

2. Complete JSON-LD structured data in the document head

Structured data signals to AI systems what type of entity your page represents, who published it, what it's about, and how it relates to other entities. We had partial schema. We expanded it to include Organization, WebSite, WebPage, and BreadcrumbList types on every page, and BlogPosting with full author, date, and description fields on every article. Critically, all of this is in the <head> as inline JSON-LD — present in the initial HTML payload, not injected by JavaScript after load.

3. Robots.txt: explicitly permitting named AI crawlers

A default robots.txt that only mentions Disallow: / for certain bots, or no robots.txt at all, may be inadvertently blocking named AI crawlers depending on their user-agent handling. We added explicit Allow directives for GPTBot, ClaudeBot, PerplexityBot, GoogleExtendedBot, and CCBot. No assumptions. No implicit permissions. Named, explicit access.

4. llms.txt: a plain-language index for AI systems

The llms.txt specification provides a structured, Markdown-formatted document at /llms.txt that tells AI systems what your site contains, what your product does, and where the most relevant content lives. Think of it as a sitemap for language models. We published ours at https://regencleo.ai/llms.txt with sections covering our product capabilities, blog topics, and authoritative pages. AI crawlers that support the specification now have a direct index rather than inferring our content structure from crawl patterns.

Why "platform-agnostic" matters

A common temptation when optimising for AI visibility is to target specific platforms. ChatGPT is dominant in consumer queries, so optimise for GPTBot. Perplexity drives B2B research, so focus there. Google AI Overviews affect organic traffic, so prioritise GoogleExtendedBot.

This framing is wrong, and acting on it creates fragile optimisation that ages badly as platform market share shifts.

The fixes above work because they're improvements to the fundamental readability of your HTML. A page that's server-rendered with complete structured data and explicit crawler permissions is readable by every AI system that indexes the web — current platforms, future platforms, and enterprise AI tools that crawl the public web for proprietary knowledge bases. The improvement is infrastructure-level, not platform-specific.

Compare this to prompt engineering for specific AI products, or optimising for the specific citation patterns of one engine. Those approaches require ongoing maintenance, re-optimisation as products update, and provide zero benefit on platforms you didn't target. Server-side fixes are permanent. They compound. They transfer.

What a score of 96 looks like in practice

The score is a diagnostic, not a vanity metric. What the movement from 44 to 96 reflects is that AI crawlers can now read our site completely. Every page contains its full content in the initial HTML. Every page carries validated structured data. Our crawler permissions are explicit. Our llms.txt gives AI systems a direct path to our most authoritative content.

In citation terms: AI engines now have complete, accurate information about what we do and what we've published. The gap between what we say and what AI systems can extract and cite has closed.

The remaining four points are primarily content density and entity authority — areas that improve over time as we publish more, earn more citations, and build more structured relationships between our content and third-party sources. Those compound. The technical foundation makes compounding possible.

The diagnostic you should run today

Open any page on your site. In your browser, go to View Source (Cmd+U on Mac, Ctrl+U on Windows). Read what's there without running any JavaScript.

If your headline is present, your body copy is present, and your key claims are readable in raw HTML — you're in reasonable shape. If you see empty divs, script tags, and minimal text — you're in the same position we were four days ago.

The gap between a score of 44 and a score of 96 is not a content gap. It's an infrastructure gap. And it's closable in days, not quarters.

AI search is not a separate channel to optimise for later. It's the channel your buyers are already using. The question is whether your site exists inside it.