English Version

Introduction: What "Invisible to AI" Actually Means as a Technical Check

Many brand teams notice their websites never appear in answers from ChatGPT, Perplexity, or Gemini and assume the problem is content quality. Before content quality, though, there is a more fundamental question: can an AI crawler physically read your page at all?

This guide skips content strategy entirely. It covers three technical checks you can run in ten minutes, in sequence:

1. View page source — confirm whether body copy already exists as plain HTML.
2. Disable JavaScript and refresh — confirm whether content survives a no-script environment.
3. Run a curl request — confirm whether the server's raw response is open to crawler-style requests.

These three steps mirror the real access sequence an AI crawler uses: fetch raw HTML first, decide whether to execute scripts second, apply robots rules third. Break any link in that chain and downstream content quality becomes irrelevant.

According to Google's JavaScript SEO Basics documentation, crawlers handle JavaScript in two distinct waves: the first wave fetches raw HTML only; the second wave renders scripts. If your content depends entirely on the second wave, a crawler may abandon the page after the first.

"Googlebot processes JavaScript and can index dynamically rendered content, but the timing and completeness of that rendering is not guaranteed." — Google Search Central, JavaScript SEO Basics

That asymmetry creates a concrete benchmark: if more than 90 percent of your body copy is injected by JavaScript at runtime, your page is effectively blank during the first-wave crawl. This is not a theoretical risk — it is the structural consequence of client-side rendering (CSR) architecture.

Step 1: View Source and Confirm Body Copy Exists in the HTML

This step takes roughly two minutes and delivers the clearest signal of all three checks.

How to do it:

Open your homepage or a core landing page in any browser, then pull up the raw source:

• Chrome / Edge: press Ctrl+U (Windows) or Cmd+Option+U (Mac).
• Alternatively, prefix the URL in the address bar with view-source: — for example, view-source:https://yoursite.com.

Once the source is open, press Ctrl+F and search for:

• The first sentence of your main body paragraph (copy the opening ten characters directly from the live page).
• Your product or service description.
• The exact text of your H1 heading.

How to interpret the result:

A common misread: teams see <div id="app"></div> and treat it as a normal loading state. It is actually the problem itself. An AI crawler's first-wave fetch returns exactly that empty container — it does not wait for JavaScript to fill it in.

This single check separates sites that output server-side rendered (SSR) HTML directly from those that rely on client-side scripts to inject content after load.

Step 2: Disable JavaScript, Refresh, and Check Whether Core Content Holds

The source view is a static snapshot. Disabling JavaScript and refreshing is a live simulation of what a crawler sees when it does not execute scripts.

How to do it in Chrome:

1. Press F12 to open Developer Tools.
2. Press Ctrl+Shift+P (Mac: Cmd+Shift+P) to open the command palette.
3. Type Disable JavaScript and select the option.
4. Close the DevTools panel and press F5 to reload the page.
5. When finished, reopen DevTools and run Enable JavaScript to restore normal behavior.

Firefox users can navigate to about:config, search for javascript.enabled, set it to false, and revert after testing.

What to check on the reloaded page:

• Is the H1 heading still visible?
• Is the core product or service description readable?
• Does the navigation menu render?
• Is the primary CTA button text present?

Risk signals — flag any of the following:

• Page is entirely blank or shows only a skeleton screen.
• Body paragraphs disappear; only structural chrome remains.
• A "Please enable JavaScript" message appears.
• Navigation or product listings vanish completely.

"If your content is only available with JavaScript enabled, search engines that do not execute JavaScript will not be able to index your content." — Google Search Central, Introduction to robots.txt

Academic research on how large language models retrieve and prioritize information (arXiv:2311.09735) indicates that LLMs show a measurable preference for structured, parseable plain text during both training and retrieval phases. Pages that rely on dynamic rendering occupy a structurally disadvantaged position in the knowledge-acquisition pipeline. This aligns directly with Google's published guidance: making core content readable without script execution is a baseline requirement for AI visibility, not an optional enhancement.

Step 3: Run curl to Simulate a Crawler Request and Inspect the Raw Response

The first two steps use the browser. This step requires a command-line terminal. A curl request bypasses the browser rendering layer entirely and replicates the HTTP-level behavior of a crawler.

Base command (macOS / Linux terminal; Windows users: PowerShell or WSL):

curl -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" \
  -L -s -o response.html -w "%{http_code}" \
  https://yoursite.com

This command does three things:

• -A sets a Googlebot User-Agent string.
• -L follows redirects automatically.
• The response body is saved to response.html; the HTTP status code is printed to the terminal.

Then inspect response.html:

grep -c "<p" response.html           # count paragraph tags
grep "noindex\|nofollow" response.html    # check for crawl-blocking directives
grep "<title" response.html          # confirm title tag is present

Interpretation table:

On robots.txt specifically, the Google Search Central robots.txt introduction is explicit: Disallow: / blocks all compliant crawlers from the entire site, and crawler-specific rules such as User-agent: GPTBot block the corresponding AI system independently. Reading your robots.txt requires no technical background — open yoursite.com/robots.txt directly in a browser.

Decision Step: Map Your Tech Stack to Risk Level, Then Set Fix Priority

After completing all three steps you have concrete signals. The next move is translating those signals into a fix sequence rather than stopping at a pass/fail verdict.

Tech stack risk reference:

Fix priority logic:

1. Fix robots.txt restrictions first. If crawlers are explicitly blocked, no other fix matters until access is restored.
2. Fix CSR structure second. Pure CSR projects need SSR or static pre-rendering introduced. This is the highest engineering effort but also the highest leverage change.
3. Add structured data last. JSON-LD, FAQ schema, and similar markup only deliver value once the underlying HTML is readable without scripts.

If your site is built on a pure Vue or React bundle and all three checks returned high risk, the practical options are: migrate to Next.js or Nuxt.js in SSR or SSG mode, or introduce a pre-rendering proxy service such as Prerender.io as an interim measure.

Technical access is the first of six gates in a complete AI visibility audit. BrandGEO runs the full six-gate check against any public URL and generates deployable fix artifacts — including a suggested llms.txt, robots.txt adjustments, JSON-LD, and FAQ structured data — with before-and-after re-check to measure the change in AI citation rate.

FAQ

Q1: Do AI crawlers have the same technical access requirements as Googlebot?

The underlying principles are the same, but the execution differs. Googlebot operates a two-wave rendering pipeline and eventually processes many JavaScript-heavy pages. Several AI crawlers — including OpenAI's GPTBot and Anthropic's anthropic-ai bot — behave closer to a first-wave-only fetch: they retrieve raw HTML and do not reliably execute JavaScript. A page that passes all three checks above presents a materially lower access risk to the majority of AI crawlers in circulation today.

Q2: My site is a single-page application (SPA). Do I have to rebuild the entire architecture?

Not necessarily. A pre-rendering proxy (Prerender.io or a self-hosted headless-Chrome renderer) can serve static HTML to crawlers without changing the user-facing experience. Treat this as a medium-term solution. Long-term, migrating to SSR or SSG architecture is more durable for both AI visibility and conventional SEO.

Q3: Does a Disallow in robots.txt affect every AI system?

It depends on the rule. User-agent: * combined with Disallow: / blocks all crawlers that respect the robots protocol, which includes Googlebot, GPTBot, and other mainstream AI crawlers. Robots.txt is an honor-system protocol, so non-compliant crawlers are unaffected. If you want to block only a specific AI crawler — GPTBot, for example — you can write a targeted rule without affecting other crawlers. See the official Google robots.txt documentation for rule syntax.

Q4: All three checks show low risk, so why doesn't any AI mention my brand?

Technical access is a necessary condition, not a sufficient one. Once a crawler can read your page, AI citation also depends on content quality, authority signals, citation density across the web, and structured data. Passing these three checks means you have removed the most basic barrier. Content-layer work is a separate evaluation.

Q5: How often should I repeat these three checks?

Re-run the checks after any framework upgrade or major site redesign, after adding new third-party scripts or CMS plugins, and whenever you notice a significant drop in AI-tool brand mentions. Under normal operating conditions, a quarterly check is a reasonable baseline.

Q6: Are Next.js and Nuxt.js automatically safe?

No. Both frameworks support SSR, SSG, and CSR modes, and the rendering strategy can differ page by page. In Next.js, a page using getStaticProps is SSG and safe; a page using a client-side fetch hook is effectively CSR. Do not assume safety based on the framework name. Run Step 1 and Step 2 on each critical page individually.

Run the free audit on your site

If any of the three checks flagged a problem — or if you want a complete AI visibility report covering technical access, content structure, structured data, and robots configuration — enter your URL at BrandGEO. The audit runs against your public URL, generates a report and deployable fix package, and requires no account registration.

Who should use this guide?

It is for teams evaluating Is Your Site Invisible to AI? The 10-Minute Test who need clear steps, evidence, and risk boundaries.

What should I confirm before acting?

Confirm the target audience, public evidence, citable site pages, and the structured content that needs attention first.

How do I tell whether the work is effective?

Track brand mentions in AI answers, cited sources, indexed pages, structured-data status, and the content quality-gate results.

What common mistakes reduce reliability?

Do not present unverified claims as facts. Keep a reachable source for every material statement and re-check accessibility after publishing.

How often should I re-check?

After deployment, verify accessibility and structured data first, then track citations and indexing on a regular cadence.

Source-backed data points

• Is Your Site Invisible to AI? The 10-Minute Test source 1：Manual archive: tasks/2026-07-12-manual-digest-source.md.
• Is Your Site Invisible to AI? The 10-Minute Test source 2：Manual archive: tasks/2026-07-12-manual-digest-source.md.