Why Can't AI Find My Website?

TL;DR

If AI search can't find your website, it's almost certainly not because your content is bad—it's because your site hasn't "opened the door" at a technical level. AI crawlers are blocked, structured data is missing, key content is hidden inside JavaScript, and there's no llms.txt file telling AI who you are. Open BrandGEO, run a free audit in 30 seconds, and see your Top Issues list—each one tells you exactly which door is shut.

How Does AI Search "Find" a Website?

Before we get into why AI can't find you, take 30 seconds to understand how AI search engines discover your site. It's similar to Google search, but with a few critical differences.

The Three-Step Process: Crawl → Understand → Answer

Traditional search engines (Google, Bing) work like this: crawlers fetch your pages → index the content → return a list of matching links when someone searches. AI search engines (ChatGPT, Perplexity, Google AI Overviews) add an extra step: crawlers fetch your pages → understand the content and store it in a knowledge base → when someone asks a question, the AI pulls from that knowledge base to generate a complete answer.

The difference is in the final step: traditional search gives you a list of links to choose from, while AI search gives you a direct answer. If AI's knowledge base doesn't contain your information, it simply won't mention you when generating that answer—regardless of how good your content is.

How Are AI Crawlers Different from Traditional Crawlers?

AI search engines use their own crawlers to fetch web pages. OpenAI has GPTBot (for training and data collection) and OAI-SearchBot (for the ChatGPT search feature), Anthropic has ClaudeBot, and Perplexity has PerplexityBot. [1][3]

According to Cloudflare's data, from May 2024 to May 2025, GPTBot request volume grew 305%, and PerplexityBot grew an astonishing 157,490%. [1] AI crawlers now account for approximately 26.7% of verified bot traffic. [4] Dutch indie maker and entrepreneur Pieter Levels put it bluntly in Cloudflare's crawler research: "I'm 100% fine with AI crawlers… very important to rank in LLMs." [1]

But these AI crawlers have one critical limitation: the vast majority do not execute JavaScript. [3][6] They send an HTTP request, receive the raw HTML returned by the server, and move on. If your key content—product descriptions, FAQs, pricing tables—is rendered client-side by JavaScript, what AI crawlers see is an empty shell.

Five Common Reasons AI Can't Find Your Website

Now that you understand how AI search works, here are the five most common "closed doors." You may not have noticed any of them, but each one directly affects whether AI can read and understand your site.

Reason 1: robots.txt Is Blocking AI Crawlers

robots.txt (a text file in your website's root directory that tells various crawlers which pages they can and can't access) is the first gate AI crawlers must pass. If your robots.txt contains a Disallow rule for GPTBot or ClaudeBot, these crawlers will skip your entire site—no matter how good your content is.

This is more common than you'd think. Cloudflare's data shows that among the top 10,000 websites globally, 14% have robots.txt rules specifically targeting AI crawlers, with GPTBot being the most frequently blocked (312 domains set Disallow rules). [1] Many websites don't intentionally block AI crawlers—they simply never checked whether their robots.txt contains rules that inadvertently block AI bots, or they're using a third-party template with conservative default settings.

Reason 2: Missing Structured Data (Schema Markup)

Structured data (schema markup—standardized code tags that label what a page is about) helps AI understand the meaning of your content, not just read a wall of text. For example, your page shows "$29.99," but without schema markup, AI doesn't know if that's a product price, shipping cost, or membership fee. Add a Product schema type, and AI accurately understands it's a product priced at $29.99.

Data shows that only about 40% of websites use any form of structured data. [7] But among pages cited by ChatGPT, 71% use structured data, and sites with correct schema markup are 3.2x more likely to be cited in AI answers. [7] In plain terms: when AI has information to choose from, it prioritizes sources that have structured data over those that don't.

Reason 3: No llms.txt File

llms.txt (a plain text file designed for AI crawlers, placed in your site's root directory, telling AI "who I am, what I do, and where my key pages are") is a relatively new standard. Think of it as a cover letter for AI: your brand name, a one-line positioning statement, and a list of links to your main product or service pages.

As of June 2026, only 8.7% of the world's top 1,000 websites have deployed an llms.txt file; among the top 10,000, the figure is 5.61%. [2] Only 7.4% of Fortune 500 companies have implemented llms.txt. [2] This means the vast majority of websites don't have this "cover letter," and AI crawlers have to guess who you are and which pages matter when they visit your site.

Reason 4: Key Content Is Hidden Inside JavaScript

If your website uses frontend frameworks like React, Vue, or Angular (collectively known as single-page applications or SPAs), there's a very practical problem: AI crawlers don't execute JavaScript. seoClarity's internal data shows that on modern SPAs relying on client-side rendering, 50% to 80% of meaningful content is invisible to AI bots. [6] An analysis of over 500 million GPTBot fetches found zero evidence of JavaScript execution. [6]

This means: if your product descriptions, FAQs, or pricing information are dynamically rendered by JavaScript after the page loads, GPTBot, ClaudeBot, and PerplexityBot see either an empty page or a skeleton with just the navigation bar. They don't wait for JavaScript to finish—they take the HTML and leave.

Here's a simple way to check: right-click on your page and select "View Page Source" (not "Inspect Element"). If you can see your product names, prices, and descriptions in the source code, they're in the raw HTML and AI crawlers can read them. If you can't, that content is JavaScript-rendered, and AI crawlers most likely can't see it.

Reason 5: Outdated Content

AI search engines tend to favor fresh, recently updated content. If most of the articles on your site were written two or three years ago and haven't been touched since, AI will rank them behind more recent competitor content when generating answers. [8] For more on how content freshness specifically affects AI citations, see Why AI Never Cites Your Old Articles.

Check Where You Stand Right Now

Which of the five reasons above apply to you? Rather than guessing, spend 30 seconds finding out.

BrandGEO's free lite audit (no registration, no payment required) gives you a Top Issues list in 30 seconds—each issue corresponds to one of the "closed doors" above, with a title and detailed explanation of what the problem is and why it affects AI's understanding of your site.

If you're not sure what GEO (Generative Engine Optimization) is, take 5 minutes to read that primer first.

The lite audit's Top Issues list—each issue is a "door that hasn't been opened," telling you exactly why AI can't find you.

Step by Step: Find Your Specific Issues with BrandGEO

Step 1: Open the BrandGEO Homepage and Enter Your Domain

Visit brandgeo.app, type your website domain into the search box on the homepage, and click "Free Audit." No registration, no login required.

Enter your domain on the homepage to get started—no registration needed.

Step 2: Wait 30 Seconds, Then Review Your Score and Top Issues

After the audit completes, you'll see two things:

• A score—reflecting how well AI crawlers can read and understand your site overall.
• A list of Top Issues—specific problems affecting your AI visibility.

The score is a summary of multiple deterministic checks—it's not a subjective assessment, but calculated from verifiable indicators like "can AI crawlers access your pages," "is structured data present," and "does an llms.txt file exist."

Audit results include a score and Top Issues overview—the higher the score, the easier AI can read your site.

Step 3: Read Each Top Issue's Details

Each issue has a title and detailed explanation. Focus on two things in the details:

• "What": What this problem means (e.g., "Your robots.txt blocks GPTBot").
• "Why": How it affects your AI visibility (e.g., "GPTBot is ChatGPT's crawler—blocking it means ChatGPT search can't access your page content").

Match each issue against the "five reasons" above, and you'll pinpoint exactly which doors are closed.

Step 4: Decide Your Next Move

Now that you've identified the problems, you have two directions:

• Fix them yourself: Take action on each issue. Missing llms.txt? See How to Add llms.txt to Your Website, No Coding Required. Robots.txt or schema issues? See What Are robots.txt and Schema Markup? Do Small Businesses Need Them?. Don't know how to code? See Just Copy This Prompt to Your AI Coding Assistant.
• Run the full audit first: Click "Get Full Report," sign in, and BrandGEO runs a complete six-gate audit with raw evidence for every check item. Details in 10-Minute Self-Check: Can AI Actually Read Your Website?.

What You'll See After the Audit

If you move to the full report, the Report page shows the HTTP access status for each AI crawler (GPTBot, ClaudeBot, PerplexityBot, etc.)—not guesswork, but actual detection results showing whether each returns 200 (OK) or 403 (blocked). Structured data detection tells you which schema types are deployed and which are missing. Brand Entity recognition checks whether AI can correctly identify your brand name and business type.

The Fix Center (Fix page) generates repair tasks based on your specific issues: if llms.txt is missing, it generates the file content for you; if JSON-LD structured data is missing, it generates the corresponding code—you just copy and paste. Each task has a "Verify After Deployment" button that confirms whether the fix took effect with one click.

After fixing, run a re-audit and the Report page shows the Delta score change. For the complete re-audit loop, see Does GEO Optimization Actually Change Your Score?.

For the end-to-end workflow, jump to How to Complete the Entire GEO Process Solo, No Tech Skills Required.

Frequently Asked Questions

My website ranks fine on Google. Is it normal that AI can't find it?

Normal—but not something to accept. Google rankings rely on Googlebot crawling and signals like PageRank; AI search relies on different crawlers (GPTBot, ClaudeBot, etc.) and different content understanding methods. Good Google rankings don't mean AI crawlers can also access your pages. Cloudflare's data shows GPTBot and Googlebot are completely separate crawlers with different growth trajectories. [1] The two systems need to be addressed separately.

How do I know if robots.txt is the problem?

The quickest way: visit yourdomain.com/robots.txt in your browser (e.g., myshop.com/robots.txt) and check if the file mentions GPTBot, ClaudeBot, or PerplexityBot. If you see User-agent: GPTBot followed by Disallow: /, you're blocking ChatGPT's crawler. You can also use BrandGEO's audit, which automatically detects each AI crawler's access status.

My site is built with Shopify / WordPress / Wix. Do these issues affect me?

Yes, but to varying degrees. These platforms typically don't block AI crawlers by default (robots.txt issues are less common), but structured data and llms.txt usually need to be configured manually or via plugins. WordPress has schema plugins available; Shopify includes basic product schema but may be incomplete. Regardless of your platform, running a BrandGEO audit will show you exactly what's missing.

AI crawlers don't execute JavaScript—is my React / Next.js site completely out of luck?

Not necessarily. Next.js supports server-side rendering (SSR) and static site generation (SSG) by default, which means key content appears in the raw HTML that AI crawlers can read. The problem arises with purely client-side rendered (CSR) pages—if your product pages or core content only load in the browser, AI crawlers genuinely can't see them. The fix: ensure core content uses SSR or SSG, not pure CSR. BrandGEO's audit checks the difference between your page's "raw word count vs. rendered word count"—a large gap indicates significant JavaScript-rendered content.

Sources

[1] Cloudflare. (2025). From Googlebot to GPTBot: Who's Crawling Your Site in 2025. Pieter Levels quote from the same source.

[2] Rankability. (2026). LLMS.txt Adoption: 8.7% of the Top 1,000 (June 2026).

[3] Contently. (2026). AI Crawlers Explained: GPTBot, ClaudeBot, and PerplexityBot.

[4] Digital Applied. (2026). AI Crawler & Bot Traffic Statistics 2026: Key Data.

[5] Coronium. (2026). The Closing Web in 2026: How AI Crawler Blocking and Pay-Per-Crawl Changed Web Scraping.

[6] seoClarity. (2026). Optimizing Single-Page Applications for SEO & AI Search. Internal data cited from the same source.

[7] Iorso. (2026). Schema Markup for Small Business: Get Cited by AI Search.

[8] Aggarwal, P., Murahari, V., Rajpurohit, T., Kalyan, A., Narasimhan, K., & Deshpande, A. (2024). GEO: Generative Engine Optimization. arXiv:2311.09735.

Last updated: 2026-07-23

Can't figure out why AI ignores your site? Check now. Open BrandGEO and run a free audit on your website →