Definition
An AI crawler is a web bot operated by an AI company to collect content for model training, retrieval-augmented search, or both. Major AI crawlers include GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot (Perplexity AI), Google-Extended (Google, for Gemini training), and Bytespider (ByteDance). Each crawler has its own user-agent string and is controllable via robots.txt. For GEO, the critical distinction is purpose: training crawlers collect data that shapes a model's parametric knowledge (what it "knows" without searching), while search crawlers retrieve pages in real time for citation in answers. Blocking a training crawler keeps you out of future model weights; blocking a search crawler removes you from live search answers. Most GEO programs allow search crawlers while making a case-by-case decision on training crawlers. Maintain an up-to-date crawler list and review robots.txt quarterly — AI companies launch new bots faster than the industry tracks them.