QWE AI Academy The #1 mistake when people chase AI search citations? They ship AEO checklists – llms.txt, aggressive chunking, schema theater – while the real answer still hydrates in client-side JavaScript and robots.txt blocks the bots that actually quote pages.
I’ve watched solid Google rankings stay invisible in ChatGPT or Perplexity for exactly that split. Google’s line is blunt: foundational SEO plus unique, people-first pages; the fashionable levers mostly aren’t levers for Google’s generative features.
Reader scenario: you rank, but AI never names you
Picture a B2B SaaS help center. Your “how to connect the API” guide sits top 3 on Google. A buyer asks ChatGPT or Perplexity the same thing. Two competitors get cited. A random blog gets cited. You don’t.
That isn’t bad luck. Answer engines retrieve, extract, synthesize. They need HTML they can fetch, passages they can lift intact, and reasons to trust the claim. Blue-link rank helps Google AI Overviews. It does not auto-win every other engine.
Think of a storefront with a perfect window display and a locked back door. Shoppers on the street (Googlebot with rendering) see the goods. Delivery drivers who only read the packing slip on the dock (many AI search crawlers) get an empty carton. Same URL. Different view.
What “AI search” means here
Google AI Overviews / AI Mode, ChatGPT search, Perplexity, Claude search surfaces – systems that answer with citations, not only ten blue links. Marketers say AEO or GEO. GEO’s academic root is Aggarwal et al., KDD 2024.
Success still starts with indexable, crawlable pages people can use – Google’s generative AI optimization guide mythbusts llms.txt, special AI markup, and forced chunking for Google’s own AI features. Skip those as Google levers.
Other products bring their own agents. Training crawlers (GPTBot, ClaudeBot) are not search/retrieval crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot). “Block AI” as one switch is how teams opt out of training and quietly opt out of citations.
Practical setup: access first, copy second
Engines can’t cite what they never fetch or parse.
1. robots.txt by bot role
Open /robots.txt. Allow search and user-fetch agents even when you Disallow pure training agents. A pattern many operators use (match it to your policy):
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Googlebot
Allow: /Spoof the user-agent and fetch. Named AI bots often won’t treat a bare User-agent: * Allow as enough – spell the search agents out. Re-check CDN and WAF rules; a challenge page reads like a block.
2. The answer must sit in the first HTML response
The catch is rendering. As of 2025-2026 vendor notes and fetch studies, most AI-native search crawlers do not run JavaScript the way Googlebot does. Pricing, steps, definitions that hydrate after load show up as a shell. Server-render or statically generate anything you want quoted. Disable JS, reload – that’s closer to their view.
Pro tip: curl one money URL (or view source) and search for the core claim string. Missing from raw HTML? Fix rendering before you rewrite the essay.
3. Answer-first, self-contained sections
Extractors like passages that name the subject and still make sense alone – practitioner write-ups often land in a rough 40-120 word band. Lead the page and each H2 with the direct answer, then proof.
- Claim or definition first
- One concrete number or named source next
- Caveats and steps after
Question-shaped headings help people and machines. Submit sitemaps in Google Search Console and Bing Webmaster Tools. ChatGPT search has used Bing (and related Microsoft infrastructure) as a primary documented retrieval path – see OpenAI’s ChatGPT search announcement – so a clean Google index with a thin Bing presence is a quiet failure mode.
Advanced: what the GEO paper actually rewarded
Visibility jumps of roughly 30-40% (position-adjusted) showed up when pages gained statistics, expert quotations, and outbound citations – in the GEO paper (arXiv:2311.09735) benchmark. Keyword stuffing lagged the baseline.
Don’t paste random numbers. Swap “usage is growing fast” for “According to [named study], X rose 18% in Q2.” Keep Article/FAQ/Organization JSON-LD aligned with visible text. Google does not treat structured data as a special AI-search requirement; it still helps classic SEO and rich results when it matches the page.
Pillar pages plus supporting URLs with plain-language internal links give query fan-out somewhere coherent to land. Same product, author, and org names on-site and on third-party profiles – engines stitch entities, not orphan posts.
Do all engines weight those signals the same way? Probably not. Public research still lags product changes. That’s the uncomfortable part of this field.
Honest limits
You don’t run the synthesizer. Multi-source answers can name you with no click, mis-attribute a line, or prefer a journalist’s summary over your docs. Training-cutoff memory still answers a lot of prompts; live crawl is only part of the pie.
Measurement stays messy. Search Console is growing generative reporting for Google; other engines give owners almost nothing. Prompt-blast “rank trackers” overfit one chat session – I wouldn’t bet a quarterly OKR on them. Some Google AI surfaces may need eligibility beyond plain indexing; check Search Console docs as of your publish date.
Spend the hour on HTML answers and explicit search-bot Allows. Treat llms.txt as optional agent docs if you want it – not a ranking switch for Google.
FAQ
Is classic SEO enough for AI Overviews?
For Google, mostly yes: crawlable, indexable, snippet-eligible, unique content people use. Same core stack grounds the generative layer.
Should I block all AI bots for privacy?
Split the decision. GPTBot/ClaudeBot → training. OAI-SearchBot, Claude-SearchBot, PerplexityBot → citeable retrieval. Block training if that’s policy; keep search agents allowed if you want those products to quote you. Then verify the WAF isn’t “helping” by challenging every unknown UA.
Do I need special “AI schema” or Markdown files?
No for Google generative features – Google says llms.txt and special AI markup aren’t required. Standard schema that mirrors on-page copy is still good hygiene. A vendor blog may hype Markdown side files; if Google and that blog disagree, trust the engine you’re targeting and re-read the docs quarterly. This corner of search moves.
Next action: One highest-intent URL. Raw HTML fetch – core answer present without JS? OAI-SearchBot/PerplexityBot not Disallowed? Sitemap in Bing Webmaster Tools? Rewrite the opening ~80 words into a sourced standalone answer. Re-run the same buyer prompt in ChatGPT and Perplexity a week later.