QWE AI Academy Most teams treat Agentic Engine Optimization like a new robots.txt checkbox. That’s backwards. If the first GET still arrives stuffed with nav chrome, legal footers, and a novel-length preamble, no index file saves you. The agent truncates, skips, or invents the missing bits – and your product analytics stay quiet.
Key takeaway: For developer docs, redesign around shared context budgets first, then add discovery files. Bolt-on-only is the weaker path. As of Osmani’s April 2026 post: quick starts under ~15K tokens, API refs under ~25K.
Brief background: what agents actually do
Agentic Engine Optimization (AEO) got a public name on Addy Osmani’s April 11, 2026 post: structure technical content so coding agents can use it, not merely render it. Five signals sit at the center – discoverability, parsability, token efficiency, capability signaling, access control.
Different job from Answer Engine Optimization (citations inside chat-style answers) and from classic SEO. Coding agents often collapse a human multi-page path into one or two HTTP requests, strip markup, count tokens, then load or silently drop the page. Turns out one research callout on that same post is brutal: Cisco’s Secure Firewall Management Center REST API Quick Start measured 193,217 tokens. Plenty to eat a practical agent window people discuss in the ~100K-200K band.
Broader academic framing shows up as “Agentic AI Optimisation” for agent-platform fit (arXiv:2504.12482). Docs teams feel a narrower pain: failure with no error banner.
Method A vs Method B
Two patterns show up in the wild. Straight comparison:
| Dimension | Method A: Bolt-on discovery | Method B: Token-first redesign |
|---|---|---|
| Core move | Ship robots.txt allows, /llms.txt, maybe skill.md / AGENTS.md |
Split and rewrite pages to hard token budgets; front-load outcomes; serve clean Markdown |
| Effort | Hours to a weekend | Days to weeks on a real docs set |
| Fixes | Findability and “can I access this?” | Whether the fetch fits remaining context and parses cleanly |
| Failure mode | Agent finds a 150K page and still truncates or hallucinates | Under-signaling if you skip capability files |
| Best for | Quick triage; already-lean docs | API/reference-heavy portals agents implement from |
Method A feels productive because the files are new and visible. Method B is uglier editorial work. For coding-agent traffic, Method B wins – discovery without budget fit is a false green light. Pair B with the lightest A layers once pages fit.
Think of the context window like a carry-on already half full of socks (user code), a laptop (tool traces), and a snack bag (chat). Your doc isn’t getting an empty suitcase. A “compliant” 24K page still loses the tail when the agent is halfway through a task.
The catch is simple: budget is shared, not page-local. That’s the whole argument for redesign before ceremony.
Token-first walkthrough
Do this in order. Skip the romance of a perfect stack on day one.
1. Measure before you moralize
Pull token counts on the real entry doors – quick start, auth, top three API resources. Rough pass: characters ÷ 4. Better: a real tokenizer. Flag pages past the guidance Osmani published in April 2026 (not law – guidance): quick starts ~<15,000, conceptual guides ~<20,000, per-endpoint API refs ~<25,000, avoid singles past ~30,000 without chunking. Keep llms.txt itself lean (~<5,000).
# Scaffold + audit with the community CLI
npx agentic-seo --url https://docs.yourproduct.com
npx agentic-seo init # llms.txt, AGENTS.md, skill.md stubs
agentic-seo on GitHub scores Discovery, Content Structure, Token Economics, Capability Signaling, and UX Bridge out of 100 (A-F). Structural heuristics, no API key, not a Google project. Handy triage. Not a promise any agent will behave – more on that under edge cases.
2. Split and front-load
Chunk by task or endpoint – not “whole product.” Outcome first. Code sample right under the claim. Tables for parameters; they compress harder than nested prose. Sidebar/footer noise? Out of the parse path.
Pro tip: Markdown twins (append
.mdor a raw route) when the platform allows it. Agents usually pay less token tax on clean MD than on full HTML chrome.
3. Then discovery and capability signals
Host a curated /llms.txt. Jeremy Howard floated the idea in September 2024; the spec lives at llmstxt.org – H1 name, short summary, H2 sections with links and one-line notes, token hints so agents can budget. Add skill.md-style cards (what it can do, required inputs, constraints, key links) so fit gets decided before a full read. In repos, AGENTS.md is drifting toward the agent-facing README: structure, conventions, sandboxes, rate limits, MCP links.
- Audit
robots.txtso known AI patterns aren’t blocked by accident. - Point
llms.txtat the new lean pages. - Surface token counts in the index and/or page meta.
- Ship a “Copy for AI” control that pastes clean Markdown.
- Watch server logs for agent fingerprints (axios, got, curl variants) – client analytics often show ~0 dwell.
Same order he sketches on the post: robots → llms.txt → measure tokens → skill.md for top APIs → Copy for AI → traffic monitoring.
If “optimized for agents” only means prettier files on a fat page, what exactly did we optimize – the catalog, or the thing the model can finish reading?
Edge cases that break the happy path
Silent lockout. A broad Disallow in robots.txt doesn’t throw to your dashboard. Agents simply never read you. Fix access before debating prose quality. (Osmani’s Layer 1 problem: misconfigured robots, zero traffic, zero errors.)
Shared-window math. Under 25K on the page still isn’t a ticket in. Heavy session already? Truncation or parametric guesses. Users blame the model, not your doc length.
Two Google-shaped stories. Osmani writes for developer portals and coding agents; his own disclosure notes Search/Webmaster guidance does not officially recommend llms.txt. Search coverage has also stressed normal SEO for AI Overviews and pushed back on separate markdown-for-LLM pages as a Search strategy. You can be agent-ready for a coding IDE agent and still follow different rules for organic Search. Don’t paste one playbook onto the other without naming the audience.
Score ≠ selection. A high agentic-seo grade means files and structure look right. The README is explicit: community tooling, directional only – not a Google endorsement, not a guarantee of agent behavior.
FAQ
Is Agentic Engine Optimization the same as Answer Engine Optimization?
No. Same acronym, different job. One chases citations inside generated answers. Osmani’s AEO asks whether a coding agent can fetch, parse, and act without choking on tokens or structure.
Do I need llms.txt if Google Search doesn’t use it?
For pure Google organic / AI Overview goals: Search has said normal SEO is the path; llms.txt isn’t a ranking input there. For docs portals aimed at coding agents – the audience in Osmani’s framing – a curated markdown map still helps pick the right URL without walking the whole tree. Ship it for that job. Agent behavior shifts; re-check the agents you actually care about.
What’s the minimum viable pass for a small API docs site?
Unblock robots for the agents you care about. Split anything past ~25-30K. Get the quick start under ~15K with the first successful call shown early. Add a short llms.txt with token notes. Run npx agentic-seo once; fix red Discovery/Token items. Skip a full skill.md garden until the top three endpoints are lean.
Otherwise you’re decorating a door that doesn’t fit the hallway.
Tonight: highest-traffic quick start → measure tokens → cut or split until it’s under 15K with “hello world” in the first screen → then add llms.txt. That loop beats a month of acronym slides.