QWE AI Academy Here’s a number most llms.txt tutorials bury: Ahrefs’ log study of roughly 137,000 domains found 28% published an llms.txt – and 97% of those files received zero requests in May 2026. No bots. No humans. Nothing. (Ahrefs llms.txt study)
That doesn’t mean the file is useless. It means the popular story – “drop this at the root and ChatGPT will suddenly understand your brand” – is the wrong problem statement. The real friction is different.
The problem agents actually hit
Your API shape, pricing rules, docs tree – an agent still has to get them somehow. Scrape noisy HTML (nav, ads, JS shells)? Guess from a giant sitemap.xml? Or burn tokens crawling until the window fills with junk.
Context is still finite. HTML-to-text is lossy. Latency and money scale with every wasted token. Jeremy Howard’s llms.txt proposal (September 2024; v2 modified August 10, 2026) attacked that gap: a small Markdown file that hands the model a curated map plus links to clean .md versions of the pages that matter.
So why do existing “solutions” still fall short?
Why robots.txt and sitemap.xml don’t solve agent navigation
They answer different questions.
| File | Job | Format | What it does not do |
|---|---|---|---|
| robots.txt | Access control | Allow/disallow directives | Never explains what the good pages mean |
| sitemap.xml | Full URL inventory | XML | No prioritization, no descriptions, often no Markdown mirrors |
| llms.txt | Curated meaning map | Markdown | Does not block crawlers; not a ranking lever |
| llms-full.txt | One-shot corpus | Concatenated Markdown | Can blow past context if you dump everything |
Per the official spec, llms.txt coexists with the others. It is not a substitute. Crawlers that already honor robots.txt have no obligation to fetch it, and most search-oriented AI bots simply don’t go looking.
The catch is who shows up when anything shows up at all. Of the traffic that existed in that Ahrefs window, about 96% was bots – with coding agents a real slice of the AI share. Pull beats push.
Ship a pull-ready llms.txt (v2 rules)
Build for the case where a human or agent is pointed at your docs – Cursor, Claude, a custom RAG loader, an MCP tool – not for imaginary background discovery.
1. Place it correctly
Serve plain-text Markdown at /llms.txt (root) or under a path you control, e.g. /docs/llms.txt. A file covers URLs under its path; when several apply, agents should prefer the most specific one. That detail matters for GitHub Pages project sites that never touch the domain root.
2. Follow the fixed order
Only the H1 is required. Everything else is convention with a strict sequence:
# Your Project Name
> One tight paragraph: what it is, who it's for, the non-negotiable constraints.
Optional plain paragraphs or bullets (no extra headings here) - tips, version notes, "we are not FastAPI-compatible", etc.
## Docs
- [Quickstart](https://example.com/docs/quickstart.md): 5-minute path to first success
- [API reference](https://example.com/docs/api.md): Auth, endpoints, errors
## Examples
- [Todo CRUD app](https://example.com/examples/todo.py): Idiomatic patterns
## Optional
- [Deep dive](https://example.com/docs/advanced.md): Skip when context is tight
Link format is fixed: required [title](url), optional : notes. Prefer absolute URLs. Point at LLM-friendly Markdown – v2 allows both page.html.md and page.md – not the HTML chrome.
3. Add discoverability headers (v2)
On HTML pages, expose:
rel="alternate" type="text/markdown"→ the .md twinrel="describedby"→ the llms.txt that covers this path
HTTP Link: headers work even for non-HTML assets and can be set at the CDN without touching every template. Full change log: llmstxt.org/changes.html (v2, August 2026).
4. Keep it small; use llms-full.txt only when justified
The index should stay easy to load whole. If agents need the entire docs blob in one fetch (common for API references that fit under a few hundred thousand tokens), ship a separate llms-full.txt. Don’t force full content into the index file.
Pro tip: After you write the file, open a fresh agent chat and say only: “Read https://yoursite.com/llms.txt and tell me how to do X.” If it can’t figure out the path from that alone, fix the descriptions and link targets before you celebrate.
Real-world shape: docs-first sites
OpenAI, Anthropic, and Gemini ship llms.txt-style indexes on developer docs. Mintlify-hosted projects often get one for free; GitBook, Yoast, and Wix have generators too.
FastHTML’s public sample – the one the spec site itself leans on – pairs a short summary with Docs / Examples / Optional and .md or raw GitHub links. Agent fetches clean source. Marketing shell stays out of the context window.
That’s the usage pattern logs actually support: software documentation and agent workflows. A bakery can still publish hours, menu .md links, allergy notes. Value shows up when someone deliberately loads the file – not from silent GPTBot discovery.
Gotchas the cheerful guides skip
- Mis-served paths: Common Crawl’s July 2026 content pass found large numbers of HTTP 200s at
/llms.txtthat weren’t valid Markdown maps – on the order of ~40% text/html (SPA catch-alls) and ~10% robots.txt bodies living at the path. Some files even carried leftover ChatGPT citation markers or policy language the spec never defined. ValidateContent-Typeand the body yourself. (Common Crawl analysis) - Security: Treat any remote llms.txt your agent reads as untrusted data. Scans of thousands of live files turned up hundreds of install instructions aimed at unclaimed npm/PyPI names (Fortune 500-linked samples included); agents with shell access followed them – typosquatting in practice, including a Clerk-related package incident. Don’t put live
pip install/npm ilines with loose names in your own file either. (Security reporting on agent-followed installs) - ## Optional changed: In v1, tooling could drop that section mechanically. v2 treats it as a human convention only – agents view/search, then follow links. Auto-expansion left the proposal. Don’t assume omission.
- No ranking theater: Ship llms.txt for agent ergonomics when someone points a tool at your docs. Presence alone is not access control and not a substitute for robots.txt or sitemap.xml.
Is the glass half empty because 97% of files sit untouched? Or half full because the ones that matter – docs an engineer pastes into Cursor – actually get used on demand? Depends whether you’re optimizing for vanity crawls or for the next person (or agent) trying to call your API correctly.
FAQ
Does ChatGPT or Claude automatically read my llms.txt?
No. Plan for pull. Coding agents and tools you point at the URL consume it today; major chat providers have not committed to robots.txt-style automatic fetches on every crawl.
Do I still need robots.txt and sitemap.xml?
Yes – drop either and you regress. Example: a docs site that deleted sitemap.xml after adding llms.txt waited longer for Google/Bing to notice new reference pages, while agents still needed the Markdown map for meaning. robots.txt stays the allow/disallow layer (training vs search user-agents). sitemap.xml stays the full inventory. llms.txt adds descriptions and clean .md targets. Many teams also ship llms-full.txt for one-shot context.
What’s the minimum valid file I can ship this afternoon?
One H1 with the project name. That’s “valid” per the proposal.
Useful minimum is stricter: blockquote summary + one ## section of 5-15 annotated links to your highest-signal Markdown pages. Host UTF-8 text. Confirm robots.txt isn’t blocking the agents you care about. Run the agent test above. On Mintlify, GitBook, Yoast, or Wix, check auto-generation before hand-rolling.
Next action: Create /llms.txt with your real product name, a one-paragraph blockquote, and links to the three pages an agent must read to stop hallucinating your API. Paste that URL into Claude or Cursor and ask a real task. Fix whatever it still guesses wrong.