Your docs are already being read by AI
Eric Disque · 2026-05-08
Nearly half of all traffic to documentation sites now comes from AI agents.
Not from developers reading. From Cursor, Claude Code, ChatGPT, and Perplexity retrieving, chunking, and embedding your content into responses that a developer then trusts without clicking through. Your docs are a data source now. Most of them were not written to be one.
That gap is the thing worth thinking about.
Two audiences, one document
Docs written for humans optimize for narrative. They build a mental model. They meet the reader where they are, anticipate the question behind the question, and reward skimming. A human can infer meaning from context, tolerate ambiguity, and ask a follow-up.
An AI agent does none of that. It retrieves discrete chunks, assigns relevance scores, and passes the result directly into a generation step. It does not benefit from your carefully constructed section arc. It does not read the paragraph that sets up the next one. It sees a block of text, decides whether it answers the query, and either uses it or skips it.
These are not compatible optimization targets. A doc that works well for both is not an accident. It requires a deliberate structural choice: clean semantic hierarchy, explicit parameter descriptions, well-bounded sections that carry meaning without surrounding context, and a consistent vocabulary so the retrieval step can find the right chunk at all.
Most docs were written for one audience. They are underserving both.
The staleness problem is now a retrieval problem
Stale documentation has always been an embarrassment. A developer follows outdated steps, hits an error that no longer matches the message in the guide, and files a support ticket or gives up.
That failure was recoverable. A human reader notices friction. They check the date, look for a newer version, search the issue tracker. They apply judgment.
An AI agent does not recover the same way. It retrieves stale content with the same confidence as current content. There is no friction signal. The response it generates is fluent and wrong in exactly the way that is hardest to debug—the kind of wrong that looks right until something breaks downstream.
Staleness is not a documentation quality problem anymore. It is an accuracy problem in every AI-assisted workflow your users depend on.
What llms.txt and MCP actually solve
The llms.txt convention and Model Context Protocol are not features you add on top of good documentation. They are signals that your documentation is structured enough to be useful.
llms.txt tells an AI system what exists, how it is organized, and what is in scope. It is a table of contents for machines. If your docs do not have one, AI crawlers will infer structure from whatever they find, which means the important sections compete for retrieval weight with the dated tutorial you have not touched in two years.
MCP goes further. It allows an AI agent to query your documentation as a live source at inference time, not as a snapshot it indexed last week. For anything that changes on a release cycle—APIs, configuration options, supported integrations—this matters. An agent calling an MCP-connected doc source gets what is true now, not what was true when the index was last rebuilt.
Neither of these replaces writing good documentation. They are infrastructure for documentation that is already well-structured. If your content is ambiguous, under-specified, or organized around how you think about the product rather than how your users approach tasks, llms.txt makes that ambiguity more efficiently retrievable.
The docs-as-code answer
The core problem is that documentation and code are maintained on different cycles.
Code changes with every PR. Documentation changes when someone remembers. The gap between them is documentation debt, and it compounds the same way technical debt does: slowly at first, then all at once when a new engineer trusts the wrong source.
The only durable fix is to close that cycle. Documentation updates triggered by code changes, reviewed in the same workflow, merged with the same tooling. Not a Notion ticket. A pull request.
When that is in place, the staleness problem shrinks to the review gap. You are no longer asking writers to notice that something changed. You are asking reviewers to validate that the generated update is correct before it merges. That is a smaller, more tractable task.
It also produces documentation that is structured by default. Docs generated from code analysis inherit the structure of the code. They describe what is there, not what someone remembered was there. That is the kind of documentation that retrieves well.
The spec that did not exist
Consider what happens when an AI coding assistant encounters an undocumented system.
It does not refuse. It infers. It constructs a plausible-sounding account from patterns in the surrounding code, similar projects it has seen, and general knowledge about how systems like this usually work. The output is confident. It is often mostly right. It is occasionally wrong in ways that are expensive to debug.
The answer is not to tell developers not to use AI tools. The answer is to give those tools accurate, structured, current documentation to retrieve from.
When documentation is a first-class artifact—versioned, validated, updated alongside code—it becomes the ground truth that AI agents reach for. When it is not, they improvise.
Improvisation at scale is not a documentation problem. It is a product reliability problem.
What is worth measuring
Teams that take documentation seriously in 2026 are tracking things they were not tracking before.
Which sections generate the most support tickets despite existing? That tells you where the docs are present but wrong, or present but not structured for retrieval.
Where do developers abandon an AI assistant and open a ticket instead? That tells you where the AI could not find a usable answer—which is usually a gap in the documentation, not a failure of the model.
What does AI agent traffic look like relative to human traffic, and which sections are only being read by agents? Those sections are probably not written for machine retrieval. They are due for a structural pass.
None of this is exotic instrumentation. It is treating documentation the way you treat any other system that has users and produces outcomes: with metrics, feedback loops, and a clear sense of what failure looks like.
Grounded is not a feature
The word "grounded" gets used a lot right now. What it means in practice is simple: the AI's answer came from a specific, verifiable source, and you can check it.
Grounded documentation is documentation that is current enough to retrieve from, structured enough to chunk correctly, and accurate enough to trust. That is not a model property. It is a documentation property. The model retrieves what you give it.
The teams that figured this out early are the ones whose AI-assisted developer workflows do not produce mysterious bugs from six-month-old configuration advice. Their docs are not smarter. They are just maintained.
That is the whole thing, really. Docs that are maintained, structured, and connected to the systems they describe. Not a new idea. Just one that has become much harder to avoid.