How to audit your API docs: before your users do it for you

Most API documentation gets audited the same way: someone opens a spreadsheet, clicks through every page, checks if the content looks right, and files a report. It happens once a year if it happens at all, and by the time the audit surfaces problems, developers have already hit those problems, filed support tickets, or quietly moved to a competitor.

The reason documentation audits don't stick is structural, not cultural. Every form of documentation feedback that exists today, from support tickets to usage analytics to chatbot transcripts, is reactive. It requires a real user to fail first. That's not a quality cycle, it's damage assessment.

This article introduces a different approach: a proactive quality cycle for API documentation using the PDCA framework and agent-based testing that the docs team can run on their own cadence, without waiting for users to tell them what's broken.

Why documentation audits don't work (and why teams keep running them anyway)

Traditional documentation audits are manual, infrequent, and disconnected from how developers actually use the docs. They check whether pages exist and whether the content looks correct, but not whether a developer can follow them from intent to outcome. That's why the same problems show up cycle after cycle.

The deeper issue is that documentation teams don't lack feedback. They lack feedback they can act on before a user gets hurt.

The four feedback channels and what they actually tell you

Most documentation teams rely on some combination of four inputs, and each one has a structural limitation that no amount of tooling fixes.

Support tickets capture the failures loud enough to generate a ticket. A developer who can't figure out your authentication flow and contacts support will show up here. A developer who can't figure out your authentication flow and evaluates a competitor instead will not. The silent failures, the ones where someone gives up without telling you, never appear in this channel.

Usage analytics show where developers dropped off, but not why. A page with a high bounce rate might be badly written, or it might have answered the question in the first paragraph. Without context on the developer's goal, analytics tell you where attention ends but not what went wrong. You're reading shadows on the wall.

AI chatbot transcripts are useful, but only after a user has already hit a wall. The chatbot is patching around documentation failures in real time, which is valuable for the user in that moment but means the docs team is learning about problems after they've already caused friction. The chatbot is a bandage, not a diagnostic.

User research and interviews offer the highest fidelity insight but at the lowest frequency. You run these quarterly at best, only with users who are willing to talk, and the findings take weeks to translate into documentation changes. By the time you act, the product has shipped two more releases.

None of these are bad inputs. We use all of them. But they share a common limitation: they all require a real user to fail before the docs team learns anything.

The missing piece: a proactive Check phase

If you've worked in product or engineering, you know PDCA: Plan, Do, Check, Act. It's the standard quality cycle across manufacturing, software development, and process improvement. Documentation teams execute Plan and Do well enough, defining goals and writing content, and they try to Act on whatever feedback they receive. But the Check phase, the part where you systematically evaluate whether the docs actually work, has historically had no proactive mechanism. Every existing input requires someone outside the docs team to encounter a problem first.

That's not a quality cycle. It's a reactive loop with a structural gap where the Check should be. The docs team has no way to find problems on their own cadence, which means quality depends entirely on the speed and willingness of users to report failures. And most users don't report anything. They just leave.

What a real quality cycle looks like for API documentation

PDCA applied to documentation means defining a measurable goal, running the docs against that goal, reading the results, and fixing what failed. The difference from a traditional audit is that the Check phase is proactive and repeatable, not a one-time manual review that happens annually.

Plan: define a goal that maps to a developer outcome

The goal is not "check if the auth page is accurate." It's "can someone accomplish the auth flow using only the docs?"

That distinction matters. A page-level audit can tell you that the authentication documentation exists, is up to date, and has correct code samples. It can't tell you whether a developer who reads that page, then the quickstart, then the API reference, can actually complete the flow without getting stuck. The goal has to mirror what a real developer is trying to accomplish.

Good goals are specific and user-centered: "integrate webhook authentication for a marketplace seller," "complete first API call using the SDK," or "set up a test environment and run a sample transaction." They're bespoke per product but tend to repeat within domains. Payments, identity, and infrastructure APIs each have a set of common integration goals that most developers attempt.

Do: run the Check with an agent

This is where the method becomes concrete. Give an AI agent a goal and the documentation as its only source of truth, then let it try to accomplish the goal.

The boundary conditions matter. The agent can use the docs portal, an MCP server over the docs, OpenAPI specs, and Postman or Bruno collections if they're published. It can search the open web for supporting context, like how to use a particular tool or what a specific HTTP status code means. What it cannot do is use the web for product-specific knowledge. If the answer isn't in the docs, the agent should struggle, fabricate, or fail, because that's exactly what a developer relying on the docs would do.

The agent decides how to approach the goal based on what the docs provide: hit the API directly, build a small SDK consumer, or walk through a UI flow. What this produces is a full transcript: every step the agent takes, where it stalls, what it fabricates, and where it hits a dead end.

Check: read the transcript

The transcript is the artifact that didn't exist before. It's the Check phase made concrete: a record of what happened when a goal-driven actor tried to accomplish something using only the documentation.

Findings tend to cluster into predictable categories:

• Gap: information is missing entirely. The docs never explain how to get an API key, but every subsequent step assumes you have one.

• Contradiction: two pages disagree. The authentication guide says tokens expire after 30 minutes, the SDK reference says 60.

• Ordering problem: a step assumes a prerequisite that's covered in a different section with no cross-reference.

• Ambiguous reference: it's unclear which endpoint, parameter, or version the docs are referring to.

• Fabrication: the agent invented something because the docs were silent. This is the most revealing category, because it marks the exact point where a real developer would either guess wrong or give up.

Act: fix and re-run

Fix the documented issues, then re-run the same goal. Compare the transcripts. The delta between runs is the measurable improvement: fewer steps, fewer dead ends, completed where previously failed.

This cycle runs on a cadence the docs team controls. Not the user's cadence, not the support team's cadence, not the quarterly research schedule. The docs team decides when to run it and how often, which is what makes it a quality cycle rather than a feedback channel.

We call this a Cold Read

A Cold Read is borrowed from theater: an actor performs a script with no rehearsal, no character work, and no prior context. They read what's on the page and try to perform it. When applied to documentation, the agent reads the docs cold and tries to complete a real goal. What it surfaces is what a first-time developer would hit.

The theater metaphor carries weight because it manages expectations honestly. A cold read is known to be imperfect. Nobody expects a flawless performance from an actor who just picked up the script. The name communicates the right level of expectation: this is a rough first pass that reveals what works and what doesn't, not a polished simulation.

It also centers the right thing. The documentation is the script. The agent is the actor, the instrument, not the protagonist. The thing being tested is the script, not the actor's ability. When a cold read goes badly in theater, the director looks at the script. When a Cold Read goes badly on documentation, the docs team looks at the content.

There's built-in honesty in this framing. The agent is not pretending to be a real developer. It doesn't get frustrated, it doesn't have domain expertise, and it sometimes succeeds using training data when the docs are silent. What it does reliably is surface a category of friction that real users experience but rarely articulate. A developer who gets stuck on step four of your integration guide usually doesn't file a ticket saying "the ordering of your auth flow assumed I'd already created an API key on a different page." They just spend an hour figuring it out, or they don't. The Cold Read surfaces that friction explicitly, with a transcript you can act on.

How this differs from Docs as Tests: tools like Doc Detective and the Docs as Tests methodology verify documentation mechanics. Did the code sample compile? Does the link resolve? Did the API call return the expected response? Those checks matter, and they live in the Do phase of PDCA, helping you ship correct artifacts. A Cold Read tests something different: whether the docs as a system can guide a goal-driven actor from intent to outcome. That's the Check phase. Both matter, and they test different things. Explore Docs as Tests

What a Cold Read actually finds (and what to do about it)

We've been running Cold Reads on documentation projects over the past few months. The findings cluster into the same categories each time, and most of them are invisible to traditional page-level audits because they only surface when someone tries to follow the docs end to end.

Finding one: the invisible prerequisite. A webhook authentication flow in a payments API assumed the developer had already created an API key, but the instructions for creating that key were three pages back in the getting started guide with no cross-reference. The agent fabricated a header pattern that looked plausible but didn't match the actual API. On a second run after we added a back-reference and a callout box, the agent completed the flow without fabrication.

Finding two: the undecidable quickstart. A quickstart guide listed five integration methods (SDK, direct API, hosted checkout, and two variants) in a flat list with no guidance on which to choose based on the developer's situation. The agent picked one essentially at random, hit an incompatibility with its stated goal, backtracked, and tried another. After we added a decision-tree opening that asked "what are you trying to build?" before presenting the options, the agent chose correctly on the first attempt.

Finding three: the contradiction between siblings. An SDK setup page documented token refresh behavior that contradicted the authentication guide. Both pages were technically correct for their own context, but a developer who read both (which most developers would) would get conflicting instructions. We aligned the language and added a note clarifying when each refresh behavior applies. The second run showed no confusion on this point.

One we're still working through: a flow where the docs are technically accurate but the sequence is confusing enough that even with correct information, the agent backtracks twice before completing. The content isn't wrong, but the ordering doesn't match how a developer would naturally approach the task. We haven't figured out the right restructuring yet, which is part of the point: not everything resolves in one cycle.

When the problem isn't the docs

Sometimes a Cold Read surfaces something the documentation can't fix because the product itself is the problem. An API that requires six sequential calls to accomplish what should be a two-call operation isn't a documentation failure. The docs accurately describe a confusing product.

These findings might be the most useful output of the whole process, because they give the docs team something concrete to bring to PMs and engineering teams. Instead of "the docs feel confusing," you can show a transcript: "here's an agent trying to accomplish a standard integration goal, following the docs correctly, and getting stuck because the API requires three intermediate steps that could be combined." That changes the conversation from "fix the docs" to "should we fix the product?" and it gives the documentation team a seat at that table.

How a Cold Read fits alongside what you're already doing

A Cold Read doesn't replace support tickets, analytics, chatbot transcripts, or user research. It fills the gap between them. Each input operates at a different latency, catches a different category of problem, and is controlled by a different team.

The pattern is clear: every channel except the Cold Read depends on someone outside the docs team encountering a problem first. The Cold Read is the only input where the docs team controls both the timing and the scope. That's what makes it a proactive Check rather than another reactive signal.

This doesn't mean the other channels become less important. Support tickets surface real-world problems that agents miss, analytics reveal usage patterns across the entire portal, and user research provides depth that no automated process can replicate. The Cold Read adds a layer that none of them provide: the ability to test documentation quality before a single user touches it.

Running this as a cycle, not an audit

The value of a Cold Read isn't in the single run. It's in the cadence. PDCA is not a project but a quality cycle, and the person who keeps it running is the one who owns documentation quality.

The natural trigger points are predictable:

• After a release: re-run Cold Reads against the flows that changed. If the API added a new parameter or deprecated an endpoint, the docs for that flow need to be tested against a real goal, not just checked for accuracy.

• After a support ticket cluster: when multiple developers report the same problem, run a Cold Read against the flow the tickets reference. If the agent hits the same wall, the docs are the issue. If the agent completes the flow, the problem might be environmental or contextual, and that's useful to know too.

• Before a launch: run Cold Reads against the onboarding path for a new product or feature before it goes live. Find the gaps before the first developer does.

The delta between runs is the documentation team's measurable output. "Here's what was failing three weeks ago. Here's what we changed. Here's the improvement on the re-run. And here's the question we're taking to the product team this week." That's a documentation team operating as a system with a feedback loop, not a team that writes pages and hopes for the best.

We're still learning how to run this well. Some parts of the method are stable, like the goal-definition process and the transcript categorization. Other parts are still being tuned, like how to reliably distinguish an agent limitation from a genuine doc gap, and how to handle cases where the agent succeeds using training data rather than the docs in front of it. We're publishing as we go because PDCA is iterative by definition, and claiming to have it all figured out would undermine the entire premise.

Key takeaways

• Every form of documentation feedback that exists today is reactive, requiring a real user to fail before the docs team learns anything.

• PDCA gives documentation a quality cycle, but only if the Check phase is proactive. Until now, there hasn't been a good mechanism for that.

• A Cold Read uses an AI agent to test whether docs can guide a goal-driven actor from intent to outcome, surfacing gaps, contradictions, and ordering problems that page-level audits miss.

• Cold Reads complement existing feedback channels. They don't replace support tickets, analytics, or user research, and they aren't a substitute for Docs as Tests, which verifies mechanics rather than goal completion.

• The value is in the cadence, not the single run. After releases, after support ticket clusters, before launches.

We're publishing what we learn as we run this. If you're thinking about documentation quality cycles or running something similar, the full series on how this is evolving will be published here.