Can you trust AI to write your docs? The grounding problem

Ask a developer how they feel about letting AI write their documentation and you'll usually get a flinch. Not because the writing is bad — modern models write clean prose — but because of one specific failure mode: the confident hallucination. The doc that describes a --retry flag you never added. The README section about an endpoint that doesn't exist. The changelog entry crediting a feature from a different repo entirely.

This fear is correct. It's also the single most important thing to engineer around if you want AI anywhere near your docs. Let's be precise about why it happens and what actually fixes it.

Why models hallucinate docs specifically

A language model predicts plausible text. When you ask it to "document this change," it draws on everything it has ever seen about how code like that is usually documented — not on what your code actually does. If your function looks like it should have a timeout option because thousands of similar functions do, the model will cheerfully document a timeout option. It's not lying; it's pattern-matching, and the pattern says "this kind of function has a timeout."

Docs are especially vulnerable because they're descriptive. There's no compiler to reject a wrong sentence the way there is for wrong code. A hallucinated doc looks exactly as confident and well-formatted as a correct one. That's what makes it dangerous: it passes the eye test.

Why "just review it" isn't enough by itself

The obvious answer is "a human reviews it before merge." That's necessary — but on its own it's weak, for a human reason: reviewing fluent, confident text for subtle factual errors is hard and boring. When the prose reads well, your brain pattern-matches "looks fine" and approves. The hallucination most likely to slip through is precisely the one that's plausible — which is most of them.

So review has to be the last line of defense, not the only one. You want the machine to have already thrown out the obvious fabrications before a human ever sees the draft, so the human is reviewing something that's already structurally constrained to be true.

The fix: grounding

The constraint that does the heavy lifting is grounding: the generated text may only reference things that are actually present in the change it's documenting.

Concretely, for docs generated from a diff:

• Extract the identifiers in the draft — function names, file paths, flags, symbols.
• Check each one against the diff being documented.
• If the draft mentions an identifier that isn't in the diff, reject the draft. Don't publish it, don't "soften" it — reject and regenerate.

This flips the model's job. It can no longer be a creative writer inventing plausible features; it's forced to be a summarizer of something that demonstrably exists in the change. The hallucinated --retry flag can't survive, because --retry isn't in the diff. The endpoint that doesn't exist can't be documented, because it's not in the code that changed.

Grounding doesn't make the model smarter. It makes the failure mode structurally impossible for the most dangerous class of error.

Grounding is one gate, not the whole answer

Grounding handles "don't invent things." A few other checks matter too, and together they form a pipeline a draft has to pass before it becomes a PR:

• Grounding — no identifiers absent from the diff (the big one).
• Relevance — the change is actually documentation-worthy (a one-line constant tweak might not be).
• Scope — it documents this change, not the whole repo.
• Format / structure — it fits your existing doc conventions.
• And finally, human review — now meaningful, because it's reviewing a draft that already cleared the automated gates.

docs-keeper runs seven such gates before opening a PR. The point isn't the number — it's the principle: constrain the model so the scary failures can't reach you, then let a human make the final call on something trustworthy.

So — can you trust it?

You shouldn't trust an unconstrained model with your docs. Pasting a diff into a chat window and pasting the output into your repo is exactly the workflow that earns the flinch.

But a constrained, grounded, review-gated system is a different thing. It can't invent an API, it shows you its work as a PR, and you approve every change. That's not "trust the AI" — that's "trust the constraints, verify the output." Which is how you should treat any automation that touches your repo.

That's what docs-keeper is built on. Every doc PR it opens is grounded in your diff and reviewed by you — and you can read a real one to see exactly what survives the gates.

→ See a real grounded docs PR · Install free on GitHub