Skip to content

feat(code-context): docs may not match the code; say what the history shows (0.2.18) - #42

Merged
alphali merged 1 commit into
mainfrom
claude/docs-may-drift
Nov 10, 2026
Merged

alphali merged 1 commit into
mainfrom
claude/docs-may-drift

Conversation

@alphali

@alphali alphali commented Nov 10, 2026

Copy link
Copy Markdown
Collaborator

Server side: getdoable/trd#715 (the same rule in the Round's question text).

What changes in behavior

doable-answer-questions gains one answer rule. The agent must never assume the workspace's docs match its code:

  • A README, docs/ page, spec, changelog or code comment only routes the search. implemented_behavior comes from inspected code alone.
  • A doc that names a feature, route or option the agent cannot find in the code is not evidence that it exists.
  • When a doc disagrees with the code, the agent reads the history of both: which one changed later, and whether a later commit message, changelog entry or test says the code change was intended.
  • The agent still submits both findings and the conflict, and does not decide which side is wrong. The conflict says what the history shows, such as "a later code change says this was intended, so the doc may be out of date" or "the doc is newer than this code". It contains no commit ids, branch names or dates.

The new requirement costs one or two extra read-only git log / git show calls when docs and code disagree. No schema change: the history rides in the existing conflict description. A doc claim still arrives as desired_behavior.

Tests

  • npm test: release verifier for 0.2.18, workflow check, node tests 6/6.
  • Replay on a fixture whose README was left behind by two intended code changes: 5 stale claims, and one newer doc that the code contradicts. 4 runs per arm:
    • stale claims flagged as possibly out of date: 0.2.17 5/20, this rule 20/20;
    • the newer doc wrongly called stale: 0/4 in both arms;
    • no conflict carried a commit id, branch name or date.
  • Not run: TESTING.md scenario 32 in an installed host; other answer models; the GitHub workflow, whose shallow checkout gives the agent no history.

Version 0.2.18. TESTING.md scenario 32.

🤖 Generated with Claude Code

… shows (0.2.18)

doable-answer-questions now tells the agent never to assume the workspace's docs
match its code. A README, docs page, spec, changelog or comment only routes the
search; implemented behavior comes from inspected code alone, and a doc never fills
in a step, label, route or outcome the code does not show. When a doc disagrees with
the code, the agent reads the history of both and says in the conflict which one
changed later and whether that change says it was intended, in product words with no
commit ids, branch names or dates. It submits both findings and the conflict either
way and does not decide which is wrong.

Replay on trd's crewdesk-drift fixture (Opus 5.5, 4 runs per arm): stale README
claims flagged 20/20 (0.2.17: 5/20), newer release notes wrongly called stale 0/4.
Change Set: getdoable/trd docs/change-sets/code-context-stale-docs-history.yaml.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@alphali
alphali merged commit 9220f4d into main Nov 10, 2026
1 of 2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant