Skip to content

ARS audit: fix stuck parents and late-response race; cut redundant merge work - #264

Merged
maximusunc merged 1 commit into
develfrom
claude/eloquent-hamilton-dhe670
Nov 18, 2026
Merged

maximusunc merged 1 commit into
develfrom
claude/eloquent-hamilton-dhe670

Conversation

@maximusunc

Copy link
Copy Markdown
Collaborator

An audit of the ARS from submit through fanout, premerge, merge and post-process. Three bugs fixed, four performance changes that leave the output the same.

Bugs

  1. A parent could stay Running forever when the blocklist removed every result from a merge. The completion check only counted a finished merged version toward its origin if it still had results (result_count > 0), so the counts never matched. Parents are exempt from the watchdog, so nothing ever ended the query. A finished ars-ars-agent child now always counts, since every merged version folds exactly one result-bearing ARA child (completion.evaluate_completion). This was pinned upstream behaviour; the test is updated.
  2. A failed /submit left an orphan parent Running forever. If storing the query or queuing the fanout failed, the client got a 400 but the parent row already inserted stayed Running. As a cache leader it also left identical submits WAITING on it. Submit now ends that parent E/500 through a new lifecycle.fail_parent, which the fanout's no-children failure path now shares.
  3. A timed-out child's late response could be merged after its parent had completed. If the watchdog timed a child out (E/598) while premerge was still processing its response, premerge flipped it back to Done and merged it. Premerge now re-reads the child after its pool call and rejects the late response. Intake's terminal flips are conditional on the child still being Running. One exception: a child already handed to the merge is still flipped Done, so its merged version balances the completion count.

Performance (same output)

  • No deep copy of both messages in every fold. mergeMessages(..., copy_inputs=False) merges the freshly decoded inputs in place. On a synthetic 31 MB pair one fold went from 3.7s to 1.7s. A new test checks the goldens with the copy turned off.
  • One save per merged version instead of two. The bare-fold save before post-processing was never read. The single remaining save is retried (FINAL_SAVE_ATTEMPTS) before the version ends E/422.
  • Ranker fusion no longer decodes whole ARA responses on every fold. Premerge saves a compact ranking (the results' node bindings) at ars:ranking:<pk>, deleted when the tree completes. The fusion reads it, falls back to the full response when it is missing, and skips finished children with no results without loading anything.
  • The query's parameters are read once per drain, not once per fold.

All of this is recorded in docs/ARS_PARITY_REGISTER.md: deviation 23 for the performance work, and three rows under "Stuck or unbounded state" for the bugs.

Tests

New tests cover:

  • the zero-result merge counting toward completion
  • a submit failure after the insert ending the parent, and fail_parent itself
  • a late response for a timed-out child not being merged, and the conditional status flip
  • the ranking being saved by premerge and preferred by the fusion, and childless rankers being skipped
  • the incoming ranking being snapshotted before the in-place fold
  • the final-save retry
  • the goldens matching with copy_inputs=False

The full unit suite passed locally (2,659 passed, 246 skipped). One test failed in that run because it stubbed merge_one with the old signature; I fixed it and re-ran the ARS and merge-lock tests, which passed. Skipped tests include those that need a live Postgres (SHEPHERD_TEST_PG_DSN).

Noticed but not changed

  • When a query's results first become the cached answer, the whole merged payload is decoded and re-saved just to add one log line. This happens once per cached query.
  • ?trace=1 loads a message's full stored data just to show its query graph.
  • query_event_subscribe, query_event_unsubscribe and get_status don't check that pks is a list.
  • If the merge's post-drain wake enqueue fails, a late child stays unmerged and its parent waits.

Generated by Claude Code

…rge work

Bugs:
- A Done merged version only counted toward completion with results > 0,
  so a merge whose results the blocklist removed entirely left the parent
  (watchdog-exempt) Running forever. Every Done merged version now counts.
- /submit answered 400 on a query-store or fanout-enqueue failure but left
  the inserted parent Running forever, and as a cache leader held identical
  submits WAITING on it. It is now ended E/500 via lifecycle.fail_parent,
  which the fanout's no-children failure path now shares.
- A child the watchdog timed out while premerge was processing its
  response was flipped back to Done and merged, possibly under a parent
  that had already completed. Premerge re-reads the child and rejects the
  late response; intake's terminal flips are conditional on Running (a
  child already handed to the merge is still flipped Done, so its merged
  version balances).

Performance (merge path, behavior-preserving):
- The fold no longer deep-copies both freshly decoded messages before
  merging (~2x faster fold on a 31 MB pair); goldens checked both ways.
- Each merged version is encoded and written once instead of twice; the
  single save is retried before it is E/422.
- Premerge saves a compact ranking (node bindings) per validated child, so
  ranker fusion stops decoding whole ARA responses on every fold; finished
  children with no results are skipped outright.
- Query parameters are read once per drain, not once per fold.

Documented in docs/ARS_PARITY_REGISTER.md (deviation 23 and three
stuck-state rows).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S2yuREkLYdATjDz3Goze6d
@codecov

codecov Bot commented Nov 18, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 92.59259% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 57.80%. Comparing base (7a17335) to head (fdd6766).
⚠️ Report is 1 commits behind head on devel.

Files with missing lines Patch % Lines
workers/ars_premerge/worker.py 83.33% 2 Missing and 3 partials ⚠️
shepherd_utils/ars/db.py 81.81% 1 Missing and 1 partial ⚠️
workers/ars_merge/worker.py 96.42% 0 Missing and 1 partial ⚠️
Files with missing lines Coverage Δ
shepherd_server/aras/ars.py 69.64% <100.00%> (+0.48%) ⬆️
shepherd_utils/ars/completion.py 100.00% <100.00%> (ø)
shepherd_utils/ars/lifecycle.py 93.47% <100.00%> (+1.07%) ⬆️
shepherd_utils/ars/merge.py 84.64% <100.00%> (ø)
shepherd_utils/ars/ranker_fusion.py 92.56% <100.00%> (+0.18%) ⬆️
workers/ars_fanout/worker.py 90.62% <100.00%> (-0.64%) ⬇️
workers/ars_merge/worker.py 85.59% <96.42%> (+0.82%) ⬆️
shepherd_utils/ars/db.py 50.74% <81.81%> (+1.58%) ⬆️
workers/ars_premerge/worker.py 87.62% <83.33%> (-0.10%) ⬇️

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4ffdf50...fdd6766. Read the comment docs.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@maximusunc
maximusunc merged commit bdf8fbc into devel Nov 18, 2026
2 checks passed
@maximusunc
maximusunc deleted the claude/eloquent-hamilton-dhe670 branch November 18, 2026 18:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants