Skip to content

fix: export visible replay completions - #869

Draft
zhongkechen wants to merge 2 commits into
mainfrom
codex/fix-854-otel-replay-completions
Draft

zhongkechen wants to merge 2 commits into
mainfrom
codex/fix-854-otel-replay-completions

Conversation

@zhongkechen

@zhongkechen zhongkechen commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Issue #, if available

Fixes #854.

Description of changes

A callback can complete while an execution is suspended, then be skipped when replay returns its parent's cached result. Both OTel views now retain visible terminal updates from invocation input and checkpoint changes, deliver them through the existing terminal span path, and export unvisited completions before Invocation closes.

Parent registration drains pending children under the actual replay segment. Completed parent contexts remain available for later updates, and invocation-end draining materializes pending ancestors before descendants. Successful observations are deduplicated per invocation; a newly opened Invocation-view ReplayChildren or virtual-context segment still receives its own natural End. A child processor failure during registration is logged without interrupting attachment of its parent's context.

This PR changes no core hook scheduling, testing-library behavior, status mapping, exception timestamps, provider flushing, or conformance handlers. Export recovery and finalization failure isolation remain #855; status/timestamp changes remain #853.

Design

  • Design note (rule 3): not needed; this repairs delivery using the existing snapshot and hook API and introduces no public API or configuration. The issue supplies the parent/delivery contract.
  • Root cause (rule 5): natural End hooks cannot observe children skipped by replay, active-span registries discard completed parent identities, and operation-level deduplication alone cannot distinguish a new context segment.
  • Simpler options considered (rule 2): an invocation-end-only snapshot scan cannot attach pending children while their actual replay parent is active. Treating the entire snapshot as new work repeats exports for ordinary replay records. Synthesizing core End hooks changes every plugin's contract. The fix therefore retains only explicit terminal updates and uses the existing per-plugin registry lock for claims and parent-index dequeue.
  • parent_first uses the standard-library topological ordering of pending updates. It does not add a second graph, repair cycles, manufacture missing operations, or treat non-recording references as proof that a terminal parent was exported.
  • Provider callbacks run outside the registry lock. Successful span completion, rather than attempted delivery, updates the deduplication set. Undelivered metadata remains available, without adding the span-reconstruction/retry machinery tracked by [OTel]: Recover failed completion exports and isolate invocation finalization errors #855.
  • Net change: +354 runtime source lines; public API members added: zero.

Assumptions

  • Operation parents form the durable service's tree. Cyclic parent input is unsupported; no speculative cycle recovery is added.
  • updated_operations contains only metadata visible to the SDK. An ID without its operation cannot reconstruct a completion. Service-shaped fixtures exercise this boundary through the public handler, but deployed-service qualification is unverified in this PR.
  • Fallback parentage cannot recreate a missing parent span. Invocation view uses Invocation for an unknown parent, while Execution view derives its logical parent reference. Visible completed input-context metadata can supply a logical reference after actual replay parents have had a chance to register.
  • Deduplication is per invocation and does not promise exactly-once backend delivery across failed/redelivered invocations or processor errors.

Validation

Python 3.13.17, Hatch 1.16.5:

  • Before implementation, the initial focused regression set failed on main: 39 failed, 3 passed. The first commit contains those tests.
  • hatch run dev-otel:test -q --tb=short: 853 passed before adding the final fixture/matrix extensions.
  • Final additional public-handler and snapshot cases: 34 passed; the expanded terminal-status matrix and partial-start cleanup set: 54 passed.
  • hatch run dev-otel:typecheck: 52 source files clean.
  • hatch fmt --check in the OTel package and git diff --check: passed.
  • Built core and OTel distributions with hatch build.
  • hatch run test-wheel-otel:test -q --tb=short: 865 OTel tests and 12 lifecycle cases passed, with installed wheel hashes and source bytes verified. The final expanded snapshot set also passed separately against those installed wheels: 54 passed.

Coverage includes input/checkpoint/natural deduplication; all terminal statuses; concurrent End/checkpoint delivery; full snapshots containing pending parent and child with an unavailable ancestor; actual replay parent preference and late updates; ReplayChildren/virtual segment re-entry; child processor isolation; cached-parent body skipping; unknown updated IDs; overlapping public invocations and warm reuse; and both public timed-branch replay and closing-batch concurrency.

No existing test expectations changed. New tests are adapted from the relevant #758 regressions, with populated-snapshot, overlap, and registration-failure cases added. The normal installed-wheel lane continues to use the testing dependency selected by the existing installer; changing that validation boundary belongs to #863.

Coordination with the other scoped fixes

Combined local validation with #865 (#853), #866 (#852), #867 (#861),
#868 (#860), and #870 (#862) passed 5,058 core/OTel/testing tests.
The composed OTel source also passed type checking (53 files) and formatting.

The branches remain independently based on main. Rebasing this PR after #865
and #866 requires resolving three overlapping hunks: retain both core README
paragraphs, use #853's normalized end_timestamp in Execution view's
_finish_operation_end while retaining #854's parent registration through
span.end, and pass error=info.error to Invocation view's _end_span before
retaining the completed parent reference. Those resolutions were used for the
combined validation; no exporter-retry or flush changes were added.

Decisions not covered by the issue or the design note

  • Use standard-library topological ordering for the invocation-end snapshot, so a completed input parent reference cannot suppress its pending terminal export.
  • Retain only non-recording references after context completion to preserve parent identity without retaining recording spans.
  • Log ordinary child-export failures during parent registration and leave their completion metadata pending; comprehensive retry policy remains separate.

Checklist

  • Every new branch has a test that fails without it (rule 4).
  • For a bug fix: the first commit adds tests that fail on current code (rule 5).
  • Every edited existing test expectation is listed in the description, with its reason (rule 5); none changed.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[OTel]: Export unvisited replay completions with the correct parent context

1 participant