[OTel]: Recover failed completion exports and isolate invocation finalization errors #855

Split from PR #758 — preserve replay completion telemetry, using reference head 0d407e1. The reference contains multiple concerns and outstanding review feedback; use its relevant implementation and tests as a starting point, not as a patch to merge wholesale.

Baseline inspected: main at 2e5135fc6f6c05024458e0be333a8325612abd6c. This issue is based on source and PR-review inspection; no runtime reproduction was executed when filing it. The scenarios below are the implementation and validation work to perform.

Problem and reproduction

The execution-view terminal path marks an operation ended before span start/end succeeds; both views can lose or replace span state when a processor raises. A repeated notification can then be suppressed even though completion was never exported. During invocation shutdown, a failure ending one span can also prevent unrelated spans and Invocation/Workflow roots from finishing.

Use the existing natural operation hooks on main with a recording provider and a processor that fails once on start or end, then send the terminal notification again or finish the invocation. Add a permanently failing operation alongside a healthy one. Check exported identities, attempt counts, root finalization, and restored context.

Scope and implementation guidance

• Keep in-progress claims separate from successful-delivery bookkeeping. A failed claim must be released; only successful span completion records success. Prevent concurrent terminal notifications from overtaking an active attempt.
• Retain the undelivered operation and its actual span/parent data for retry through another natural notification or invocation End. Preserve span ID, parent, links, attributes, status, original timestamps, and exception events.
• Account for OTel marking a span ended before a processor raises: calling end() again on that same ended object is insufficient. Use the smallest supported recovery mechanism and document why it is needed.
• Complete a failed context segment before opening a replacement segment with the same operation ID. Preserve its identity and do not mistake an old checkpoint for the new segment's End.
• Finalize operations independently. During invocation End, allow at most one ordinary retry per failed completion, then continue other completions, open-span cleanup, Invocation, Workflow when appropriate, context release, and flush. A failed phase must not skip later cleanup phases.
• Keep non-retryable control failures for the entire invocation, including failures before finalization starts. KeyboardInterrupt / SystemExit must not be retried during cleanup or converted into ordinary exporter failures.
• Preserve owner-thread/ContextVar cleanup and per-invocation state isolation. Do not move lifecycle hooks or change the shared factory API.

Acceptance criteria

• A transient start/end failure can recover without poisoning the successful-delivery ledger; a concurrent duplicate cannot overtake the failing attempt.
• A retried span preserves its identity, parent, timestamps, attributes/links, and exception events; it does not accumulate duplicate exception events within the reconstructed span.
• A permanent failure exhausts the bounded invocation-End attempts while unrelated completions and applicable Invocation/Workflow finalization still run.
• A control exception raised during a public virtual-child terminal hook causes exactly one processor attempt, survives later invocation cleanup without another export, and propagates as the original failure.
• Ordinary child-export failures during parent setup do not prevent attachment of the parent's tracing context; user spans and correlated logs keep the intended parent.
• Cleanup restores the owner's context and logger correlation after partial Start, normal completion, failure, and overlapping invocations. No invocation clears another's state.
• Report errors after cleanup. Specify and test the observable policy: preserve a single original error; aggregate multiple errors without dropping control exceptions; report recovered transient failures consistently with the reference's documented policy.
• Document partial-delivery limits: a processor may export and then raise, so retry can repeat delivery. This is not exactly-once telemetry or a durable exporter acknowledgment.

Code and reference tests

Primary code: terminal rendering, context-segment entry, on_invocation_end, and cleanup in both OTel plugins. The reference's _invocation_state.py contains InvocationFinalization; treat it as a candidate design, not a required new abstraction.

• test_completion_export_retry.py
• test_invocation_finalization_failures.py
• test_finalization_parent_start.py
• test_invocation_owned_state.py
• test_log_filter.py
• test_completion_export_retry_int.py
• test_invocation_finalization_failures_int.py
• test_shared_invocation_state_int.py

Additional owner-isolation controls: execution_test.py and plugin_invocation_overlap_int_test.py. These protect existing factory/handler behavior; they do not require a new core lifecycle design.

Review findings to resolve

• Maintainer-reproduced early successful-ledger poisoning.
• Control failure retried when a fresh finalization object is created: the reference still needs a whole-invocation regression, not only an End-drain test.
• Child export interrupts parent context setup: cover both views and user-span/log parentage.
• Changed error/test contracts need explicit rationale: in particular, explain why test_invocation_is_released_when_span_shutdown_fails changes from a single RuntimeError to an ExceptionGroup when both Invocation and Workflow now attempt cleanup.

Independence and boundaries

Implement and validate directly on main using natural End hooks and already-open spans. Input/checkpoint completion discovery and missing-parent fallback belong to the separate replay-completion issue and are not prerequisites. This issue owns delivery attempts, span recovery, and finalization error isolation; route any later snapshot-driven delivery through the same path rather than introducing a second retry queue. Coordinate shared plugin-file edits without requiring a stacked branch.

Provider force_flush() scheduling/time limits belong to the flush issue; use the current call while testing cleanup. Preserve existing status and timestamp rules, and preserve the refined rules if the outcomes issue lands first. The already-closed context/lifecycle issues #642 and #643 are regression constraints, not tasks to reimplement. Handler error payload changes in #772 are outside scope.

Validation

From the repository root:

hatch run dev-otel:test
hatch run dev-otel:typecheck

Run hatch fmt --check from packages/aws-durable-execution-sdk-python-otel. Add focused unit tests and public-handler integration coverage for the behavior described above. For OTel changes, also build the core and OTel wheels with hatch build in their package directories and run hatch run test-wheel-otel:test from the root as documented in CONTRIBUTING.md. Use existing test environments; this split does not require testing-library or CI-filter changes. Document any changed existing test expectations and their rationale in the implementing PR.

Related issues in this split

• #852 — Core terminal error snapshot isolation
• #853 — Terminal span status and exception timestamps
• #854 — Replay completion discovery, parentage, and segment identity
• #856 — Bounded, independent provider flushing