Repository navigation
[Bug]: Isolate terminal plugin error snapshots from checkpointed state #852
Description
Split from PR #758 — preserve replay completion telemetry, using reference head 0d407e1. The reference contains multiple concerns and outstanding review feedback; use its relevant implementation and tests as a starting point, not as a patch to merge wholesale.
Baseline inspected: main at 2e5135fc6f6c05024458e0be333a8325612abd6c. This issue is based on source and PR-review inspection; no runtime reproduction was executed when filing it. The scenarios below are the implementation and validation work to perform.
Problem and reproduction
OperationInfo.from_operation() already copies ErrorObject.stack_trace, but PluginExecutor._end_info() separately constructs OperationEndInfo and passes through the checkpoint's error object. A plugin that clears or appends to info.error.stack_trace in on_operation_end can therefore mutate the stored error later seen by another hook or replay. The frozen outer dataclass does not make the nested list immutable.
Reproduce by storing a FAILED operation with stack_trace=["original frame"], registering a plugin that mutates the terminal hook's stack trace, then reading the original operation and the invocation-end snapshot. Repeat through natural operation delivery and queued checkpoint-end delivery, followed by replay of the failure.
Scope and implementation guidance
- Make the existing conversion factory subclass-aware: a classmethod returning
Selfand constructingcls(...)preserves calls throughOperationInfo,OperationStartInfo, andOperationEndInfo. - Route
_end_info()throughOperationEndInfo.from_operation(operation)so operation type, custom subtype, result, attempt, timestamps, replay flag, and copied error use one conversion path. - Reuse
_copy_error; copy the mutable stack-trace list while preserving the meanings of absent errors, absent stack traces, and empty lists. Avoid a broad deep-copy or snapshot-scheduling refactor. - Add a short explanation of snapshot ownership to the plugin API documentation where useful.
Acceptance criteria
- Mutating a terminal hook's error does not change the checkpointed error or a later freshly constructed hook snapshot.
- Cover STEP, CALLBACK, CHAINED_INVOKE, and CONTEXT failures, through both natural and checkpoint delivery.
- A completed failed operation still raises the original stored failure on replay without re-executing user code or repeating its side effect.
- Factory calls return the requested info subclass and preserve
is_replayed, custom subtype, result, attempt, and timestamps. - Existing terminal-hook delivery counts and no-plugin behavior remain unchanged.
Code and reference tests
Primary code: plugin.py (OperationInfo.from_operation, _copy_error, and PluginExecutor._end_info).
Reference tests: plugin_test.py, especially test_operation_info_factory_preserves_subclass_and_replay_flag and test_plugin_error_mutations_do_not_change_checkpoint_or_later_hooks.
Maintainer rationale: reuse the converter and copy errors.
Independence and exclusions
This is a core-only fix that can be implemented and merged directly on main, without any OTel completion, retry, status, or flush work. It does not promise isolation between two plugins receiving the very same hook object; the contract here is isolation from persisted state and subsequent snapshots.
Validation
From the repository root:
hatch run dev-core:test
hatch run dev-core:typecheckRun hatch fmt --check from packages/aws-durable-execution-sdk-python. Add focused unit tests and public-handler integration coverage for the behavior described above. For OTel changes, also build the core and OTel wheels with hatch build in their package directories and run hatch run test-wheel-otel:test from the root as documented in CONTRIBUTING.md. Use existing test environments; this split does not require testing-library or CI-filter changes. Document any changed existing test expectations and their rationale in the implementing PR.
Activity
fix(testing): resolve replay PR conflicts