Skip to content

[Bug]: Isolate terminal plugin error snapshots from checkpointed state #852

Closed

Description

@zhongkechen

Split from PR #758 — preserve replay completion telemetry, using reference head 0d407e1. The reference contains multiple concerns and outstanding review feedback; use its relevant implementation and tests as a starting point, not as a patch to merge wholesale.

Baseline inspected: main at 2e5135fc6f6c05024458e0be333a8325612abd6c. This issue is based on source and PR-review inspection; no runtime reproduction was executed when filing it. The scenarios below are the implementation and validation work to perform.

Problem and reproduction

OperationInfo.from_operation() already copies ErrorObject.stack_trace, but PluginExecutor._end_info() separately constructs OperationEndInfo and passes through the checkpoint's error object. A plugin that clears or appends to info.error.stack_trace in on_operation_end can therefore mutate the stored error later seen by another hook or replay. The frozen outer dataclass does not make the nested list immutable.

Reproduce by storing a FAILED operation with stack_trace=["original frame"], registering a plugin that mutates the terminal hook's stack trace, then reading the original operation and the invocation-end snapshot. Repeat through natural operation delivery and queued checkpoint-end delivery, followed by replay of the failure.

Scope and implementation guidance

  • Make the existing conversion factory subclass-aware: a classmethod returning Self and constructing cls(...) preserves calls through OperationInfo, OperationStartInfo, and OperationEndInfo.
  • Route _end_info() through OperationEndInfo.from_operation(operation) so operation type, custom subtype, result, attempt, timestamps, replay flag, and copied error use one conversion path.
  • Reuse _copy_error; copy the mutable stack-trace list while preserving the meanings of absent errors, absent stack traces, and empty lists. Avoid a broad deep-copy or snapshot-scheduling refactor.
  • Add a short explanation of snapshot ownership to the plugin API documentation where useful.

Acceptance criteria

  • Mutating a terminal hook's error does not change the checkpointed error or a later freshly constructed hook snapshot.
  • Cover STEP, CALLBACK, CHAINED_INVOKE, and CONTEXT failures, through both natural and checkpoint delivery.
  • A completed failed operation still raises the original stored failure on replay without re-executing user code or repeating its side effect.
  • Factory calls return the requested info subclass and preserve is_replayed, custom subtype, result, attempt, and timestamps.
  • Existing terminal-hook delivery counts and no-plugin behavior remain unchanged.

Code and reference tests

Primary code: plugin.py (OperationInfo.from_operation, _copy_error, and PluginExecutor._end_info).

Reference tests: plugin_test.py, especially test_operation_info_factory_preserves_subclass_and_replay_flag and test_plugin_error_mutations_do_not_change_checkpoint_or_later_hooks.

Maintainer rationale: reuse the converter and copy errors.

Independence and exclusions

This is a core-only fix that can be implemented and merged directly on main, without any OTel completion, retry, status, or flush work. It does not promise isolation between two plugins receiving the very same hook object; the contract here is isolation from persisted state and subsequent snapshots.

Validation

From the repository root:

hatch run dev-core:test
hatch run dev-core:typecheck

Run hatch fmt --check from packages/aws-durable-execution-sdk-python. Add focused unit tests and public-handler integration coverage for the behavior described above. For OTel changes, also build the core and OTel wheels with hatch build in their package directories and run hatch run test-wheel-otel:test from the root as documented in CONTRIBUTING.md. Use existing test environments; this split does not require testing-library or CI-filter changes. Document any changed existing test expectations and their rationale in the implementing PR.

Related issues in this split

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugpkg:sdkPackage: aws-durable-execution-sdk-python

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions