Skip to content

fix(core): only end a stream that was actually started - #1000

Open
xihaian251 wants to merge 3 commits into
apache:mainfrom
xihaian251:fix/astream-result-post-end-stream-pairing
Open

xihaian251 wants to merge 3 commits into
apache:mainfrom
xihaian251:fix/astream-result-post-end-stream-pairing

Conversation

@xihaian251

Copy link
Copy Markdown

Fixes #999.

Application.astream_result calls post_end_stream from its error path, but the
callback is only meaningful after pre_start_stream has run. That ordering does not
hold in two cases:

  • the halt action is not streaming — the if not next_action.streaming branch
    delegates to _astep and returns early on success, so pre_start_stream never runs;
  • input processing fails before the streaming phase.

Hooks that keep per-stream state cannot pair the callback. LocalTrackingClient and
S3TrackingClient index self.stream_state[app_id, action, partition_key], which only
pre_start_stream populates, so post_end_stream raises KeyError from inside the
except block and replaces the exception the application raised:

tracked=False: ValueError: original failure
tracked=True:  KeyError: ('<app_id>', 'fail', None)

Net effect: attaching a tracker changes which exception the caller sees.

Changes

  • track whether pre_start_stream completed, and only call post_end_stream from the
    error path when it did;
  • add a core test that a failing non-streaming action does not fire post_end_stream
    without pre_start_stream;
  • add a tracking test that a failing action keeps its original exception with
    LocalTrackingClient attached.

Semantics of the guard

stream_started is set only after the pre_start_stream hook call returns, so a hook
that raises during initialization is not followed by post_end_stream. The alternative
(set it before the call, so every adapter is paired even on partial initialization) is
also defensible; I picked "post only for a completed pre" because it guarantees the
error path can never mask the original exception. Happy to switch if you prefer the
other contract.

How I tested this

  • new tests: 2 passed
  • tests/core/test_application.py: 143 passed (142 before)
  • tests/tracking/test_local_tracking_client.py: 18 passed (17 before)
  • black 23.11.0 --line-length=100, isort 5.12.0 (--profile black), flake8 6.1.0:
    clean on the three changed files
  • verified that both new tests fail on 8bfd7268 without the source change

Not run: the full pre-commit suite (frontend lint needs npm), and the whole tests/
tree. Linux / Python 3.9–3.12 not tested (local environment: Windows 11, Python 3.13.1).

Notes

astream_result calls post_end_stream from its error path, but the callback is only
valid after pre_start_stream has run. That does not hold when the halt action is not
streaming (the branch delegates to _astep and returns early on success) or when input
processing fails before the streaming phase, so hooks that keep per-stream state
cannot pair the callback. LocalTrackingClient.post_end_stream indexes stream_state,
which only pre_start_stream populates; the resulting KeyError is raised from inside
the except block and replaces the exception the action raised.

Track whether pre_start_stream completed and only call post_end_stream on the error
path when it did. Adds a core test for the hook pairing and a tracking test that the
action's exception survives.
@github-actions github-actions Bot added the area/core Application, State, Graph, Actions label Oct 27, 2026
@skrawcz
skrawcz requested a balanced review from Copilot October 28, 2026 05:34

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues were identified for the change guarding stream cleanup when initialization has not completed.

No open issues 🎉

What changed in this PR

Fixes #999 by preventing Application.astream_result() from ending a stream whose initialization never completed, preserving the original exception when tracking is enabled.

Changes:

  • Guard error-path post_end_stream calls on successful completion of pre_start_stream.
  • Add regression tests for lifecycle-hook pairing and original-exception propagation with local tracking.
File Description
tests/​tracking/​test_local_tracking_client.py Verifies local tracking preserves a failing action’s original exception.
tests/​core/​test_application.py Verifies a failing non-streaming action triggers neither stream lifecycle hook.
burr/​core/​application.py Calls error-path stream cleanup only after stream initialization completes.

Review effort: Balanced

@xihaian251

Copy link
Copy Markdown
Author

CI failure analysis
I investigated the failures in Build Burr, attempt #2, and confirmed that they are unrelated to the three files changed by this PR.

  1. Python 3.9 jobs (4 failures)
    All four jobs fail during actions/setup-python@v7, before dependency installation or test execution:
    The version '3.9' with architecture 'x64' was not found for Ubuntu 26.04.
    The latest upstream main is affected by the same failures. This appears to be a runner compatibility issue rather than a test regression. Pinning the Python 3.9 matrix entries to ubuntu-24.04 may resolve it.
  2. Pre-commit failure
    Black reformatted two example files unrelated to this PR:
    examples/image-telephone/application.py examples/integrations/hamilton/image-telephone/application.py
    Both files were unchanged from the original PR base. Upstream has since addressed their formatting in commit 5e934dde, and pre-commit now passes on main.
  3. Passing checks
    Python 3.10, 3.11, and 3.12 test matrices passed.
    Release Validation passed, including its installation and packaging checks.
    The two new regression tests passed locally, and both failed on the unpatched baseline.
    I'll keep this PR focused on fixing astream_result() masks action exceptions with a tracker KeyError when the action is non-streaming #999 rather than including unrelated CI configuration changes.
    Thanks for reviewing!

The guard skips post_end_stream when pre_start_stream never completed. The
opposite case -- a streaming action that reaches pre_start_stream and then
fails -- had no coverage, so the guard could be tightened to always skip
cleanup without any test noticing.
mohitdixit02 pushed a commit to mohitdixit02/burr that referenced this pull request Nov 3, 2026
Adds tests/core/test_lifecycle_input_semantics.py: 135 cases across the six execution
entry points x sync/async/streaming actions x sync/async hooks x tracking on/off x
fresh/resumed applications x four input shapes. Test-only; no production change.

Follow-up to apache#1019, carrying issue apache#1017.

The suite judges by observables rather than return values: one shared event list that the
input spy, the lifecycle hooks and the actions all append to as they run
(process_inputs / pre_run_step / action / post_run_step / pre_start_stream /
post_end_stream), _process_inputs call counts, pre/post pairing, and object identity
between the __context a hook receives and the one its action receives. No test assigns to
_dependency_factory or reads any other private attribute.

Coverage apache#1019's own tests do not reach: astep() and the _astep -> _step delegate for a
synchronous action, the four delegating wrappers (iterate/aiterate/stream_iterate/
astream_iterate), asynchronous hooks, class-based actions, fresh vs resumed sequence ids on
every entry point, and the one-numbering-rule property for rejected steps.

Entry-point x action-kind pairings come from measurement rather than assumption, and the
eight unsupported pairings are asserted about instead of being skipped -- no case executes
and is then excused.

test_action_exception_not_masked_under_async_streaming is a strict xfail documenting apache#1000:
post_end_stream fires for a stream that never started and its KeyError masks the action's
ValueError. That is apache#1000's defect, not apache#1019's. The marker is strict so it reports a
failure the moment apache#1000 lands rather than silently going stale.

@YongGoose YongGoose left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I checked this on top of current main (36e3ff3, which now includes #1027) in a Linux container with Python 3.12. The patch applies cleanly.

  • With the local tracker, the repro from #999 raises KeyError: ('<app_id>', 'fail', None) on main and the original ValueError with this change.
  • test_astream_result_failing_non_streaming_action_does_not_end_stream and test_astream_result_failing_action_keeps_original_exception fail on main and pass here. test_astream_result_failing_streaming_action_ends_the_stream_it_started passes on both, as expected.
  • tests/core and tests/tracking: 507 passed, 1 skipped.
  • black 23.11.0, isort 5.12.0 and flake8 6.1.0 with the pre-commit settings are clean on the three changed files.

#1016 makes the same source change for the same issue (a stream_started flag set after pre_start_stream and checked before post_end_stream). This one was opened first and also tests that a stream that did start still gets its post_end_stream.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/core Application, State, Graph, Actions

Projects

None yet

Development

Successfully merging this pull request may close these issues.

astream_result() masks action exceptions with a tracker KeyError when the action is non-streaming

3 participants