Skip to content

fix(core): propagate early close through streaming actions - #1027

Merged
skrawcz merged 8 commits into
mainfrom
fix/astream-close-propagation
Nov 6, 2026
Merged

skrawcz merged 8 commits into
mainfrom
fix/astream-close-propagation

Conversation

@skrawcz

@skrawcz skrawcz commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Stopping a streaming action early (break, closing stream_iterate / astream_iterate, cancelling the consuming task) leaves the action's generator chain suspended until garbage collection (sync) or loop.shutdown_asyncgens() (async). Anything the action holds open (HTTP/SSE streams, DB cursors, OTel spans, async with blocks) is released late and in an arbitrary context. The cause is that every forwarding layer (container → runner → action wrapper → pydantic wrapper → user generator) iterates with a bare for / async for and never closes the generator it reads from.

async with contextlib.aclosing(app.astream_iterate(halt_after=["act"])) as steps:
    async for _, container in steps:
        async for _ in container:
            break
        break
# main:   the action's `finally` hasn't run (it runs at loop shutdown, with
#         "aclose(): asynchronous generator is already running")
# branch: it has run, and the step has been reported

Behavior changes

main this PR
Action cleanup after an early stop deferred to GC / event-loop shutdown runs immediately: at the break (sync), or when aclosing / aclose() / task cancellation unwinds (async)
post_run_step for the stopped step GeneratorExit, late, in an arbitrary context after cleanup, exactly once: GeneratorExit when closed, CancelledError when cancelled. Persisters still save failed; state is not updated
Errors while stopping a cleanup error is only logged by the finalizer the first error wins: a cleanup error (also what hooks receive), else a post_run_step hook error. close() / aclose() raise it, and every repeat call raises the same error; logged at warning after a bare break. Never replaces an error that was already stopping the stream
container.close() / await container.aclose() — new: stop without draining. Idempotent; a no-op once the stream completed
Closing a stream while it is producing an item (from a hook or another task) — raises RuntimeError telling you to cancel the consuming task; the stream is left alone
exception type in post_run_step, post_run_execute_call, tracking client Exception / Optional[Exception] Optional[BaseException] (annotation only)

Unchanged: fully consumed streams, and ASGI client disconnects. On a disconnect, Starlette/FastAPI cancel through an anyio cancel scope, which keeps cancelling every await inside it. The action's cleanup starts, but an await inside it is cancelled unless wrapped in anyio.CancelScope(shield=True). Burr's await on async post_run_step hooks and persisters is cancelled too, so the stopped step isn't recorded. Resume still re-runs it, because the last saved step is the one before. Whether Burr should shield post-step hooks is a separate question.

Release-notes worthy: cleanup errors now reach hooks, and close() / aclose() raise errors from cleanup or from post_run_step hooks.

What users do

Plain Python:

  • sync: break;
  • async: break inside async with contextlib.aclosing(...), or call await container.aclose(). Python never closes an async generator at a break.
  • web servers: the async pattern; the server's cancellation does the rest.

The docs ("Stopping a stream early" in streaming-actions.rst) and the new examples/streaming-early-stop cover each case. The example prints a timeline for each demo and includes a FastAPI server.

Implementation

  • Containers: the new close() / aclose(); iterating closes the action before reporting.
  • stream_iterate / astream_iterate: closing either one closes the in-flight container.
  • Runners: the streaming _run_* / _arun_* runners close the action generator in finally.
  • Action wrappers: FunctionBasedStreamingAction, type_eraser and the pydantic wrappers use closing / aclosing.
  • Example tooling: anyio is added to the tests extra.

Testing

  • New early-stop tests: a "Stopping a streaming action early" section in tests/core/test_application.py, covering sync and async, single- and multi-step. It tests every stop path, ordering, exactly-once reporting, persistence and resume, cleanup errors, first-error-wins (cleanup vs. hook errors, the same error on every repeat close), close-while-producing, timing independence, and the anyio cancel scope (with shielded and unshielded awaited cleanup). Against main, 30 of these fail.
  • Pydantic and example tests: closing a pydantic streaming action mid-stream (in test_burr_pydantic.py), and the example's timelines (in test_streaming_early_stop_example.py).
  • Full suite: 772 passed. The only failures are environment-only (a langchain serde test, and langfuse not installed) and happen on main too.
  • Pre-commit is green.
  • Python versions: the streaming tests pass on 3.10, 3.12, 3.13 and 3.14.
  • Real server: uvicorn + curl hang-up.
  • Downstream: an internal consumer could drop its AsyncExitStack workaround.

Checklist

  • PR has an informative and human-readable title (this will be pulled into the release notes)
  • Changes are limited to a single goal (no scope creep)
  • Code passed the pre-commit check & code is left cleaner/nicer than when first encountered.
  • Any change in functionality is tested
  • New functions are documented (with a description, list of inputs, and expected output)
  • Placeholder code is flagged / future TODOs are captured in comments
  • Project documentation has been updated if adding/changing functionality.

Closing Application.astream_iterate() (or abandoning an
AsyncStreamingResultContainer) mid-action left the action's generator
chain suspended until GC / loop.shutdown_asyncgens(), so resources held
by the action (HTTP/SSE streams, cursors, spans) were released late, in
an arbitrary context, with cleanup errors only logged.

- Add AsyncStreamingResultContainer.aclose(): closes the action chain
  without draining it, runs the post-step callback once, idempotent.
- astream_iterate closes the in-flight container when it is closed.
- Every async forwarding layer (_arun_*_streaming_action,
  FunctionBasedStreamingAction, type_eraser, pydantic
  async_action_generator) now closes the generator it wraps.
@github-actions github-actions Bot added area/core Application, State, Graph, Actions area/integrations External integrations (LLMs, frameworks) area/typing Mypy, type hints, pydantic labels Nov 3, 2026
@skrawcz
skrawcz requested review from andreahlert, elijahbenizzy and jernejfrank and a balanced review from Copilot and removed request for Copilot November 3, 2026 07:32

@jernejfrank jernejfrank left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hey, lgtm. Makes sense to me to propagate closing exceptions as well

AsyncStreamingResultContainer.aclose() passed exception=None to the
post-step callback when the stream was closed before completing, so
post_run_step hooks saw an interrupted step as a success and persisters
saved it as "completed" without its writes; resuming then skipped the
action. Report a close-before-completion with GeneratorExit, matching
the previous GC-finalizer behavior.

- widen the callback's exception type to Optional[BaseException]
- tests: hook exception on aclose and on task cancellation, persister
  status + resume, aclose after completion still reports success,
  multi-step AsyncStreamingAction and pydantic streaming paths
- docs: "Stopping a stream early" section and astream_result docstring
@github-actions github-actions Bot added the area/website burr.apache.org website label Nov 3, 2026
Stefan Krawczyk added 2 commits November 3, 2026 08:59
Storing the __aiter__ generator on the container created a container <->
iterator reference cycle (the iterator's closure references the
container). When a caller broke out of the stream and dropped the
container without calling aclose(), the iterator's finalizer -- and so
the post-step callback/hooks/persisters -- was deferred until cyclic GC
instead of running on refcount drop as it did before. Keep a weakref
instead; aclose() still closes a live iterator.
Mirror the async fix on the sync path. Closing Application.stream_iterate()
(or abandoning a StreamingResultContainer) mid-action left the action's
generator suspended until GC, because each forwarding layer used a bare
`for` loop and never closed the generator it read from.

- StreamingResultContainer.close() (new, public): stops the stream without
  draining it, closes the action's generator chain, and runs the post-step
  callback exactly once. A close before completion is reported as a failed
  step (GeneratorExit); after completion it still reports success. Cleanup
  exceptions propagate. The __iter__ generator is held weakly to avoid a
  container <-> iterator reference cycle.
- stream_iterate: on GeneratorExit at its yield, close the in-flight
  container before re-raising.
- _run_single_step_streaming_action / _run_multi_step_streaming_action:
  `finally: generator.close()`.
- pydantic action_generator: wrap the inner generator in
  contextlib.closing(...). (type_eraser and _stream_run_and_update already
  use `yield from`, which propagates close().)
- callback exception typed Optional[BaseException].
- tests mirroring the async ones (close mid-action, persister status +
  resume, multi-step, idempotent close, close after completion, abandoned
  container without cyclic GC, cleanup errors, pydantic); docs cover sync.
@skrawcz skrawcz changed the title fix(core): propagate early close through async streaming actions fix(core): propagate early close through streaming actions Nov 3, 2026
Runnable documentation for stopping a streaming action early (user stop,
moderation/budget check, client disconnect), covering the close API added
in this PR:

- application.py: one demo per pattern -- break + close()/aclose(),
  contextlib.closing/aclosing around stream_iterate/astream_iterate,
  cancelling the consuming task, what persisters record and that resuming
  re-runs the step, and cleanup errors raised by close(). Each demo prints
  a timeline (cleanup vs. post_run_step) so the behaviour is visible.
- server.py: FastAPI SSE endpoint; on client disconnect the action's
  resources are released right away and hooks see CancelledError.
- README.md: the patterns, a "what gets reported" table, and how to tell
  a deliberate stop from a real failure in a hook. Notes that async
  post_run_step hooks/persisters are cancelled on an ASGI disconnect, so
  disconnects are only recorded by sync hooks.
- notebook.ipynb (executed) + statemachine.png, per examples conventions.
- tests/core/test_streaming_early_stop_example.py runs the demos and
  asserts the timelines, so the example can't silently go stale.
- docs: cancellation is reported as CancelledError (not GeneratorExit);
  link to the example. examples/README.md index + pyproject sdist exclude
  (matching streaming-overview).
@github-actions github-actions Bot added area/ci Workflows, build, release scripts area/examples Relates to /examples labels Nov 3, 2026
Make stopping a stream early behave the same however it happens: the
action's cleanup runs first, then the step is reported to hooks and
persisters exactly once, as failed, with no state update -- independent
of timing. What users write is plain Python: sync code just `break`s;
async code `break`s inside `async with contextlib.aclosing(...)` (or
calls `aclose()`).

- `for ... break` (or dropping the iterator) now closes the action before
  the post-step callback runs, instead of reporting while the action's
  resource is still open and leaving cleanup to GC. This also removes the
  container <-> iterator cycle, so the weakref workaround is gone.
- async: `aclose()` racing the event loop's async-generator finalizer
  (after `async for ... break`) waits for that close and gets its outcome,
  so cleanup errors are no longer lost or reported after the hook,
  whatever the number of awaits in between.
- Cleanup errors: hooks receive them; `close()`/`aclose()` raise them,
  also on repeated calls. After a bare `break` there is no caller to
  raise to, so they are logged at warning.
- First error wins: a cleanup error raised while another error is
  already stopping the stream no longer replaces it; it is logged at
  warning (shared `_close_action_generator` helpers in the runners).
- `close()`/`aclose()` while the stream is producing an item (from a
  hook, or from another task) raises a RuntimeError that says to cancel
  the consuming task, instead of recording a bogus failure while the
  stream carries on (and possibly commits).
- Hook/callback types: `exception` is `Optional[BaseException]`
  (GeneratorExit/CancelledError are BaseExceptions).

Tests: the early-stop tests in test_application.py are consolidated
into one section on shared scaffolding (sync + async, single- and
multi-step) and cover the timing race, cleanup errors, first-error-wins,
cancellation and close-while-running. Docs and the streaming-early-stop
example now teach "sync: break; async: aclosing; servers: let
cancellation do it".
@github-actions github-actions Bot added area/hooks Lifecycle hooks, interceptors area/tracking Telemetry, tracing, OpenTelemetry labels Nov 3, 2026
…ehaviour

Async applications shouldn't mix in sync hooks (they block the event loop),
so the async demos and server.py now use async hooks and abuild(), and the
README no longer recommends sync hooks for recording client disconnects.

Starlette/FastAPI cancel a disconnected response through an anyio cancel
scope, which also cancels Burr's await on async post_run_step hooks and
persisters, so the stopped step isn't recorded. This is unchanged from main.
The action is still cleaned up, and resume is unaffected: the last saved step
is the one before, so the interrupted action runs again.

- add cancel_task_demo (task.cancel(): hooks see CancelledError)
- client_disconnect_demo now reproduces the cancel-scope case in-process,
  with an async persister and a resume check
- add a core test pinning the cancel-scope behaviour (cleanup runs, nothing
  saved for the step, resume lands on it)
- add anyio to the tests extra (used directly by the tests and example)
@xihaian251

Copy link
Copy Markdown

Thanks for the implementation. I independently exercised a 35-scenario matrix covering synchronous and asynchronous early termination. I agree that the core close-propagation mechanism is sound, and I recommend merging the PR.

I found two additional edge cases worth documenting and addressing:

1. AnyIO cancellation can interrupt awaited resource cleanup (Medium–High).

I reproduced a case where an async streaming action owns a resource through async with, and its __aexit__ performs an awaited release operation. When the consumer is cancelled through an AnyIO task-group cancel scope, the observed result is:

release_started  = True
release_finished = False

The same behavior reproduces with a bare async generator and no Burr code involved. Plain asyncio.Task.cancel() completes the release, so this is specifically related to AnyIO's level-cancellation semantics.

This makes the new documentation statement that "the action is still cleaned up" too strong. The current cancellation test also does not exercise an awaited cleanup operation.

I suggest clarifying that cleanup may be interrupted when it contains awaits under an active cancellation scope, and adding a regression test that captures this behavior.

2. Repeated close/aclose calls can expose inconsistent exceptions (Medium).

I reproduced the same issue in both synchronous and asynchronous containers.

When both the action cleanup and post_run_step hook raise:

  • First close/aclose raises HookError.
  • Subsequent calls raise CleanupError.

When only the hook raises:

  • First close/aclose raises HookError.
  • The second call returns None, although the step was never recorded.

This appears to result from the callback exception replacing the cleanup exception in the first call, combined with _callback_realized being set before callback execution.

The behavior is inconsistent with the aclose() docstring, which promises that cleanup exceptions are raised again on subsequent calls.

I suggest defining an explicit exception-precedence contract and adding tests for both failure combinations.

Neither finding invalidates the main early-close implementation. I am sharing these as collaborative follow-up findings rather than requesting changes to the core mechanism.

When stopping a stream raised, the error depended on sync vs async and on
which call it was: async let a post_run_step hook error replace the cleanup
error on the first aclose() and then raised the cleanup error afterwards;
a hook error alone was raised once and then lost, although the step had
not been recorded.

Contract, identical for sync and async:
- the first error wins: an error from the action's cleanup (what hooks
  receive), else an error from the post_run_step callback;
- a callback error superseded by a cleanup error is logged, not lost;
- every close()/aclose() raises that same error.

Also pin and document that under an anyio cancel scope (Starlette client
disconnect) an await inside the action's own cleanup is cancelled unless
shielded, instead of claiming the action "is still cleaned up".

Reported by @xihaian251 in #1027.
@skrawcz

skrawcz commented Nov 4, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the thorough matrix, both findings were real.

1. Agreed: under an anyio cancel scope, an await in the action's own cleanup is cancelled (not Burr-specific, as you found). The docs and example no longer say the action "is still cleaned up". They now say cleanup starts and an await in it is cancelled unless wrapped in anyio.CancelScope(shield=True), and the cancel-scope test pins both the unshielded and the shielded case.

2. Fixed in 049ee0a. Sync and async now follow one rule: the first error wins (the cleanup error, else the post_run_step hook error), a hook error superseded by a cleanup error is logged, and every close()/aclose() raises the same error. Both failure combinations are covered, for sync and async, via explicit close and via break then close.

@xihaian251

Copy link
Copy Markdown

Thanks for addressing both findings so thoroughly. I appreciate the clarification around AnyIO cancellation semantics and the consistent error-handling contract. The added tests cover the cases I reported. LGTM from my side!

@skrawcz
skrawcz merged commit 36e3ff3 into main Nov 6, 2026
26 checks passed
@skrawcz
skrawcz deleted the fix/astream-close-propagation branch November 6, 2026 04:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/ci Workflows, build, release scripts area/core Application, State, Graph, Actions area/examples Relates to /examples area/hooks Lifecycle hooks, interceptors area/integrations External integrations (LLMs, frameworks) area/tracking Telemetry, tracing, OpenTelemetry area/typing Mypy, type hints, pydantic area/website burr.apache.org website

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants