Repository navigation
fix(core): propagate early close through streaming actions - #1027
Conversation
Closing Application.astream_iterate() (or abandoning an AsyncStreamingResultContainer) mid-action left the action's generator chain suspended until GC / loop.shutdown_asyncgens(), so resources held by the action (HTTP/SSE streams, cursors, spans) were released late, in an arbitrary context, with cleanup errors only logged. - Add AsyncStreamingResultContainer.aclose(): closes the action chain without draining it, runs the post-step callback once, idempotent. - astream_iterate closes the in-flight container when it is closed. - Every async forwarding layer (_arun_*_streaming_action, FunctionBasedStreamingAction, type_eraser, pydantic async_action_generator) now closes the generator it wraps.
jernejfrank
left a comment
There was a problem hiding this comment.
hey, lgtm. Makes sense to me to propagate closing exceptions as well
AsyncStreamingResultContainer.aclose() passed exception=None to the post-step callback when the stream was closed before completing, so post_run_step hooks saw an interrupted step as a success and persisters saved it as "completed" without its writes; resuming then skipped the action. Report a close-before-completion with GeneratorExit, matching the previous GC-finalizer behavior. - widen the callback's exception type to Optional[BaseException] - tests: hook exception on aclose and on task cancellation, persister status + resume, aclose after completion still reports success, multi-step AsyncStreamingAction and pydantic streaming paths - docs: "Stopping a stream early" section and astream_result docstring
Storing the __aiter__ generator on the container created a container <-> iterator reference cycle (the iterator's closure references the container). When a caller broke out of the stream and dropped the container without calling aclose(), the iterator's finalizer -- and so the post-step callback/hooks/persisters -- was deferred until cyclic GC instead of running on refcount drop as it did before. Keep a weakref instead; aclose() still closes a live iterator.
Mirror the async fix on the sync path. Closing Application.stream_iterate() (or abandoning a StreamingResultContainer) mid-action left the action's generator suspended until GC, because each forwarding layer used a bare `for` loop and never closed the generator it read from. - StreamingResultContainer.close() (new, public): stops the stream without draining it, closes the action's generator chain, and runs the post-step callback exactly once. A close before completion is reported as a failed step (GeneratorExit); after completion it still reports success. Cleanup exceptions propagate. The __iter__ generator is held weakly to avoid a container <-> iterator reference cycle. - stream_iterate: on GeneratorExit at its yield, close the in-flight container before re-raising. - _run_single_step_streaming_action / _run_multi_step_streaming_action: `finally: generator.close()`. - pydantic action_generator: wrap the inner generator in contextlib.closing(...). (type_eraser and _stream_run_and_update already use `yield from`, which propagates close().) - callback exception typed Optional[BaseException]. - tests mirroring the async ones (close mid-action, persister status + resume, multi-step, idempotent close, close after completion, abandoned container without cyclic GC, cleanup errors, pydantic); docs cover sync.
Runnable documentation for stopping a streaming action early (user stop, moderation/budget check, client disconnect), covering the close API added in this PR: - application.py: one demo per pattern -- break + close()/aclose(), contextlib.closing/aclosing around stream_iterate/astream_iterate, cancelling the consuming task, what persisters record and that resuming re-runs the step, and cleanup errors raised by close(). Each demo prints a timeline (cleanup vs. post_run_step) so the behaviour is visible. - server.py: FastAPI SSE endpoint; on client disconnect the action's resources are released right away and hooks see CancelledError. - README.md: the patterns, a "what gets reported" table, and how to tell a deliberate stop from a real failure in a hook. Notes that async post_run_step hooks/persisters are cancelled on an ASGI disconnect, so disconnects are only recorded by sync hooks. - notebook.ipynb (executed) + statemachine.png, per examples conventions. - tests/core/test_streaming_early_stop_example.py runs the demos and asserts the timelines, so the example can't silently go stale. - docs: cancellation is reported as CancelledError (not GeneratorExit); link to the example. examples/README.md index + pyproject sdist exclude (matching streaming-overview).
Make stopping a stream early behave the same however it happens: the action's cleanup runs first, then the step is reported to hooks and persisters exactly once, as failed, with no state update -- independent of timing. What users write is plain Python: sync code just `break`s; async code `break`s inside `async with contextlib.aclosing(...)` (or calls `aclose()`). - `for ... break` (or dropping the iterator) now closes the action before the post-step callback runs, instead of reporting while the action's resource is still open and leaving cleanup to GC. This also removes the container <-> iterator cycle, so the weakref workaround is gone. - async: `aclose()` racing the event loop's async-generator finalizer (after `async for ... break`) waits for that close and gets its outcome, so cleanup errors are no longer lost or reported after the hook, whatever the number of awaits in between. - Cleanup errors: hooks receive them; `close()`/`aclose()` raise them, also on repeated calls. After a bare `break` there is no caller to raise to, so they are logged at warning. - First error wins: a cleanup error raised while another error is already stopping the stream no longer replaces it; it is logged at warning (shared `_close_action_generator` helpers in the runners). - `close()`/`aclose()` while the stream is producing an item (from a hook, or from another task) raises a RuntimeError that says to cancel the consuming task, instead of recording a bogus failure while the stream carries on (and possibly commits). - Hook/callback types: `exception` is `Optional[BaseException]` (GeneratorExit/CancelledError are BaseExceptions). Tests: the early-stop tests in test_application.py are consolidated into one section on shared scaffolding (sync + async, single- and multi-step) and cover the timing race, cleanup errors, first-error-wins, cancellation and close-while-running. Docs and the streaming-early-stop example now teach "sync: break; async: aclosing; servers: let cancellation do it".
…ehaviour Async applications shouldn't mix in sync hooks (they block the event loop), so the async demos and server.py now use async hooks and abuild(), and the README no longer recommends sync hooks for recording client disconnects. Starlette/FastAPI cancel a disconnected response through an anyio cancel scope, which also cancels Burr's await on async post_run_step hooks and persisters, so the stopped step isn't recorded. This is unchanged from main. The action is still cleaned up, and resume is unaffected: the last saved step is the one before, so the interrupted action runs again. - add cancel_task_demo (task.cancel(): hooks see CancelledError) - client_disconnect_demo now reproduces the cancel-scope case in-process, with an async persister and a resume check - add a core test pinning the cancel-scope behaviour (cleanup runs, nothing saved for the step, resume lands on it) - add anyio to the tests extra (used directly by the tests and example)
|
Thanks for the implementation. I independently exercised a 35-scenario matrix covering synchronous and asynchronous early termination. I agree that the core close-propagation mechanism is sound, and I recommend merging the PR. I found two additional edge cases worth documenting and addressing: 1. AnyIO cancellation can interrupt awaited resource cleanup (Medium–High). I reproduced a case where an async streaming action owns a resource through The same behavior reproduces with a bare async generator and no Burr code involved. Plain This makes the new documentation statement that "the action is still cleaned up" too strong. The current cancellation test also does not exercise an awaited cleanup operation. I suggest clarifying that cleanup may be interrupted when it contains awaits under an active cancellation scope, and adding a regression test that captures this behavior. 2. Repeated close/aclose calls can expose inconsistent exceptions (Medium). I reproduced the same issue in both synchronous and asynchronous containers. When both the action cleanup and
When only the hook raises:
This appears to result from the callback exception replacing the cleanup exception in the first call, combined with The behavior is inconsistent with the I suggest defining an explicit exception-precedence contract and adding tests for both failure combinations. Neither finding invalidates the main early-close implementation. I am sharing these as collaborative follow-up findings rather than requesting changes to the core mechanism. |
When stopping a stream raised, the error depended on sync vs async and on which call it was: async let a post_run_step hook error replace the cleanup error on the first aclose() and then raised the cleanup error afterwards; a hook error alone was raised once and then lost, although the step had not been recorded. Contract, identical for sync and async: - the first error wins: an error from the action's cleanup (what hooks receive), else an error from the post_run_step callback; - a callback error superseded by a cleanup error is logged, not lost; - every close()/aclose() raises that same error. Also pin and document that under an anyio cancel scope (Starlette client disconnect) an await inside the action's own cleanup is cancelled unless shielded, instead of claiming the action "is still cleaned up". Reported by @xihaian251 in #1027.
|
Thanks for the thorough matrix, both findings were real. 1. Agreed: under an anyio cancel scope, an 2. Fixed in 049ee0a. Sync and async now follow one rule: the first error wins (the cleanup error, else the |
|
Thanks for addressing both findings so thoroughly. I appreciate the clarification around AnyIO cancellation semantics and the consistent error-handling contract. The added tests cover the cases I reported. LGTM from my side! |
Stopping a streaming action early (
break, closingstream_iterate/astream_iterate, cancelling the consuming task) leaves the action's generator chain suspended until garbage collection (sync) orloop.shutdown_asyncgens()(async). Anything the action holds open (HTTP/SSE streams, DB cursors, OTel spans,async withblocks) is released late and in an arbitrary context. The cause is that every forwarding layer (container → runner → action wrapper → pydantic wrapper → user generator) iterates with a barefor/async forand never closes the generator it reads from.Behavior changes
mainbreak(sync), or whenaclosing/aclose()/ task cancellation unwinds (async)post_run_stepfor the stopped stepGeneratorExit, late, in an arbitrary contextGeneratorExitwhen closed,CancelledErrorwhen cancelled. Persisters still savefailed; state is not updatedpost_run_stephook error.close()/aclose()raise it, and every repeat call raises the same error; logged at warning after a barebreak. Never replaces an error that was already stopping the streamcontainer.close()/await container.aclose()RuntimeErrortelling you to cancel the consuming task; the stream is left aloneexceptiontype inpost_run_step,post_run_execute_call, tracking clientException/Optional[Exception]Optional[BaseException](annotation only)Unchanged: fully consumed streams, and ASGI client disconnects. On a disconnect, Starlette/FastAPI cancel through an anyio cancel scope, which keeps cancelling every
awaitinside it. The action's cleanup starts, but anawaitinside it is cancelled unless wrapped inanyio.CancelScope(shield=True). Burr's await on asyncpost_run_stephooks and persisters is cancelled too, so the stopped step isn't recorded. Resume still re-runs it, because the last saved step is the one before. Whether Burr should shield post-step hooks is a separate question.Release-notes worthy: cleanup errors now reach hooks, and
close()/aclose()raise errors from cleanup or frompost_run_stephooks.What users do
Plain Python:
break;breakinsideasync with contextlib.aclosing(...), or callawait container.aclose(). Python never closes an async generator at abreak.The docs ("Stopping a stream early" in
streaming-actions.rst) and the newexamples/streaming-early-stopcover each case. The example prints a timeline for each demo and includes a FastAPI server.Implementation
close()/aclose(); iterating closes the action before reporting.stream_iterate/astream_iterate: closing either one closes the in-flight container._run_*/_arun_*runners close the action generator infinally.FunctionBasedStreamingAction,type_eraserand the pydantic wrappers useclosing/aclosing.anyiois added to thetestsextra.Testing
tests/core/test_application.py, covering sync and async, single- and multi-step. It tests every stop path, ordering, exactly-once reporting, persistence and resume, cleanup errors, first-error-wins (cleanup vs. hook errors, the same error on every repeat close), close-while-producing, timing independence, and the anyio cancel scope (with shielded and unshielded awaited cleanup). Againstmain, 30 of these fail.test_burr_pydantic.py), and the example's timelines (intest_streaming_early_stop_example.py).maintoo.curlhang-up.AsyncExitStackworkaround.Checklist