Skip to content

fix(controller): await non-coroutine awaitables in local workers - #703

Open
cccccindy-hub1 wants to merge 1 commit into
microsoft:mainfrom
cccccindy-hub1:fix/non-coroutine-awaitables
Open

cccccindy-hub1 wants to merge 1 commit into
microsoft:mainfrom
cccccindy-hub1:fix/non-coroutine-awaitables

Conversation

@cccccindy-hub1

Copy link
Copy Markdown

Summary

_run_local_reconciler_worker detects async results with inspect.isawaitable but passes them straight to asyncio.run, which only accepts real
coroutines. A valid custom awaitable returned by an agent's synchronous run() fails on Python 3.12 with ValueError: a coroutine was expected, so the
local worker exits with code 1 and the rollout never completes.

Closes #666

Problem

On Python 3.12, inspect.isawaitable is broader than asyncio.run: it returns True for any object implementing __await__, while asyncio.run (via
Runner.run -> loop.run_until_complete -> tasks.ensure_future) only accepts genuine coroutines. The mismatch is silent at the type level because
asyncio.run is annotated to accept Coroutine[Any, Any, T] and the result was passed with a # type: ignore[arg-type].

class Deferred:
    def __await__(self):
        async def complete():
            await asyncio.sleep(0)
        return complete().__await__()

class Agent:
    def run(self):
        return Deferred()

On main, the worker prints ValueError: a coroutine was expected and returns 1 without executing the awaitable. Synchronous results and normal coroutine
results already work.

Change

Wrap the awaitable in a small coroutine before handing it to asyncio.run, so the await happens inside a real coroutine. The # type: ignore[arg-type]
comment is no longer needed and was removed.

Tests

Added tests/controller/test_local_reconciler_worker.py with four cases covering the worker entry point:

- synchronous run() result returns 0
- coroutine run() result returns 0
- custom awaitable (__await__) returns 0 and is actually executed (asserted via a completion marker)
- a raising agent returns 1

The tests patch importlib.import_module to return a module carrying the test agent class, so they do not depend on pytest's import mode or the filesystem.

Verification

- Root cause confirmed on Python 3.12: inspect.isawaitable(Deferred()) is True while asyncio.run(Deferred()) raises ValueError: a coroutine was expected.
- Mutation check: reverting the fix makes test_awaitable_result_is_executed fail; the test genuinely covers the bug.
- pytest tests/controller/test_local_reconciler_worker.py -> 4 passed.
- pytest tests/controller/test_local_reconciler.py tests/controller/test_local_reconciler_env.py tests/controller/test_local_reconciler_platform.py
tests/controller/test_local_reconciler_worker.py -> 27 passed.
- ruff check and ruff format --check pass on both touched files.

The change only affects the local worker entry path and does not touch the reconciler loop, Kubernetes controller, or server.
AI-assisted development. Human review requested before merge.
Closes #666

asyncio.run only accepts real coroutines; a custom awaitable returned by a
synchronous run() fails with 'ValueError: a coroutine was expected'. Wrap the
awaitable in a coroutine before handing it to asyncio.run, and drop the now
unneeded type: ignore. Add tests for sync, coroutine, custom-awaitable and
raising agents. Closes microsoft#666
Copilot AI balanced review requested due to automatic review settings November 10, 2026 08:17

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues were identified for the change enabling local workers to handle non-coroutine awaitables.

No open issues 🎉

What changed in this PR

Fixes local worker handling of non-coroutine awaitables, addressing #666. The change allows custom awaitables returned by an agent’s synchronous run() method to execute on Python 3.12.

Changes:

  • Wraps awaitable results in a coroutine before passing them to asyncio.run, removing the argument-type suppression.
  • Adds four regression tests covering synchronous results, coroutines, custom awaitable execution, and agent exceptions.
File Description
tests/​controller/​test_local_reconciler_worker.py Adds worker entry-point tests, including a completion assertion for custom awaitables.
agentlightning/​controller/​local_reconciler.py Adds a coroutine wrapper to await results accepted by inspect.isawaitable.

Review effort: Balanced

@cccccindy-hub1

Copy link
Copy Markdown
Author

@microsoft-github-policy-service agree

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Local workers fail for non-coroutine awaitable run results

2 participants