Repository navigation
feat(extras): suspend a session MicroVM after a job - #845
Merged
Merged
Conversation
A SuspendMicrovm retry can run after the MicroVM resumed and freeze the next job. The default client now makes one attempt, and the before-send handler starts no attempt after a resume hook.
A resend of a known job got 503 while the worker refused jobs. The durable function then resumed the MicroVM only to get the duplicate answer. The worker now answers a known callback ID with 202 first.
The 503 note for a resent job said that nothing would suspend the MicroVM again. The idle time restarts after the refusal, so the worker does suspend it again. The real cost is a resume of the MicroVM for nothing. A resume hook can stop the first SuspendMicrovm attempt. Nothing was sent then. The classification docs now say that an earlier attempt, if there was one, may have suspended the MicroVM. The README now states the cost of one attempt. A first-attempt 5xx or dropped connection counts as unknown, and the worker refuses jobs for up to 30 seconds on a MicroVM that did not suspend. It also states why a resent job whose callback is gone gets 202 after the terminate hook.
The JavaScript worker logs an info line when a resume hook arrives during its SuspendMicrovm call. This worker logged nothing until the unknown warning at the end of the call. It now logs the same line, with the same text and the call's microvmId. The resume hook decides under the lock and logs after it. Only the first resume hook during a call logs the line. A resume hook after the call returned logs nothing, because no attempt is left to stop. New listener tests cover a resume hook before the first attempt, a suspend hook after that resume and before the call returns, a second resume hook during the call, and a resume hook after the call returned. The README documents the line. It also states that the JavaScript worker cancels the attempt in flight, while this worker can only stop the next attempt.
…spend-retry-and-duplicates
A job request with "suspendAfter": true asks the worker to suspend its MicroVM when the job ends. The worker answers such a request with the header X-Microvm-Suspend-After: accepted. It suspends only the MicroVM that the run hook named, once no other work runs. The call uses the same path as the suspend when idle.
A session job with suspend_after=True asks the worker to suspend the MicroVM after the job. The next job of the session checks the state and resumes the MicroVM. A job whose delivery gets a 502 with MICROVM_CONNECT_FAILED, or no answer, also checks the state. So a replay that lost the session's flag still resumes the MicroVM.
…after # Conflicts: # packages/aws-durable-execution-sdk-python-extras/tests/microvm/worker_composed_test.py # packages/aws-durable-execution-sdk-python-microvm-worker/README.md # packages/aws-durable-execution-sdk-python-microvm-worker/src/aws_durable_execution_sdk_python_microvm_worker/worker.py # packages/aws-durable-execution-sdk-python-microvm-worker/tests/lifecycle_hooks_test.py
A suspend refusal means that the MicroVM is suspending already. So a suspend request that meets one is done. The suspend hook now clears a pending request, and a suspendAfter job that ends during a refusal sets none. Before, the request fired after the next job, even when that job did not ask for it. Tests cover a job that ends between the suspend and resume hooks, and a request that waits when the suspend hook arrives. The terminate test now asserts that the request is cleared.
A replay of a completed job calls vm.invoke again, so it sets the suspend_after record again. Only a completed map item, parallel branch, or child context skips its function. The suspend_after job itself checks the state, because the record is set before the job starts. A worker that takes more than 10 seconds to start listening gives the first delivery no answer. The operation then makes one GetMicrovm call, so a session that never suspends can make one extra call. The warning about a worker that ignores the field is logged once per session in each execution environment. The README snippet sets heartbeat_timeout on both jobs, as the example does.
The map test with one suspend_after item relies on items that run at once. MapConfig(max_concurrency=3) states that, instead of relying on the default.
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
yaythomas
reviewed
Oct 26, 2026
ResumeMicrovm answers ConflictException while a transition is in progress. The state check marked the resume as sent after that answer. So a MicroVM that was still suspending became SUSPENDED and was never resumed. The wait then ended with MicrovmEndpointUnavailableError and "Last state: SUSPENDED". The check after a sign of a suspended MicroVM had this flaw for a conflict during SUSPENDING. The check before delivery and the check after a gateway error had it for a conflict during SUSPENDED. A conflict now leaves the resume unaccepted. So the next SUSPENDED answer sends ResumeMicrovm again. A PENDING answer after a conflict still waits for RUNNING. The poll interval and the deadline bound the calls.
The simulated endpoint copied the X-Microvm-Suspend-After value from the worker's answer into its own answer. A value from an answer could then reach send_header. The endpoint now sends the header with the fixed value "accepted", and only when the worker answered that value. The session accepts only that value. So the tests still prove that an old worker gets a warning and a current worker does not.
The simulated endpoint sends X-Microvm-Suspend-After only when the worker accepted suspendAfter. No test checked that a current worker's acknowledgement reaches the session. So a lost header would have passed. The suspend-after test now checks that the session logs no warning about an ignored suspendAfter. Without the header, the test fails.
ParidelPooya
marked this pull request as ready for review
October 26, 2026 20:06
ParidelPooya
added a commit
to aws/aws-durable-execution-sdk-js
that referenced
this pull request
Oct 26, 2026
## Summary This PR adds `suspendAfter` to `vm.invoke`. A session job with `suspendAfter: true` asks the worker to suspend its MicroVM when the job ends. The next job of the session resumes the MicroVM. So a session can stop paying for a running MicroVM during a long wait, without auto-suspend. ## Why 1. A session has one way to suspend its MicroVM today: auto-suspend after an idle time. 2. The developer often knows the exact point where the MicroVM can suspend, such as the job before a long callback or wait. 3. A suspend from the durable function would need its own durable step, and it could freeze a job that still runs. 4. So the worker suspends the MicroVM after the job, once no other work runs. The durable function only marks the job. The Python SDK adds the same option in aws/aws-durable-execution-sdk-python#845. ## Changes Worker (`@aws/durable-execution-sdk-js-microvm-worker`): - The job request can carry `"suspendAfter": true`. The worker accepts only JSON `true`. The job document stays version 1, so older workers ignore the field. - The worker answers such a request with the header `X-Microvm-Suspend-After: accepted`. A second delivery gets the header when the first request carried the field. - The worker suspends only the MicroVM that the run hook named, never one from a job request. - When a `suspendAfter` job ends, after its report, the worker records a request. The request fires when no job, report, run hook, or suspend call runs, and the worker is not terminating, closed, or refusing jobs. - The call uses the existing suspend path: the refusal of jobs, one attempt, the abort on a `resume` hook, the outcome classification, and the 30-second grace. - A request is done when any suspend starts, or when a suspend refusal is in effect. A `terminate` hook and `close()` clear it. A `resume` hook never sets it. - Without auto-suspend, a suspend refused for now is not retried. - New log lines: `suspending the MicroVM after a job` and `cannot suspend the MicroVM after a job: the run hook named no MicroVM`. The failed-suspend warning adds `trigger` (`idle` or `job`) as its last field. Extras (`@aws/durable-execution-sdk-js-extras`): - `MicrovmInvokeOptions.suspendAfter?: boolean`. `microvm()` rejects it, and a value that is not a boolean is rejected, both before any durable operation. - The session sets an in-memory flag before the job's child context. From then on, each job checks the state before delivery, and resumes a suspended MicroVM, as in resume mode. - The flag is not rebuilt when a completed `map` item, `parallel` branch, or child context replays. So every session job also has a fallback. A delivery that gets HTTP 502 with `x-aws-proxy-error: MICROVM_CONNECT_FAILED`, or no answer, makes one state check, resumes a suspending or suspended MicroVM, and sends the job again. A resume that the service rejects with a conflict while the MicroVM is still suspending is sent again once the MicroVM is suspended. It runs at most one check per 2 seconds. - A denied `GetMicrovm` in the fallback logs one warning per job, and the delivery keeps retrying. A denied `ResumeMicrovm` while the MicroVM is suspended fails the job at once, and the message names `lambda:ResumeMicrovm`. - A 202 without the acknowledgement header logs one warning per session: the worker is too old, and the MicroVM keeps running. - README sections, TSDoc, and CHANGELOG are updated. The docs say when to use the option, and when not: on back-to-back jobs, on `map` items, and on the last job. ## Design choices Alternatives considered: 1. `vm.suspend(ctx)` from the durable function. It needs its own durable step, and it can freeze a job that still runs. Rejected. 2. A suspend inside `vm.invoke`, before the job's child context checkpoints. A result over 256 KB makes the child context run again on replay, a crash before the checkpoint repeats the suspend, and parallel jobs can still run. Rejected. 3. A session-level option. It applies to every job, but the developer knows which job comes before a long wait. Rejected. 4. A state check before every session job, without the fallback. It costs one `GetMicrovm` call per job, also in sessions that never suspend. Rejected in favor of the fallback, which makes no extra call unless a delivery fails with `MICROVM_CONNECT_FAILED` or gets no answer. What the design keeps out: the job document stays version 1, so older workers keep working. The worker suspends only the MicroVM that the run hook named. The fallback calls `ResumeMicrovm` early when the state is `SUSPENDING`. A live test showed the service accepts that call in every state. A conflict on `SUSPENDING` is handled: the resume is sent again once the MicroVM is `SUSPENDED`. The missing-acknowledgement warning is kept in a per-process set of at most 1,000 session IDs. A session evicted from the set can warn again. The set bounds memory in a long-lived execution environment. Existing test expectations that change: - The failed-suspend warning tests, the JSON-line log test, and the shared warning assertions add `trigger` as the last field. The seven earlier fields and the three messages are unchanged. - The test of a second delivery's acknowledgement now takes the flag from the first request, as the contract states, instead of from the resend. Follow-up work: a JavaScript example with a cloud test, a fix for a `suspendAfter` job that ends before the first run hook arrives, and removal of two fire checks that can no longer matter. ## Testing - Worker unit tests cover each rule: the target, the fire conditions, the re-checks, the clears, the acknowledgement, the logs, and the interaction with auto-suspend and the lifecycle hooks. - Extras unit tests cover the validation, the field, the warning, the flag, the fallback trigger, the rate limit, and each error rule. - Composed tests run the session against the real worker listener: a `suspendAfter` job and then a job that resumes the MicroVM; a `map` with one `suspendAfter` item; a replay that lost the flag and recovers through the fallback; an old worker without the acknowledgement. - Worker `npm test`: 341 tests pass. Extras `npm test`: 440 tests pass. `npm run lint` and `npm run format:check` pass. - The Python SDK's cloud test of the same contract passed against the live service: the MicroVM suspended after the first job, and the next job resumed it and read the file the first job wrote. By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR adds
suspend_aftertovm.invoke. A session job withsuspend_after=Trueasks the worker to suspend its MicroVM when the job ends. The next job of the session resumes the MicroVM. So a session can stop paying for a running MicroVM during a long wait, without auto-suspend.Why
The JavaScript SDK adds the same option in aws/aws-durable-execution-sdk-js#1105.
Changes
Worker (
aws-durable-execution-sdk-python-microvm-worker):"suspendAfter": true. The worker accepts only JSONtrue. The job document stays version 1, so older workers ignore the field.X-Microvm-Suspend-After: accepted. A second delivery gets the header when the first request carried the field.suspend_afterjob ends, after its report, the worker records a request. The request fires when no job, report, run hook, or suspend call runs, and the worker is not terminating, closed, or refusing jobs.resumehook, the outcome classification, and the 30-second grace.terminatehook andclose()clear it. Aresumehook never sets it.suspending the MicroVM after a jobandcannot suspend the MicroVM after a job: the run hook named no MicroVM. The failed-suspend warning addstrigger(idleorjob) as its last field._locked(), and the actions run after the block.Extras (
aws-durable-execution-sdk-python-extras):MicrovmInvokeOptions.suspend_after: bool = False.microvm()rejects it withValidationError, and a value that is not aboolis rejected, both before any durable operation.mapitem,parallelbranch, or child context replays. So every session job also has a fallback. A delivery that gets HTTP 502 withx-aws-proxy-error: MICROVM_CONNECT_FAILED, or no answer, makes one state check, resumes a suspending or suspended MicroVM, and sends the job again. A resume that the service rejects with a conflict while the MicroVM is still suspending is sent again once the MicroVM is suspended. It runs at most one check per 2 seconds.GetMicrovmin the fallback logs one warning per job, and the delivery keeps retrying. A deniedResumeMicrovmwhile the MicroVM is suspended fails the job at once, and the message nameslambda:ResumeMicrovm.mapitems, and on the last job.Examples:
microvm_session_suspend_after.pyand cloud testtest_microvm_session_suspend_after.py. The session has no auto-suspend. The first job writes a file withsuspend_after=True. A callback gate holds the second job until the test sees the MicroVM SUSPENDED. The second job resumes the MicroVM and reads the file.Testing
suspend_afterjob and then a job that resumes the MicroVM; amapwith onesuspend_afteritem; a replay that lost the flag and recovers through the fallback; an old worker without the acknowledgement.hatch run dev-microvm-worker:cov: 519 tests pass,worker.pyat 100%.hatch run dev-extras:cov: 457 tests pass, 100%. Both typechecks,hatch fmt --check,dev-examples:test, andlintcommit.pypass.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.