[Bug]: Non-cooperative work can block LMI invocation cleanup and worker recovery #726

Problem

Invocation cleanup can wait indefinitely for admitted work to finish. On Lambda Managed Instances (LMI), an invocation timeout does not forcibly terminate the function's code. A blocked invocation can therefore retain a runtime slot or executor worker and continue already-started side effects after its deadline.

The unresolved cases include application code or I/O that ignores interruption, blocked plugin hooks or custom clients, synchronous initial-state reads, and admitted checkpoint/poll transport that does not finish. An executor that runs the root inline, including a saturated CallerRunsPolicy, can block the runtime caller before it reaches a timeout-aware wait.

Requesting cancellation or rejecting new SDK work does not establish a bound on actual cleanup, caller return, or worker reclamation. This issue tracks the supported bound/recovery policy, its implementation, and evidence that the resulting behavior meets that policy.

Current Evidence

Inspected main revision: 1a501400.

• ExecutionManager.awaitTaskExit() waits on task.exit().get() without a timeout and continues waiting after interruption.
• ApiRequestDelayedBatcher.shutdown() joins the actual in-flight transport even when its observer has been aborted.
• The lifecycle documentation explicitly states that non-cooperative code can delay caller return without a hard bound.

These are source findings. Deployed LMI acceptance of the remaining bounds and recovery behavior is still outstanding.

Required Behavior

Define and implement finite budgets for SDK-controlled waits and downstream I/O, and a supported policy for invocation cleanup, caller return, and runtime-slot/worker recovery. Specify when each budget starts, what completion it measures, and which task, client, plugin, and executor configurations can meet it.

Where the invocation lifecycle has started, completion of the SDK call, whether by a response (including PENDING) or by an exception, must preserve actual root/task cleanup and the matching End on its owner thread. The recovery design must reconcile that ordering with the proposed bounds; an early response or exceptional return while admitted work or End is still running cannot satisfy cleanup acceptance.

Arbitrary application code cannot be safely forcibly terminated by the SDK. For work outside the supported cancellation boundary, document which cleanup, caller-return, and worker-recovery guarantees cannot be provided, along with residual activity, retained capacity, diagnostics, and the safe recovery path. Do not rely on process freezing, LMI invocation timeout, or unsafe JVM thread termination. A cancellation request cannot undo side effects already accepted by a downstream service.

Acceptance Criteria

• Define and implement the supported budgets for normal completion, PENDING, and deadline-triggered cleanup. SDK-controlled waits and I/O have finite bounds; non-cooperative task/hook/client and inline-executor cases have explicit handling and documented recovery limits.
• Deterministic local controls exercise a stuck task, checkpoint/poll transport, and caller dispatch blocked by inline execution. Observe the deadline, cancellation, actual task/transport exit, owner End, and caller return separately. Preserve actual cleanup and owner End ordering for both responses and exceptions.
• Deployed LMI tests exercise real invocation timeouts and demonstrate the defined cleanup, caller-return, and worker/runtime-slot recovery bounds for supported configurations. Establish overlapping invocation IDs in the same JVM, keep a healthy peer progressing, and demonstrate restored admission capacity in the original environment. Separately exercise non-cooperative cases to verify documented limits and isolation. Environment replacement, a test-only release, or eventual cleanup does not establish bounded recovery.
• Recovery preserves invocation isolation, caller-owned executor lifetime, replay operation identity, stored successes/failures, checkpoint ordering, and expected side-effect counts. Include an actual suspension/resume path and repeated timeout/recovery cycles that detect retained tasks, queued work, context, and sustained resource growth.
• Attach final deployed evidence through [Feature]: Add LMI cloud end-to-end tests for concurrency, lifecycle and OpenTelemetry #727, including tested commits/artifacts, LMI configuration, environment/request/execution identities, lifecycle ordering, timing, histories, and resource observations. Retain the existing harness acceptance limits. Report unsupported configurations and failed bounds explicitly; local tests, skipped cloud checks, cancellation requests, test escape paths, and documentation of an unresolved limitation alone do not establish bounded-recovery acceptance.

Related Tracking

#727 owns the cloud test suite used to validate this policy. Deadline admission and cooperative-cancellation correctness are tracked in #838; shared/nested bounded-pool progress in #839; invocation-owned executor isolation in #840. This issue retains the cleanup/return bounds and runtime-slot/worker recovery requirements above.

Runtime references: LMI Java runtime and execution-environment lifecycle.