[morgan-wowk]

Problem

run_loop sleeps sleep_seconds_between_queue_sweeps after every sweep, and each sweep visits at most one running container execution. With hundreds of active container executions, a single pass over the running queue takes minutes, and that time is mostly spent sleeping. A container that finished waits that long before its status lands.

Changes

• process_each_queue_once() now returns whether the next sweep should start without waiting. It returns True when either:
  • the queued handler processed a queued execution, or
  • the running handler processed a container execution, and some PENDING or RUNNING container execution was last refreshed more than _running_execution_refresh_interval ago (new internal kwarg, default 30s). This is one EXISTS query.
• run_loop skips its sleep when the sweep returns True.
• The orchestrator still sleeps between sweeps when it is idle, when everything was refreshed within the interval, and after a queue handler raises.

System impact

• A backlog of running container executions is refreshed back to back until it is caught up. Then the loop returns to its current pace.
• The extra query only runs after a sweep that processed a running container execution.
• Callers that ignore the return value see no change.

Production test

All three orchestrator PRs ran together in production as patches (https://github.com/Shopify/oasis-backend/pull/843). Before/after:

• Completion lag (container ended → status committed): p50 128s → 1.0s, p90 182s → 5.2s.
• Start detection (pod running → RUNNING committed): p50 81–250s → 4.8–14s.
• Container executions never seen as RUNNING before they ended: 34–80% → 0–0.4%.

This PR is the same change built into OrchestratorService_Sql instead of being patched in.

Related PRs

• feat(orchestrator): Prefetch a finished container's log and outputs off the loop thread #427 and feat(orchestrator): Refresh a container execution as soon as its pod or job changes #428 are independent of this one. All three merge cleanly in any order, and the combination passes the test suite.
• chore(orchestrator): Replace per-visit sweep logs with a heartbeat #419 conflicts at the end of process_each_queue_once() (its heartbeat call vs this return). The conflict is trivial: call the heartbeat, then return.

Testing

• pytest tests: 587 passed.
• New tests/test_orchestrator_sweep_pacing.py covers:
  • waiting when there is no work
  • sweeping again after launching a queued execution
  • waiting when everything was refreshed recently
  • sweeping again while a running container execution is overdue
  • waiting after a queue handler raises