Repository navigation
Follow-up: Qualify security controls with GPU-backed inference and training #39
Description
Scope
Follow-up runtime qualification for security issues #34, #35, #36, and #37 using GPU-backed inference or training. The owner explicitly excluded GPU use from the current worker implementation and test run; the product's local GPU capability is retained.
Qualification plan
- Use an operator-approved disposable GPU environment and fixed images, driver/runtime versions, policy digests, model identifiers, budgets, and resource limits. Never use an unrestricted workstation as a compromised-candidate fixture.
- Verify allowed inference can complete under the intended policy, while candidate access to host GPU devices, supervisor/evaluator assets, signing keys, unrelated files, and unapproved endpoints remains denied.
- Stop a real active GPU-backed model request and, where training is supported, a bounded training job. Verify lease/inference revocation, guest removal, released allocations, no post-stop import or stale completion, and failed teardown reporting.
- Verify evidence survives recorder/worker failures and can be validated against an independently pinned key and final head. Distinguish supervisor observations from worker submissions.
- Execute independent candidate checks without exposing expected answers or verifier keys; preserve protocol/candidate/runtime identities and authenticated results.
- Exercise concurrent cancellation, gateway restart, resource exhaustion, policy changes, and driver/runtime failure. Preserve readable evidence and an incident-style account of failed controls.
Acceptance
Record exact hardware, software versions, commands, resource/budget bounds, results, and exclusions. Attach protected evidence references and publish a redacted summary without credentials or user data. Passing mock gateway, CPU-only, or fake-model tests does not satisfy this issue. GPU testing must be explicitly scheduled with available resources before execution; no GPU tests are authorized as part of the current four implementation workers.
Activity
w4ffl35 commentedon Oct 13, 2026
Blocker status (2026-10-13): mock/CPU tests do not satisfy this issue by definition. Needs an operator-approved disposable GPU environment with fixed images/versions/budgets, explicitly scheduled. Owner excluded GPU use from current workers. Blocked on GPU resources + scheduling.
w4ffl35 commentedon Oct 14, 2026
GPU-backed security qualification report (2026-10-14)
Operator-approved disposable setup: dedicated queue
rsi-train-20261013(maxInFlight 2, lease 1h), private test gateway for restart work; shared queue/workers untouched. GPU use explicitly authorized in-session; airunner daemon (pid 1239751, 1714MB) left running, ~14GB free used.Pinned environment: RTX 5080 16GB, driver 615.71.09, Docker 29.9.0, image
headlesscode-openshell-rsi-training:local@sha256:c5c61ae0…948ea, policy sha9dd1e4a4f1bb63d7, baseQwen/Qwen3-0.6B-Base@da87bfb6, code masterc5e7b1bf(incl. #166/#167/#168). Bounds: 8 training steps seed 1337, 3 fixed eval cells, CPU evals ≤2GiB checkpoints.A. Allowed training completes under policy — PASS. Job
716506fa(TRAINING_GPU): 8/8 steps in 36.8s, adapter 9.2MB sha09904d30…; paired base+adapter evals 3/3 cells each. (Candidate rejected 0-vs-0 at smoke scale, as expected; security path is the evidence.)B. Guest confinement (replica guest: same image + gpu:1 + policy) — PASS. uid 1000, Linux caps all zero,
/devunlistable with no nvidia nodes, yet CUDA works (exactly 1 device — compute allowed, raw devices denied)./home,/opt,/writes: EACCES. TCP egress: EACCES. DNS stubbed (example.com→198.18.0.2). No workspace mounts in bare probe. Notes (not failures):nvidia-smiexposes the host GPU UUID; host CPU/mem counts visible though a 4GiB mem limit is enforced on sandbox containers.C. Stop during GPU training — PASS (fail-closed). Emergency stop mid-lease → 409
failed-to-stop: job cleanup exceeded the 5s grace, BUT guest removal verified and confirmed absent from the gateway, job4dd3d558fenced as leased+cancel-requested (can never complete/import), worker exited fail-closed (worker.ts:487), per-resourcecleanupResultspersisted. Capacity held until lease expiry, then the #165 reaper resolves (proven live on two prior jobs). Observation: 5s grace is short for torch teardown — safe as-is, tuning is optional.D. Concurrent cancellation — PASS. Two simultaneous cancels: one raced a 37s training completion (correct
false), the other hit a leased job with a live owner → clean acknowledge path tocancelledwith pure reason, worker survived and kept heartbeating, no guest residue. Queued-cancel covered by PG integration tests.E. Gateway restart mid-lease — PASS with hygiene incident. Private gateway restarted with a live training guest: worker-failure path requested cancellation (
7d2441f5leased+cancel-requested), worker exited fail-closed, GPU idle afterwards (compute revoked). INCIDENT: sandbox containers survived at docker level (idle: 0.15% CPU, 5MB, no GPU) and post-restart gatewaydeletewas ineffective (already deletedyet still listed) — operatordocker rm -frequired. Controls held; cleanup failed. No result import possible (lease fenced).F. Blind independent eval — PASS (code-verified).
evaluate_adapter.pyfeeds the model onlycell["prompt"]and comparescell["completion"]in-harness; fixed 3-cell set enforced,trust_remote_code=False, workspace-confined, all digests recorded.G. Evidence survival — PASS. Queue rows, run archives, S3 content-addressed artifacts, and control-state files all survived every worker exit (B, C, D).
Exclusions (exact blockers): recorder (PG/S3) failure injection — shared infra, unsafe to kill; GPU OOM exhaustion — would risk the co-resident airrunner daemon; driver/runtime failure injection — same; mid-run policy change — not exercised; evaluator integrity seal still
unsealed(pre-existing, out of scope). Pending confirmation: Run B submitter observes terminal cancellation at the 05:45 reaper (reaper itself already proven twice live).w4ffl35 commentedon Oct 14, 2026
Qualification complete per the report above: confinement, stop, concurrent-cancel, restart, blind eval, and evidence survival all verified live on GPU with pinned versions and exact exclusions. The 05:45 reaper observation for the Run B submitter will be appended when it lands.
w4ffl35 commentedon Oct 14, 2026
Pending confirmation closed: the Run B submitter returned
qlora-39-stop-2: failedvia its own wait path without reaper interaction, and the stopped job4dd3d558was independently reaped tocancelledat its 05:45 lease expiry by the fixed claim path (third live #165 confirmation). Full stop→fence→reap→observe loop verified end to end.