diff --git a/docs/guides/native-candidate.md b/docs/guides/native-candidate.md index 570e0d175..e2699abb7 100644 --- a/docs/guides/native-candidate.md +++ b/docs/guides/native-candidate.md @@ -389,9 +389,14 @@ keep it open, use `hack ps` and `hack logs` from another terminal, and press Ctrl-C to stop it. For detached startup, use the Compose backend. Foreground image-project `up` prints each startup step (input, runtime, images, -services) and one ready line, `Project is ready (SERVICES)`, after the graph is -ready and its run mapping is published. Ctrl-C prints the stop sequence and -`Native project stopped.` once owned cleanup is confirmed. Until the ready line, +services) and one ready line, `Project is ready (SERVICES; run RUN)`, after the +graph is ready and its run mapping is published. Under it, `up` lists each routed +service's reviewed hostnames as HTTPS URLs (with the `HACK_NATIVE_HTTPS_PORT` +port unless it is 443) and the published loopback ports. Without the native +HTTPS frontend it says that routes are not served on this host. Ctrl-C prints the +stop sequence and, once owned cleanup is confirmed, one stopped line that says +whether named volumes and the stopped run mapping were kept (`Native project +stopped.` when no run mapping was published). Until the ready line, startup holds the runtime's exclusive provider lease, so graph admission cannot interleave with another operation. A `hack ps` from another terminal during that window waits briefly, then reports that the runtime is busy (`runtime_busy`, @@ -414,8 +419,8 @@ Runtime refusals print their code and, when the runtime marks its message as a f literal that cannot carry values, that message and the service it concerns (for example `Service web: Pinned image architecture or implicit volumes are incompatible.`). Formatted runtime messages stay omitted. A start refused before the -runtime reserved any graph state reports that nothing was retained instead of -unconfirmed cleanup. +runtime reserved any graph state reports that nothing was created and no cleanup is +needed, with the startup diagnostic, instead of unconfirmed cleanup. ### Network aliases and the Compose project name @@ -887,10 +892,40 @@ is reattached under its recorded name after the same label and identity checks; newly declared volume is created fresh; a retained volume the new file no longer declares stays owned and unmounted until `hack down --remove-data` (Compose also keeps project volumes). Retained data is only ever reattached to the run and -namespace that created it. Graphs with dependency caches, live source or a pending -source-device rebind keep the narrower rules below. `hack restart` of a stopped -graph refuses and points to `hack up`. -For a shared-source graph admitted with a retained compatibility contract, ordinary +namespace that created it. + +Graphs with `hack.dependencies.*` caches reconcile the same way. A cache's identity +covers only its own inputs: the initializer's service definition and resolved +command, entrypoint, environment and user, the cache declaration, its image, the +declared lockfile and runtime-file contents, and the cache volume layout: every +service that mounts the cache volume, with its target, subpath and `:ro` mode. Any +of those services can write the shared contents, so adding, removing or changing +such a mount is a cache input; mounts stay as declared, as in Compose and v4, and +are not forced read-only. Caches are shared across linked worktrees and branch +instances only while every service other than the initializers mounts them +read-only. A service that mounts the cache read-write can change its contents +through its own image, command, environment or source, which the identity does not +cover, so that cache is scoped to this checkout and branch instance and never +shared. To share it again, mount it with `:ro` in the other services. A run created +by an earlier build whose cache has such a writer moves to its own instance cache +once instead of reusing the shared one. Any other edit to another service, such as an +environment value on the app, keeps the cache. An edit to one of those inputs, +including a lockfile, selects the cache for the new identity as a fresh `up` would, so the initializer installs into it; the old cache +is shared and is only released by this run. Each such `up` publishes the current +source snapshot. A retained named volume is never reused as a dependency cache: +declaring one refuses and names the service and the `hack.dependencies.cache-volume` +field. Caches recorded by earlier builds, whose identity also covered the whole +Compose file, are kept while that file is unchanged and replaced once on the first +edit. Live source, a cache scope carried across a source-device rebind or a pending +initializer cache release keep the narrower rules below: a Compose edit other than +a routed hostname change refuses there with `graph_plan_change_unsupported`, and a +hostname change mixed with other edits refuses with `graph_hostname_change` and the +service name. Every refusal says to revert the edit, or to run +`hack down --remove-data` and then `hack up`. `hack restart` of a stopped graph +refuses and points to `hack up`. +For a shared-source graph that reconciliation does not select (live source, a +carried cache scope, a pending source-device rebind or initializer cache release) +and that was admitted with a retained compatibility contract, ordinary source-content edits may change the reviewed plan ID: restart checks the stable execution, exclusion, mount and dependency-cache inputs before cleanup and again under the provider lease. The original installer publication and graph ownership @@ -1356,7 +1391,11 @@ run mapping before exiting, so `hack ps`, `hack down` and `hack doctor --native-cleanup inspect` select that run instead of reporting the project as not started; nothing is stopped or replayed to do so. When the receipt itself cannot be authenticated, nothing is recorded and the runtime -journal keeps the graph for explicit recovery. +journal keeps the graph for explicit recovery. When the graph owner exited +before reserving any runtime state (for example a configuration refusal during +execution compilation), inspection finds no journal for the run: startup then +reports the failure code and that nothing was created, instead of an +unconfirmed cleanup. Managed values are declared on every authored service, so the reviewed plan is the same for every profile selection, but they are delivered only to the services the @@ -2149,12 +2188,27 @@ must then have a selected binding for each alias it declares. A missing binding refuses with `E_NATIVE_PROJECT_UNSUPPORTED` ("Native host alias has no reviewed host dependency binding; configuration was not dropped."), before any runtime work; with no dependency selection at all it refuses before lifecycle hooks. +The refusal lists every unbound alias with the services that declare it and +where it comes from: the service's Compose `extra_hosts`, `internal.extra_hosts` +in project configuration, this checkout's `.hack/.internal/extra-hosts.json`, or +the main checkout's map that a linked worktree inherits. It then names the +`aliases` a binding needs and how to remove the alias instead +(`hack internal extra-hosts unset ` hides an inherited entry in this +worktree). Listener values are never printed. An entry whose target is not `host-gateway` (for example a fixed address), a name that is not an exact lowercase DNS name, or a service that already maps the name to a different target also refuses. Service-declared names are matched in the -form native review compiles them to (one trailing dot removed, lowercased), so a -service that declares `API.Example.` is covered by a binding for `api.example` and -does not receive a second spelling. Earlier native builds ignored these entries, +form native execution compiles them to: Compose interpolation resolved as the +native runtime resolves it (no ambient environment, so `${UNSET:-word}` selects +`word` and `${UNSET:+word}` selects nothing), then one trailing dot removed and +lowercased. So a service that declares `API.Example.` or +`${UNSET_PREFIX:+x-}api.example` is covered by a binding for `api.example` and +does not receive a second copy of a project-wide `api.example`. A service whose +declarations compile to the same alias twice refuses with `E_CONFIG_INVALID`, +naming the service and alias, before any runtime work; native execution would +refuse it only after the VM started. A name that refers to a required variable, +a Hack-managed value or `COMPOSE_PROJECT_NAME` is left to native review and +execution. Earlier native builds ignored these entries, which let a container resolve the public name instead of the local listener. Native reads these aliases before lifecycle hooks, because they are part of the reviewed configuration; unlike the Compose backend, a hook that changes them makes @@ -3135,6 +3189,27 @@ checks. Named users/groups remain unsupported for private delivery and refuse instead of silently becoming root. An image with no user retains Docker's normal root default; an explicit Compose numeric user takes precedence. +Services with private environment values and services with host dependency +bindings (including every service when project-wide host aliases exist) run their +process through a wrapper. The wrapper runs exactly what Docker would: the image +or Compose `entrypoint` followed by `command` (with no entrypoint, `command` +alone), and a bare executable name such as `docker-entrypoint.sh` or the +`mysqladmin` of `CMD mysqladmin ping` is resolved through the container's own +PATH after the environment is applied. A shell-form health check (`CMD-SHELL` +or a string `test`) runs through the image's shell (`/bin/sh -c` by default), as +Docker runs it. Stock images therefore need no absolute paths. What a wrapper +still cannot run is an empty process, a health test that names no executable, a +named (non-numeric) user, a stop signal other than SIGTERM, SIGINT, SIGQUIT or +SIGKILL, and a volume target at `/`, `/run` or a reserved wrapper path. Native +`up` refuses the Compose-declared cases before the VM starts, and the runtime +checks settings inherited from the pinned image before reserving anything. Both +name every refused service and field at once (for example +`service db user (use a numeric UID or UID:GID)`). `hack doctor` on the native +backend reports the Compose-declared cases in its `native process wrappers` +check, for every service native may wrap. Only profile-active services are +checked by `up` and `restart`; a service behind an unselected profile never runs +and cannot block them, so `doctor` lists such services as information only. + ### Approved-host DNS on pool restart The pinned Smol provider re-resolves approved hosts at each boot and appends the diff --git a/docs/reference/v5-model-coverage.md b/docs/reference/v5-model-coverage.md index 3e5904470..52ded7ae4 100644 --- a/docs/reference/v5-model-coverage.md +++ b/docs/reference/v5-model-coverage.md @@ -40,7 +40,7 @@ must be attached to the eventual exact integration revision. | Foreground Compose startup failing after resources started; unconfirmed cleanup; unauthenticated final inspection | native-startup-owner | `recordFailedStartupOwner` in [native-project-start](../../src/backends/native-project-start.ts) publishes the authenticated failed run with a finalization lifetime; [start regressions](../../tests/native-project-start.test.ts) cover recording, an unauthenticated receipt and a retained mapping | A refused final inspection still leaves only the runtime journal; crash between finalization registration and mapping publication; physical guest acceptance. | | Canceled/expired startup; reader cancellation versus EOF; producer/group uncertainty | pre-ready-retirement, authority-lifetime, native-file-material | Native graph finalizer/onSettled, exact Removed, hard-signal freshness, host stop/close; local pre-Ready retained/removed regressions, including separate live/foreign/hard-canceled fresh-teardown-admission assertions (HACK-1801) | Physical guest acceptance; current root aggregate; explicit executable unknown host-close control; async proof-to-stop and partial binding-retirement crash windows; post-stop source/session replacement or hard cancellation before Retire (Retire does not recheck these inputs; Safe has no post-stop interference predicate). | | Crash between reserve, publish, sync and release; stale recovery | native-frontend-recovery, publication, publication-journal, native-storage-witness | Durable intent/enrolled/released records, fresh lease and expected binding; recovery and sync-failure tests | Fsync guarantees, process-kill windows, real restarted provider observations. No missing-path authority. | -| Data lifetime independent of compute; replaced/aliased metadata; retained-data handoff across Compose plan changes | native-persistent-enrollment, native-storage-witness, native-file-material, native-plan-reconcile | Existing-only enrollment, exact birth/witness and reference fencing; copied-label/replacement tests; `provider/graph/plan_reconcile.rs` projection tests | Real SQL/content retention, external writer assumptions, allocated-byte evidence. | +| Data lifetime independent of compute; replaced/aliased metadata; retained-data handoff across Compose plan changes, including dependency caches | native-persistent-enrollment, native-storage-witness, native-file-material, native-plan-reconcile, native-cache-reconcile | Existing-only enrollment, exact birth/witness and reference fencing; copied-label/replacement tests; `provider/graph/plan_reconcile.rs` projection and cache-selection tests | Real SQL/content retention, external writer assumptions, allocated-byte evidence. | | Settled retained Compose route-attempt retirement; pinned references; interrupted deletion | **Coverage gap:** filesystem journal compaction is not modeled; native-route-stop covers a different stop protocol | Instance mutation lock and exact surviving claim witnesses in [journal compaction](../../src/lib/native-compose-route-claims.ts); [claim regressions](../../tests/native-compose-route-claims.test.ts) and [retained lifecycle integration](../../tests/native-compose-adoption-routing-generation.test.ts) | 24 successful operations keep two attempt directories and constant Linux descriptors; four integrated lifecycle operations check directory counts. Injected unlink failures cover same-store and fresh-store cleanup retries. Assumes cooperative same-user writers, the caller's mutation lock and working filesystem synchronization. No process-kill/power-loss durability proof or bound on unresolved/over-budget histories. | | Graph cleanup container stop batch: bounded in-flight stops, shared worker queue, connect-stage retry, batch-wide unreachable state, deadline expiry, worker panic | **Coverage gap:** no maintained model covers scheduling inside the stop batch; native-route-stop covers a different stop protocol | `dispatch` and `attempt` in [the stop batch](../../packages/runtime-core/src/provider/engine/stop.rs): at most 4 stops in flight, every worker joined; only `connect`/`connect_timeout` failures retry (25 ms doubling to 500 ms, within the batch deadline and 30 s of a stop's first connect failure); once one stop exhausts its window, later connect failures return at once; a deadline reached during backoff keeps the connect stage; `response`, `timeout` and `transport` never retry. Regressions in the same file's `provider::engine::stop` tests (bounded in-flight socket server, late socket, missing socket, 14-stop retry counts, persistent connect failures, oversleeping backoff, non-retried stages, worker panic) and `fourteen_service_connect_failures_keep_resumable_progress_and_retry_only_unstopped` | Tests cover exact attempt counts and the in-flight bound for fixed interleavings; no model explores all worker schedules or proves at most one delivered request per stop. Assumes a failed connect sent no request bytes and that the stop budget covers fully serialized grace periods. Graph-level resume after a partial batch is covered by tests only. Host qualification of the descriptor-pressure root cause (HACK-1302) remains open. | | Concurrent stop/publication; route claim and relay lifetime | native-route-stop, shared-https-lifetime, previous-boot-shared-https, relay-lifecycle-barrier, relay-authorization, relay-staging, relay-fence | Fenced route removal, relay owner, ready/removed protocol, authenticated observations | Native DNS/TLS/browser behavior; old-boot uncertainty and external routing. | @@ -69,6 +69,7 @@ liveness/reachability controls have their own documented assumptions. | Model | Positive distinct states | Configurations | | --- | ---: | ---: | | [native-plan-reconcile](../../tests/models/tla/native-plan-reconcile/Reconcile.tla) | 747,656 | 4 | +| [native-cache-reconcile](../../tests/models/tla/native-cache-reconcile/CacheReconcile.tla) | 2,208 | 5 | | [pre-ready-retirement](../../tests/models/tla/pre-ready-retirement/Retirement.tla) | 10,965 | 8 | | [native-persistent-enrollment](../../tests/models/tla/native-persistent-enrollment/Enrollment.tla) | 38 | 6 | | [native-frontend-recovery](../../tests/models/tla/native-frontend-recovery/Recovery.tla) | 602 | 10 | diff --git a/packages/relay-guest/README.md b/packages/relay-guest/README.md index d3a78ea81..0553b549c 100644 --- a/packages/relay-guest/README.md +++ b/packages/relay-guest/README.md @@ -136,15 +136,18 @@ shutdown and absence of surviving workers or descriptors. The Linux-only gate modes do not read stdin or install signal handlers: ```text -hack-relay-guest --await-release <32 lowercase hex generation> -- [args...] -hack-relay-guest --check-release <32 lowercase hex generation> -- [args...] +hack-relay-guest --await-release <32 lowercase hex generation> -- [args...] +hack-relay-guest --check-release <32 lowercase hex generation> -- [args...] ``` Await mode watches `/run/hack-startup` with inotify before checking for `release`, then waits on events with one anchored 120-second monotonic deadline. Check mode checks once and refuses immediately while the marker is absent. A valid release -executes the supplied absolute executable in place, preserving arguments (including -non-UTF-8 arguments), environment, uid/gid and standard descriptors. Observation +executes the supplied executable in place, preserving arguments (including +non-UTF-8 arguments), environment, uid/gid and standard descriptors. Like Docker, +a name containing `/` runs as given and a bare name (such as +`docker-entrypoint.sh`) is resolved through the container's own PATH; no shell +runs and no host PATH is consulted. Observation FDs close before exec. Graph launch must use Docker `Init=true` so normal init signal forwarding applies while waiting and after replacement. diff --git a/packages/relay-guest/src/barrier.rs b/packages/relay-guest/src/barrier.rs index 7333c9b5c..13211c745 100644 --- a/packages/relay-guest/src/barrier.rs +++ b/packages/relay-guest/src/barrier.rs @@ -1,6 +1,6 @@ //! A public generation fence, not a credential or authentication boundary. use crate::CandidateError; -use std::{ffi::OsString, os::unix::process::CommandExt, path::Path, process::Command}; +use std::{ffi::OsString, os::unix::process::CommandExt, process::Command}; fn refused() -> CandidateError { CandidateError::new("relay_guest_barrier", "Guest startup release refused.") @@ -18,7 +18,14 @@ pub(super) fn valid(args: &[OsString]) -> bool { .all(|b| b.is_ascii_digit() || (b'a'..=b'f').contains(&b)) }) && args[2] == "--" - && Path::new(&args[3]).is_absolute() + && executable(&args[3]) +} +/// Docker accepts an absolute path, a path relative to the working directory, +/// or a bare name it resolves through the container's PATH. `Command` performs +/// that same lookup with this process's (the container's) PATH; no shell runs. +fn executable(name: &OsString) -> bool { + let bytes = name.as_encoded_bytes(); + !bytes.is_empty() && bytes != b"/" && !bytes.contains(&0) } pub(super) fn run(args: &[OsString]) -> Result<(), CandidateError> { if !valid(args) { diff --git a/packages/relay-guest/src/main.rs b/packages/relay-guest/src/main.rs index 147a38f23..9df4262d5 100644 --- a/packages/relay-guest/src/main.rs +++ b/packages/relay-guest/src/main.rs @@ -30,7 +30,7 @@ use std::{ time::Duration, }; -const USAGE: &str = "hack-relay-guest --slot <0..31> --application-fd <3..2147483647>\nhack-relay-guest --slot <0..31> --listen-port <1..65535> [--listen-address ]\nhack-relay-guest --await-release -- [args...]\nhack-relay-guest --check-release -- [args...]\nRelay credential: one private stdin descriptor. Listener address is 127.0.0.1 or 127.0.0.(slot+2).\n"; +const USAGE: &str = "hack-relay-guest --slot <0..31> --application-fd <3..2147483647>\nhack-relay-guest --slot <0..31> --listen-port <1..65535> [--listen-address ]\nhack-relay-guest --await-release -- [args...]\nhack-relay-guest --check-release -- [args...]\nRelay credential: one private stdin descriptor. Listener address is 127.0.0.1 or 127.0.0.(slot+2).\n"; mod options; use options::{Mode, Options, options}; fn credential() -> Result { diff --git a/packages/runtime-core/README.md b/packages/runtime-core/README.md index 45d8d4b74..ebc850501 100644 --- a/packages/runtime-core/README.md +++ b/packages/runtime-core/README.md @@ -1037,8 +1037,16 @@ bound read-only at their targets. A compiled mount or tmpfs target that equals, contains or sits under a member target refuses before effects. Members carry no separate deadline: they live as long as the allocation, which is the container's, and are retired with it by the existing allocation cleanup. A service with values keeps the -launcher contract (numeric user, absolute entrypoint, `CMD` health with an absolute -executable). A files-only service keeps its own process and needs only a numeric image +launcher contract: a numeric user, a non-empty selected process and an exec-form +`CMD` health test. The selected process is the image or Compose `Entrypoint ++ Cmd` +(with no entrypoint, `Cmd` alone), and a `CMD-SHELL` test becomes the exec form +Docker runs (the image `SHELL`, default `/bin/sh -c`). Executables may be bare +names such as `docker-entrypoint.sh`: the launcher resolves them through the +container's PATH after applying the environment, as Docker does. Host-dependency +services get the same materialized process behind their release barrier. Every +wrapped service is checked before any runtime state is reserved; a refusal keeps +its code and lists each service and closed field (`entrypoint`, `healthcheck`, +`user`, `stop_signal`, `mounts`) in `process_refusals`. A files-only service keeps its own process and needs only a numeric image or Compose user. Live-owner restore and normalized serve-restore deliver fresh members into new allocations. `graph exec` and one-off runs refuse envelopes with files; files already in a running container are never re-delivered or renewed. @@ -1493,10 +1501,26 @@ admission would, retained volumes are projected by logical key onto their record names, newly declared volumes receive unused `reserved` names in this run and are created exclusively, and retained volumes the plan no longer declares stay in the receipt. Run, owner, namespace and the ownership plan on every label are unchanged. -`graph source-compatibility` reports `"reconcile": true` for this path. Receipts -with dependency caches, live source or a pending source-device rebind keep the -hostname/shared-source proofs. A volume still `reserved` in a stopped receipt was -never created; selection requires it absent and restore creates it. +`graph source-compatibility` reports `"reconcile": true` for this path. Dependency +caches reconcile too: a cache whose freshly resolved identity (the initializer's own +definition and inputs plus every service mount of the cache volume, not the whole +Compose digest) equals the recorded binding is +kept with its provenance; a changed identity is selected like a fresh admission +(adopted by labels or created) with a freshly published source revision, and the old +shared cache is only released. A cache that a non-initializer service mounts +read-write (`dependency_cache::instance_scoped_volumes`) binds the plan namespace +into its scope, so it is never shared across linked worktrees or branch instances; +such graphs always re-prove the cache through reconciliation, and no earlier +whole-document identity is matched for them. A retained run-owned volume newly declared as a cache +refuses with `graph_plan_reconcile` and a `plan_change` diagnostic naming the service +and field. A receipt recorded under the earlier whole-document identity keeps that +cache while the review still resolves to it (`config::retain_recorded_legacy_caches`). +Receipts with live source, a carried cache scope, pending initializer cache release +or a pending source-device rebind keep the hostname/shared-source proofs; a +non-hostname edit there refuses with `graph_plan_change_unsupported`. A volume still +`reserved` in a stopped receipt was never created; selection requires it absent and +restore creates it, except a reserved dependency cache, which may already exist as a +shared cache. Selection binds the receipt, current VM boot and verified current volume observations. Restoration rechecks these before effects, requires the old owner diff --git a/packages/runtime-core/guest/environment-launcher.zig b/packages/runtime-core/guest/environment-launcher.zig index b63b04e89..64f0008db 100644 --- a/packages/runtime-core/guest/environment-launcher.zig +++ b/packages/runtime-core/guest/environment-launcher.zig @@ -37,16 +37,18 @@ fn freshValues(value: std.json.Value, started: u64, now: u64) !std.json.ObjectMa } return values.object; } -/// Resolve a user exec in this container after applying its environment. Never -/// consult a host PATH or start a shell. Startup and health still require an -/// absolute executable and do not call this function. +/// Exec the selected process in this container after applying its environment, +/// as Docker does: a name containing `/` is executed as given (relative to the +/// working directory), and a bare name is resolved through the container's PATH +/// (Docker's default PATH when none is set). Never consult a host PATH or start +/// a shell. Startup, health and exec all use this. fn execCommand(args: []const [:0]u8, env: *const std.process.EnvMap) !void { if (args.len == 0 or args[0].len == 0) return error.Arguments; const command = args[0]; if (std.mem.indexOfScalar(u8, command, '/') != null) { return std.process.execve(allocator, args, env); } - const path = env.get("PATH") orelse "/usr/local/bin:/bin:/usr/bin"; + const path = env.get("PATH") orelse "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"; if (path.len > 8192) return error.Arguments; const argv = try allocator.alloc([]const u8, args.len); defer allocator.free(argv); @@ -120,7 +122,7 @@ fn run() !void { try std.posix.dup2(null_fd, 2); } const args = if (health or mounted_exec) raw[1..] else raw; - if (args.len < 4 or args.len > 4100 or (!mounted_exec and !std.fs.path.isAbsolute(args[3]))) return error.Arguments; + if (args.len < 4 or args.len > 4100 or args[3].len == 0) return error.Arguments; const payload = try privateFile(args[1], 32768); const expiry = try privateFile(args[2], 32); const deadline = try std.fmt.parseInt(u64, expiry, 10); @@ -135,8 +137,7 @@ fn run() !void { const now = try std.posix.clock_gettime(.BOOTTIME); if (now.sec < 0 or @as(u64, @intCast(now.sec)) >= deadline) return error.Expired; // Successful exec replaces this process: no resident wrapper and no signal forwarding shim. - if (mounted_exec) return execCommand(args[3..], &env); - return std.process.execve(allocator, args[3..], &env); + return execCommand(args[3..], &env); } pub fn main() void { run() catch { diff --git a/packages/runtime-core/src/error.rs b/packages/runtime-core/src/error.rs index 75b0abe24..7b6e92100 100644 --- a/packages/runtime-core/src/error.rs +++ b/packages/runtime-core/src/error.rs @@ -76,6 +76,74 @@ impl From for ErrorMessage { } } +/// The container setting a private process wrapper could not run, named so a +/// refusal points at the Compose or image field to change. A closed vocabulary. +#[derive(Clone, Copy, Debug, Deserialize, Eq, PartialEq, Serialize)] +#[serde(rename_all = "snake_case")] +pub enum ProcessField { + /// The selected process: Compose `entrypoint`/`command` or the image's. + Entrypoint, + /// The selected health check test. + Healthcheck, + /// The selected user (Compose `user` or the image's). + User, + /// Compose `stop_signal`. + StopSignal, + /// A mount target overlapping reserved wrapper paths. + Mounts, +} + +/// Value-free: admitted service names and closed fields only, never argv, +/// paths, users or other configuration values. +#[derive(Clone, Debug, Deserialize, Eq, PartialEq, Serialize)] +#[serde(deny_unknown_fields)] +pub struct ProcessRefusal { + pub service: String, + pub field: ProcessField, +} + +#[derive(Clone, Debug, Deserialize, Eq, PartialEq, Serialize)] +#[serde(deny_unknown_fields)] +pub struct ProcessRefusalsDiagnostic { + pub version: u8, + pub refusals: Vec, +} + +/// Closed vocabulary for the Compose field of a retained-run edit that cannot be +/// applied in place. +#[derive(Clone, Copy, Debug, Deserialize, Eq, PartialEq, Serialize)] +#[serde(rename_all = "snake_case")] +pub enum PlanChangeField { + /// A retained run-owned named volume is now declared as a dependency cache. + DependencyCacheVolume, + /// Routed hostnames changed together with something only reconciliation applies. + Hostnames, +} + +/// Value-free location of a refused retained-run edit. The service is an +/// admitted Compose identifier, never a value, path or engine ID. +#[derive(Clone, Debug, Deserialize, Eq, PartialEq, Serialize)] +#[serde(deny_unknown_fields)] +pub struct PlanChangeDiagnostic { + pub version: u8, + pub service: String, + pub field: PlanChangeField, +} + +/// Value-free refusal locations, boxed together so every +/// `Result<_, CandidateError>` stays small; serialized flat as +/// `process_refusals` and `plan_change`. +#[derive(Debug, Default, Serialize)] +pub struct RefusalDiagnostics { + /// Every service whose selected process a private wrapper refused, found in + /// one pass before any runtime state is reserved. + #[serde(skip_serializing_if = "Option::is_none")] + pub process_refusals: Option, + /// Value-free location of a refused retained-run edit; see [`PlanChangeDiagnostic`]. + #[serde(skip_serializing_if = "Option::is_none")] + pub plan_change: Option, +} + #[derive(Debug, Serialize)] pub struct CandidateError { pub code: &'static str, @@ -89,8 +157,9 @@ pub struct CandidateError { /// validated graph identifier, never an engine ID or a value. #[serde(skip_serializing_if = "Option::is_none")] pub service: Option>, + /// Boxed `str`, not `String`, to keep every `Result<_, CandidateError>` small. #[serde(skip_serializing_if = "Option::is_none")] - pub cause_code: Option, + pub cause_code: Option>, /// Boxed, like `startup_logs`, to keep every `Result<_, CandidateError>` small. #[serde(skip_serializing_if = "Option::is_none")] pub stop_failures: Option>, @@ -98,6 +167,9 @@ pub struct CandidateError { /// [`StartupLogTail`]. Never persist an error carrying it. #[serde(skip_serializing_if = "Option::is_none")] pub startup_logs: Option>, + /// Process-wrapper refusals and plan-change locations; see [`RefusalDiagnostics`]. + #[serde(flatten)] + pub refusals: Option>, } impl CandidateError { @@ -111,6 +183,7 @@ impl CandidateError { cause_code: None, stop_failures: None, startup_logs: None, + refusals: None, } } @@ -126,8 +199,35 @@ impl CandidateError { self } + /// Name the service and field of a refused retained-run edit. + pub fn with_plan_change(mut self, service: &str, field: PlanChangeField) -> Self { + self.refusals.get_or_insert_default().plan_change = Some(PlanChangeDiagnostic { + version: 1, + service: service.to_owned(), + field, + }); + self + } + + /// Name every service and field a private process wrapper refused. + pub fn with_process_refusals(mut self, refusals: Vec) -> Self { + self.refusals.get_or_insert_default().process_refusals = Some(ProcessRefusalsDiagnostic { + version: 1, + refusals, + }); + self + } + + pub fn process_refusals(&self) -> Option<&ProcessRefusalsDiagnostic> { + self.refusals.as_ref()?.process_refusals.as_ref() + } + + pub fn plan_change(&self) -> Option<&PlanChangeDiagnostic> { + self.refusals.as_ref()?.plan_change.as_ref() + } + pub fn with_cause_code(mut self, code: String) -> Self { - self.cause_code = Some(code); + self.cause_code = Some(code.into_boxed_str()); self } } @@ -162,4 +262,25 @@ mod tests { assert!(error.service.is_none(), "{invalid:?}"); } } + + #[test] + fn refusal_diagnostics_serialize_flat_and_stay_small() { + let plain = + serde_json::to_value(CandidateError::new("graph_image", "Fixed text.")).unwrap(); + assert!(plain.get("process_refusals").is_none()); + assert!(plain.get("plan_change").is_none()); + let changed = serde_json::to_value( + CandidateError::new("graph_plan_reconcile", "Fixed text.") + .with_plan_change("deps", super::PlanChangeField::DependencyCacheVolume), + ) + .unwrap(); + assert_eq!( + changed["plan_change"], + serde_json::json!({"version":1,"service":"deps","field":"dependency_cache_volume"}) + ); + assert!(changed.get("process_refusals").is_none()); + // Errors embedded with a boxed slice (stop batches) stay under clippy's + // large-error bound only while the shared error is at most 104 bytes. + assert!(std::mem::size_of::() <= 104); + } } diff --git a/packages/runtime-core/src/provider/environment.rs b/packages/runtime-core/src/provider/environment.rs index 5f565a1f2..9ca05fde2 100644 --- a/packages/runtime-core/src/provider/environment.rs +++ b/packages/runtime-core/src/provider/environment.rs @@ -197,7 +197,7 @@ impl PendingEnvironment { pub(super) fn service(&self) -> &str { &self.service } - /// Whether launcher admission (absolute entrypoint, CMD health) applies. + /// Whether launcher admission (selected process, exec-form CMD health) applies. pub(super) fn wraps_process(&self) -> bool { self.wraps } diff --git a/packages/runtime-core/src/provider/graph/cache_provenance.rs b/packages/runtime-core/src/provider/graph/cache_provenance.rs index 368b8a8cc..8d324e0eb 100644 --- a/packages/runtime-core/src/provider/graph/cache_provenance.rs +++ b/packages/runtime-core/src/provider/graph/cache_provenance.rs @@ -63,6 +63,15 @@ impl CacheProvenance { && bounded(&c.finished_at) }) } + /// Keep only initializers the reconciled graph still declares. The volume + /// identity fence is unchanged; removed services' history is dropped so the + /// receipt never names a container it no longer records. + pub(super) fn retain_initializers(&mut self, mut present: impl FnMut(&str) -> bool) { + self.initializers.retain(|service| present(service)); + let initializers = &self.initializers; + self.completed + .retain(|service, _| initializers.contains(service)); + } pub(super) fn needs_completion(&self, service: &str) -> bool { self.origin == Origin::CreatedHere && self.initializers.contains(service) diff --git a/packages/runtime-core/src/provider/graph/config.rs b/packages/runtime-core/src/provider/graph/config.rs index 9e8323dff..5de673dbe 100644 --- a/packages/runtime-core/src/provider/graph/config.rs +++ b/packages/runtime-core/src/provider/graph/config.rs @@ -19,12 +19,53 @@ pub(super) fn container_base(image: &str, labels: Value) -> Value { pub(super) struct Prepared { pub graph: Graph, pub cache_initializers: BTreeMap, + /// Earlier whole-document identities of this review's cache volumes, keyed by + /// logical volume. Only [`retain_recorded_legacy_caches`] may select one. + pub legacy_caches: BTreeMap, pub configs: BTreeMap, pub probes: BTreeMap, pub resources: BTreeMap, pub namespace: String, pub plan_id: String, } +/// A receipt recorded before cache identities bound only the initializer's own +/// definition keeps its exact cache volume while the review still resolves to +/// that recorded identity under the earlier formula. Nothing else is projected: +/// a changed input yields a different legacy identity and the current one stays. +pub(super) fn retain_recorded_legacy_caches(prepared: &mut Prepared, receipt: &Receipt) { + let mut renamed = BTreeMap::new(); + for (volume, legacy) in &prepared.legacy_caches { + let key = format!("volume:{volume}"); + let Some(resource) = prepared.resources.get_mut(&key) else { + continue; + }; + if resource.cache.as_ref() == Some(legacy) + || receipt + .resources + .get(&key) + .and_then(|old| old.cache.as_ref()) + != Some(legacy) + { + continue; + } + renamed.insert(resource.name.clone(), legacy.name()); + resource.name = legacy.name(); + resource.cache = Some(legacy.clone()); + } + if renamed.is_empty() { + return; + } + for config in prepared.configs.values_mut() { + if let Some(mounts) = config["HostConfig"]["Mounts"].as_array_mut() { + for mount in mounts.iter_mut().filter(|m| m["Type"] == "volume") { + if let Some(name) = mount["Source"].as_str().and_then(|s| renamed.get(s)) { + mount["Source"] = json!(name); + } + } + } + } +} + /// After source replay admission, preserve the original graph ownership identity /// on restored requests. Never change resource names, images, routes or cache keys. pub(super) fn retain_replay_ownership( @@ -47,6 +88,7 @@ fn retain_ownership( receipt: &Receipt, change: Option<&hostname_change::Verified>, ) -> Result<(), CandidateError> { + retain_recorded_legacy_caches(prepared, receipt); let same = change.map_or_else( || same_resource_bindings(&prepared.resources, &receipt.resources), |change| change.matches_resources(&prepared.resources, receipt), @@ -232,7 +274,10 @@ pub(super) fn prepare_delivery( "At most 32 services fit this driver profile; aggregate resource limits also apply.", )); } - let caches = if plan + let dependency_cache::Resolved { + current: caches, + legacy: legacy_caches, + } = if plan .services .values() .any(|service| service.active && service.dependency_cache.is_some()) @@ -247,14 +292,17 @@ pub(super) fn prepare_delivery( || dependency_cache::scope(&plan.source), |scope| scope.scope(&plan.source), )?; - dependency_cache::resolve( + dependency_cache::resolve_with_legacy( plan, source.current_manifest.as_ref().unwrap_or(&source.manifest), &scope, &inputs.services, )? } else { - BTreeMap::new() + dependency_cache::Resolved { + current: BTreeMap::new(), + legacy: BTreeMap::new(), + } }; let publications = loopback_publications(plan, loopback_ports)?; // Restart policies lower to the engine's own policy only for service classes @@ -683,6 +731,7 @@ pub(super) fn prepare_delivery( .collect(); Ok(Prepared { cache_initializers, + legacy_caches, graph, configs, probes, diff --git a/packages/runtime-core/src/provider/graph/dependency_cache.rs b/packages/runtime-core/src/provider/graph/dependency_cache.rs index ca6a44b63..e3bc2b500 100644 --- a/packages/runtime-core/src/provider/graph/dependency_cache.rs +++ b/packages/runtime-core/src/provider/graph/dependency_cache.rs @@ -82,12 +82,71 @@ fn selected(manifest: &ContentRevision, path: &str) -> Result, Can _ => Err(refused()), } } +/// Logical cache volumes that a service other than their initializers mounts +/// read-write. Such a consumer can change the shared contents with inputs the +/// identity does not hash (its image, command, environment or source), so the +/// cache is scoped to this checkout and branch instance (the plan namespace) +/// rather than shared across linked worktrees. Read-only consumers keep sharing. +pub fn instance_scoped_volumes(plan: &PlanData) -> std::collections::BTreeSet { + let active = || plan.services.iter().filter(|(_, s)| s.active); + active() + .filter_map(|(_, s)| s.dependency_cache.as_ref().map(|d| d.volume.clone())) + .filter(|volume| { + active().any(|(_, s)| { + s.dependency_cache + .as_ref() + .is_none_or(|d| &d.volume != volume) + && s.mounts + .iter() + .any(|m| m.kind == "volume" && &m.source == volume && !m.read_only) + }) + }) + .collect() +} + +/// Cache identities for one review: `current` names the volumes a fresh +/// admission selects; `legacy` recomputes the earlier whole-document identity for +/// the same inputs, so a retained receipt recorded under it can still be matched +/// exactly. Neither map grants volume creation, adoption or deletion authority. +pub struct Resolved { + pub current: BTreeMap, + pub legacy: BTreeMap, +} + +#[cfg(test)] pub fn resolve( plan: &PlanData, manifest: &ContentRevision, scope: &str, executable: &BTreeMap, ) -> Result, CandidateError> { + Ok(resolve_with_legacy(plan, manifest, scope, executable)?.current) +} + +fn insert_binding( + result: &mut BTreeMap, + volume: &str, + binding: CacheBinding, +) -> Result<(), CandidateError> { + if result + .get(volume) + .is_some_and(|existing| existing != &binding) + { + return Err(refused()); + } + result.insert(volume.to_owned(), binding); + Ok(()) +} + +/// The current identity binds the initializer's own reviewed service definition +/// and its cache volume declaration instead of the whole Compose document, so an +/// edit to an unrelated service keeps the cache, as Compose keeps a named volume. +pub fn resolve_with_legacy( + plan: &PlanData, + manifest: &ContentRevision, + scope: &str, + executable: &BTreeMap, +) -> Result { manifest.validate().map_err(|_| refused())?; if !hex(scope) || !hex(&plan.compose_sha256) @@ -95,7 +154,14 @@ pub fn resolve( { return Err(refused()); } - let mut result = BTreeMap::new(); + let mut current = BTreeMap::new(); + let mut legacy = BTreeMap::new(); + let scoped = instance_scoped_volumes(plan); + let instance_scope = hash(&( + "hack-dependency-cache-instance-scope-v1", + scope, + &plan.namespace, + ))?; for (name, service) in plan.services.iter().filter(|(_, s)| s.active) { let Some(declaration) = &service.dependency_cache else { continue; @@ -141,10 +207,51 @@ pub fn resolve( return Err(refused()); } } - // Compose digest binds actual argv text (the plan's argv representation is redacted). - // Script content belongs in explicitly declared runtime files; whole source revisions - // would prevent compatible branches with unrelated source changes from sharing. - let base_fingerprint = hash(&( + // Every active service that mounts the cache volume, with its target, + // subpath and access mode. Any of them can write the shared contents, so a + // writer added, removed or changed elsewhere is a cache input, even though + // the rest of that service's definition is not. + let layout = plan + .services + .iter() + .filter(|(_, s)| s.active) + .flat_map(|(name, s)| { + s.mounts + .iter() + .filter(|m| m.kind == "volume" && m.source == declaration.volume) + .map(move |m| (name, &m.target, &m.subpath, m.read_only)) + }) + .collect::>(); + // The plan's argv and literal values are redacted, so the resolved public + // command, entrypoint, environment, user and hosts are bound explicitly. + // Script content belongs in explicitly declared runtime files; whole source + // revisions would prevent compatible branches with unrelated source changes + // from sharing. The legacy identity also bound the whole Compose digest. + let writable = scoped.contains(&declaration.volume); + let current_scope = if writable { + instance_scope.as_str() + } else { + scope + }; + let current_fingerprint = hash(&( + "hack-dependency-cache-v2", + current_scope, + &declaration.volume, + image, + service, + &plan.volumes[&declaration.volume], + &plan.registry, + declaration, + &inputs, + &public.command, + &public.entrypoint, + &public.environment, + &public.user, + &public.extra_hosts, + &service.working_dir, + &layout, + ))?; + let legacy_fingerprint = hash(&( "hack-dependency-cache-v1", scope, &declaration.volume, @@ -160,38 +267,35 @@ pub fn resolve( &public.extra_hosts, &service.working_dir, ))?; - // Include the whole shared-volume layout, not only initializer mounts. - // Keep historical cache keys unchanged when no subpath is requested. - let layout = plan - .services - .iter() - .filter(|(_, s)| s.active) - .flat_map(|(name, s)| { - s.mounts - .iter() - .filter(|m| m.kind == "volume" && m.source == declaration.volume) - .map(move |m| (name, &m.target, &m.subpath, m.read_only)) - }) - .collect::>(); - let fingerprint = if layout.iter().any(|(_, _, subpath, _)| subpath.is_some()) { - hash(&("hack-cache-volume-subpaths-v1", base_fingerprint, layout))? + // The legacy identity reproduces the earlier keys exactly: its Compose + // digest covered the layout, and only subpaths were bound again. + let legacy_fingerprint = if layout.iter().any(|(_, _, subpath, _)| subpath.is_some()) { + hash(&("hack-cache-volume-subpaths-v1", legacy_fingerprint, &layout))? } else { - base_fingerprint + legacy_fingerprint }; - let binding = CacheBinding { + let binding = |scope: &str, fingerprint| CacheBinding { scope: scope.into(), fingerprint, image: image.clone(), }; - if result - .get(&declaration.volume) - .is_some_and(|existing| existing != &binding) - { - return Err(refused()); + insert_binding( + &mut current, + &declaration.volume, + binding(current_scope, current_fingerprint), + )?; + // An earlier shared cache with a writable consumer may hold another + // worktree's writes, so it is never matched again: such a receipt moves to + // its instance-scoped cache once. + if !writable { + insert_binding( + &mut legacy, + &declaration.volume, + binding(scope, legacy_fingerprint), + )?; } - result.insert(declaration.volume.clone(), binding); } - Ok(result) + Ok(Resolved { current, legacy }) } fn owned_directory(path: &Path) -> Result { @@ -694,7 +798,9 @@ mod tests { .arguments[0] .environment_references .push("INSTALLER".into()); - assert_eq!(baseline, resolve(&dynamic, &manifest, &scope).unwrap()); + // The redacted plan form is part of the initializer's own definition. + let dynamic_baseline = resolve(&dynamic, &manifest, &scope).unwrap(); + assert_ne!(baseline, dynamic_baseline); for field in [ "command", "entrypoint", @@ -716,7 +822,7 @@ mod tests { } } assert_ne!( - baseline, + dynamic_baseline, super::resolve(&dynamic, &manifest, &scope, &values).unwrap() ); } @@ -725,9 +831,53 @@ mod tests { changed.services.get_mut("deps").unwrap().image = Some(format!("sha256:{}", "e".repeat(64))); assert_ne!(baseline, resolve(&changed, &manifest, &scope).unwrap()); + // Only the earlier whole-document identity follows unrelated Compose edits. changed = plan.clone(); changed.compose_sha256 = "f".repeat(64); - assert_ne!(baseline, resolve(&changed, &manifest, &scope).unwrap()); + assert_eq!(baseline, resolve(&changed, &manifest, &scope).unwrap()); + let legacy = |plan: &PlanData| { + super::resolve_with_legacy(plan, &manifest, &scope, &executable(plan)) + .unwrap() + .legacy + }; + assert_ne!(legacy(&plan), legacy(&changed)); + assert_ne!(legacy(&plan), baseline); + // Another service, with its own environment, is not a cache input. + let mut app = plan.services["deps"].clone(); + app.dependency_cache = None; + let cache_mount = app.mounts.clone(); + app.mounts.clear(); + changed.services.insert("app".into(), app.clone()); + assert_eq!(baseline, resolve(&changed, &manifest, &scope).unwrap()); + // But a service that can write the shared cache is: adding a read-write + // mount of the cache volume changes the identity, with or without subpaths. + app.mounts = cache_mount; + assert!( + app.mounts + .iter() + .all(|m| !m.read_only && m.subpath.is_none()) + ); + changed.services.insert("app".into(), app.clone()); + let writer = resolve(&changed, &manifest, &scope).unwrap(); + assert_ne!(baseline, writer); + // Its access mode and target are part of the layout too. + let mut reader = app.clone(); + reader.mounts[0].read_only = true; + changed.services.insert("app".into(), reader); + let read_only = resolve(&changed, &manifest, &scope).unwrap(); + assert_ne!(writer, read_only); + assert_ne!(baseline, read_only); + // An environment edit to that writer keeps the identity it now has. + let mut edited = app; + edited.environment.insert( + "MARKER".into(), + crate::project::RedactedText { + environment_references: Vec::new(), + literal_redacted: true, + }, + ); + changed.services.insert("app".into(), edited); + assert_eq!(writer, resolve(&changed, &manifest, &scope).unwrap()); changed = plan.clone(); let mut other = changed.services["deps"].clone(); other.dependency_cache.as_mut().unwrap().bootstrap = false; @@ -741,8 +891,77 @@ mod tests { .as_mut() .unwrap() .bootstrap = true; - assert_eq!(baseline, resolve(&changed, &manifest, &scope).unwrap()); + // Both initializers agree on one identity; the second writer of the shared + // volume is part of it. + let shared = resolve(&changed, &manifest, &scope).unwrap(); + assert_eq!(shared.len(), 1); + assert_ne!(baseline, shared); } + #[test] + fn a_writable_consumer_scopes_the_cache_to_one_instance() { + let (_source, _home, plan, manifest) = prepared(); + let scope = "a".repeat(64); + // Two linked worktrees (or branch instances) share the scope root but + // have their own plan namespaces. + let worktree = |plan: &PlanData, namespace: char| { + let mut plan = plan.clone(); + plan.namespace = namespace.to_string().repeat(64); + plan + }; + let with_consumer = |read_only: bool| { + let mut plan = plan.clone(); + let mut app = plan.services["deps"].clone(); + app.dependency_cache = None; + app.mounts[0].read_only = read_only; + plan.services.insert("app".into(), app); + plan + }; + // Only initializers write: shared across worktrees. + assert!(instance_scoped_volumes(&plan).is_empty()); + assert_eq!( + resolve(&worktree(&plan, 'b'), &manifest, &scope).unwrap(), + resolve(&worktree(&plan, 'c'), &manifest, &scope).unwrap() + ); + // A read-only consumer cannot change the contents: still shared. + let reader = with_consumer(true); + assert!(instance_scoped_volumes(&reader).is_empty()); + let shared = resolve(&worktree(&reader, 'b'), &manifest, &scope).unwrap(); + assert_eq!(shared["deps"].scope, scope); + assert_eq!( + shared, + resolve(&worktree(&reader, 'c'), &manifest, &scope).unwrap() + ); + // A writable consumer in worktree A never shares with worktree B, and has + // no earlier shared identity a receipt could keep. + let writer = with_consumer(false); + assert_eq!( + instance_scoped_volumes(&writer), + BTreeSet::from(["deps".to_owned()]) + ); + let a = super::resolve_with_legacy( + &worktree(&writer, 'b'), + &manifest, + &scope, + &executable(&writer), + ) + .unwrap(); + let b = resolve(&worktree(&writer, 'c'), &manifest, &scope).unwrap(); + assert_ne!(a.current["deps"].scope, scope); + assert_ne!(a.current, b); + assert_ne!(a.current["deps"].name(), b["deps"].name()); + assert!(a.legacy.is_empty()); + // Inside one instance an environment edit to the writer keeps the cache. + let mut edited = worktree(&writer, 'b'); + edited.services.get_mut("app").unwrap().environment.insert( + "MARKER".into(), + crate::project::RedactedText { + environment_references: Vec::new(), + literal_redacted: true, + }, + ); + assert_eq!(a.current, resolve(&edited, &manifest, &scope).unwrap()); + } + #[test] fn missing_lockfiles_and_symlink_inputs_refuse() { let (_source, _home, mut plan, mut manifest) = prepared(); diff --git a/packages/runtime-core/src/provider/graph/foreground.rs b/packages/runtime-core/src/provider/graph/foreground.rs index 1361a3a60..0797bdaa9 100644 --- a/packages/runtime-core/src/provider/graph/foreground.rs +++ b/packages/runtime-core/src/provider/graph/foreground.rs @@ -163,7 +163,11 @@ pub fn restore_selection(candidate: &Candidate, run: &str) -> Result CandidateError { error( @@ -9,6 +10,22 @@ fn refused() -> CandidateError { ) } +/// A hostname change of `service` that is refused, named so the caller can say +/// which routed service and field the edit touched. +fn refused_for(service: &str) -> CandidateError { + refused().with_plan_change(service, PlanChangeField::Hostnames) +} + +/// The review changed, but no routed hostname did. Only receipts that plan +/// reconciliation cannot select reach this proof, so the edit is not applicable +/// in place; reporting it as a hostname change would name the wrong field. +pub(super) fn not_hostname_change() -> CandidateError { + error( + "graph_plan_change_unsupported", + "This retained graph uses live source, a carried cache scope, a pending source-device rebind or a pending initializer cache release; only a routed hostname change can be applied to it in place. Values omitted.", + ) +} + pub(super) struct Verified { pub source: source::Inputs, previous: NormalizedInputIdentity, @@ -23,7 +40,7 @@ pub(super) struct Verified { fn selected_routes( plan: &project::PlanData, receipt: &Receipt, -) -> Result, CandidateError> { +) -> Result<(BTreeMap, String), CandidateError> { let mut names = BTreeSet::new(); let routes = plan .services @@ -46,6 +63,17 @@ fn selected_routes( .map(|route| (resource.key.clone(), route)) }) .collect::>(); + let changed = routes + .iter() + .find(|(service, route)| prior.get(*service) != Some(route)) + .map(|(service, _)| service.clone()) + .or_else(|| { + prior + .keys() + .find(|service| !routes.contains_key(*service)) + .cloned() + }) + .ok_or_else(not_hostname_change)?; if routes.is_empty() || routes.len() != prior.len() || !routes.iter().all(|(service, route)| { @@ -56,13 +84,10 @@ fn selected_routes( .iter() .all(|name| names.insert(name.clone())) }) - || routes - .iter() - .all(|(service, route)| prior.get(service).is_some_and(|old| *old == route)) { - return Err(refused()); + return Err(refused_for(&changed)); } - Ok(routes) + Ok((routes, changed)) } /// Identical normalized input remains on the ordinary strict replay path. @@ -81,18 +106,29 @@ pub(super) fn prepare( if previous == next { return Ok(None); } - let routes = selected_routes(plan, receipt)?; - let binding = receipt.source.as_ref().ok_or_else(refused)?; + let (routes, changed) = selected_routes(plan, receipt)?; + let binding = receipt + .source + .as_ref() + .ok_or_else(|| refused_for(&changed))?; let cached = plan .services .values() .any(|service| service.active && service.dependency_cache.is_some()); + // Anything besides the hostnames changed too: name the routed service. let source = source::prepare_shared_hostnames( engine, plan, receipt, cached.then_some(binding.revision.as_str()), - )?; + ) + .map_err(|failure| { + if failure.code == "live_source_contract" { + refused_for(&changed) + } else { + failure + } + })?; super::super::publication::check_hostname_change( engine.guest().candidate(), &receipt.owner, @@ -357,4 +393,32 @@ pub(super) mod tests { assert!(proof.apply(&mut stale).is_err()); } } + + #[test] + fn a_review_without_a_hostname_change_is_not_reported_as_one() { + let (fixture, mut receipt, _) = transition(); + let plan = project::plan( + &Candidate::discover(&fixture.0).unwrap(), + project::PlanOptions { + branch: None, + project: &fixture.0.join("project"), + compose_file: Path::new("compose.yaml"), + profiles: &[], + }, + ) + .unwrap() + .plan; + // The routed service lost its route: a real hostname change, named. + let failure = selected_routes(&plan, &receipt).unwrap_err(); + assert_eq!(failure.code, "graph_hostname_change"); + let change = failure.plan_change().unwrap(); + assert_eq!(change.service, "web"); + assert_eq!(change.field, PlanChangeField::Hostnames); + // No route before or after: some other edit, which only reconciliation + // applies. The accurate code says so instead of naming hostnames. + receipt.resources.get_mut("container:web").unwrap().routing = None; + let failure = selected_routes(&plan, &receipt).unwrap_err(); + assert_eq!(failure.code, "graph_plan_change_unsupported"); + assert!(failure.plan_change().is_none()); + } } diff --git a/packages/runtime-core/src/provider/graph/image_process.rs b/packages/runtime-core/src/provider/graph/image_process.rs index 4e9fd6c25..31c10fdb4 100644 --- a/packages/runtime-core/src/provider/graph/image_process.rs +++ b/packages/runtime-core/src/provider/graph/image_process.rs @@ -1,8 +1,12 @@ //! Process defaults from an already verified immutable image, before wrappers. //! Compose null inherits; explicit entrypoint (including empty) suppresses image -//! CMD. No shell tokenization, PATH lookup or image selection occurs here. +//! CMD. No shell tokenization, PATH lookup or image selection occurs here: a +//! relative executable name stays relative and the wrapper resolves it through +//! the container's own PATH at exec time, as Docker does. //! See https://docs.docker.com/reference/compose-file/services/#entrypoint. +use super::process_refusals::Refused; use super::{CandidateError, error}; +use crate::error::ProcessField; use serde_json::{Value, json}; const MAX_ARGUMENTS: usize = 4096; @@ -78,47 +82,122 @@ pub(super) fn resolve(image: &Value, create: &Value) -> Result Result { +fn selected_user(image: &Value, create: &Value) -> Result { fn nonempty(value: Option<&Value>) -> Option<&Value> { value.filter(|value| !value.is_null() && value.as_str() != Some("")) } let user = nonempty(create.get("User")).or_else(|| nonempty(image.get("User"))); let user = match user { None => "", - Some(value) => value.as_str().ok_or_else(malformed)?, + Some(value) => value.as_str().ok_or_else(|| Refused { + field: ProcessField::User, + error: malformed(), + })?, }; - super::launcher::identity(&json!({"User":user}))?; + super::launcher::identity(&json!({"User":user})).map_err(|error| Refused { + field: ProcessField::User, + error, + })?; Ok(user.into()) } + +/// Docker runs a `CMD-SHELL` health check as the image's `SHELL` (default +/// `/bin/sh -c`) followed by the test text. Express exactly that as an exec-form +/// `CMD` test, which a wrapper can then prefix. Other tests are unchanged. +fn exec_form_health(create: &Value, image: &Value) -> Result, Refused> { + let refused = || { + Refused::new( + ProcessField::Healthcheck, + "environment_health", + "Private delivery requires a bounded health check test; values omitted.", + ) + }; + let Some(test) = create["Healthcheck"]["Test"].as_array() else { + return Ok(None); + }; + if test.first().and_then(Value::as_str) != Some("CMD-SHELL") { + return Ok(None); + } + let [_, Value::String(command)] = test.as_slice() else { + return Err(refused()); + }; + if command.is_empty() || command.contains('\0') || command.len() > MAX_BYTES { + return Err(refused()); + } + let shell: Vec = match image.get("Shell") { + None | Some(Value::Null) => vec!["/bin/sh".into(), "-c".into()], + Some(Value::Array(values)) + if !values.is_empty() + && values.len() <= 16 + && values.iter().all(|value| { + value + .as_str() + .is_some_and(|text| !text.is_empty() && !text.contains('\0')) + }) => + { + values + .iter() + .filter_map(|value| value.as_str().map(str::to_owned)) + .collect() + } + Some(_) => return Err(refused()), + }; + let mut exec = vec![json!("CMD")]; + exec.extend(shell.into_iter().map(Value::String)); + exec.push(json!(command)); + Ok(Some(json!(exec))) +} /// Materialize exactly the selected process for a private environment/dependency -/// wrapper. All validation precedes mutation. Relative executable names remain -/// unsupported here; private launch must never add ambient PATH resolution. -/// An empty entrypoint is preserved by `resolve` but refused by this adapter, -/// even when Cmd alone names an absolute executable (a separate argv-only gate). -pub(super) fn apply_private(create: &mut Value, image: &Value) -> Result<(), CandidateError> { - let process = resolve(image, create)?; - if !process - .entrypoint - .first() - .is_some_and(|name| name.starts_with('/') && name != "/") - { - return Err(error( +/// wrapper. All validation precedes mutation. +/// +/// Docker runs `Entrypoint ++ Cmd`; with no entrypoint, `Cmd` alone is the +/// process, so it becomes the materialized entrypoint (the same argv). An +/// executable name without `/` (such as `docker-entrypoint.sh`) is kept as is: +/// the wrapper resolves it through the container's PATH at exec time exactly as +/// Docker does, never through a host or wrapper PATH. A `CMD-SHELL` health check +/// becomes the equivalent exec-form test. Only an empty process refuses. +pub(super) fn apply_private(create: &mut Value, image: &Value) -> Result<(), Refused> { + let process = resolve(image, create).map_err(|error| Refused { + field: ProcessField::Entrypoint, + error, + })?; + let (entrypoint, cmd) = if process.entrypoint.is_empty() { + (process.cmd, Vec::new()) + } else { + (process.entrypoint, process.cmd) + }; + if !entrypoint.first().is_some_and(|name| executable(name)) { + return Err(Refused::new( + ProcessField::Entrypoint, "environment_entrypoint", - "Private delivery requires a pinned absolute entrypoint; values omitted.", + "Private delivery requires an entrypoint or command that names an executable; values omitted.", )); } let user = selected_user(image, create)?; + let health = exec_form_health(create, image)?; create["User"] = json!(user); - create["Entrypoint"] = json!(process.entrypoint); - create["Cmd"] = json!(process.cmd); + create["Entrypoint"] = json!(entrypoint); + create["Cmd"] = json!(cmd); + if let Some(test) = health { + create["Healthcheck"]["Test"] = test; + } Ok(()) } +/// An exec target Docker would accept: absolute, relative to the working +/// directory, or a bare name resolved through the container's PATH. +pub(super) fn executable(name: &str) -> bool { + !name.is_empty() && name != "/" && !name.contains('\0') +} + /// Files-only private delivery keeps the selected process unchanged and only pins the /// numeric owner that staged mode-0400 members must match. Validation precedes mutation. -pub(super) fn apply_file_owner(create: &mut Value, image: &Value) -> Result<(), CandidateError> { +pub(super) fn apply_file_owner(create: &mut Value, image: &Value) -> Result<(), Refused> { if !image.is_object() || !create.is_object() { - return Err(malformed()); + return Err(Refused { + field: ProcessField::User, + error: malformed(), + }); } let user = selected_user(image, create)?; create["User"] = json!(user); @@ -219,23 +298,98 @@ mod tests { } #[test] - fn refuses_path_lookup_or_empty_entry_without_partial_mutation() { - for image in [ - json!({"Entrypoint":["relative"],"Cmd":[]}), - json!({"Entrypoint":null,"Cmd":["/absolute"]}), + fn relative_stock_entrypoints_stay_relative_for_container_path_lookup() { + // A stock image such as `ENTRYPOINT ["docker-entrypoint.sh"]` keeps its + // exact argv; the wrapper resolves the name through the container PATH. + let image = json!({"Entrypoint":["docker-entrypoint.sh"],"Cmd":["mysqld"]}); + let mut create = json!({}); + apply_private(&mut create, &image).unwrap(); + assert_eq!(create["Entrypoint"], json!(["docker-entrypoint.sh"])); + assert_eq!(create["Cmd"], json!(["mysqld"])); + let mut relative = json!({"Entrypoint":["./bin/start"],"Cmd":null}); + apply_private(&mut relative, &json!({})).unwrap(); + assert_eq!(relative["Entrypoint"], json!(["./bin/start"])); + } + + #[test] + fn command_only_process_is_the_same_argv_as_docker_runs() { + for (image, create, argv) in [ + ( + json!({"Cmd":["python3","-m","http.server"]}), + json!({}), + json!(["python3", "-m", "http.server"]), + ), + ( + json!({"Entrypoint":["/image"]}), + json!({"Entrypoint":[],"Cmd":["/explicit","a"]}), + json!(["/explicit", "a"]), + ), + ( + json!({"Entrypoint":null,"Cmd":["/absolute"]}), + json!({}), + json!(["/absolute"]), + ), ] { - let mut create = json!({}); + let mut create = create; + apply_private(&mut create, &image).unwrap(); + assert_eq!(create["Entrypoint"], argv); + assert_eq!(create["Cmd"], json!([])); + } + } + + #[test] + fn refuses_an_empty_process_without_partial_mutation() { + for (image, create) in [ + (json!({"Entrypoint":null,"Cmd":null}), json!({})), + (json!({"Entrypoint":["/image"]}), json!({"Entrypoint":[]})), + (json!({"Entrypoint":[""],"Cmd":["x"]}), json!({})), + (json!({"Entrypoint":["/"]}), json!({})), + ] { + let mut create = create; let before = create.clone(); - assert_eq!( - apply_private(&mut create, &image).unwrap_err().code, - "environment_entrypoint" - ); + let refused = apply_private(&mut create, &image).unwrap_err(); + assert_eq!(refused.error.code, "environment_entrypoint"); + assert_eq!(refused.field, ProcessField::Entrypoint); + assert_eq!(create, before); + } + } + + #[test] + fn shell_form_health_becomes_the_exec_form_docker_runs() { + let image = json!({"Entrypoint":["docker-entrypoint.sh"]}); + let mut create = json!({"Healthcheck":{"Test":["CMD-SHELL","mysqladmin ping -h localhost"],"Retries":3}}); + apply_private(&mut create, &image).unwrap(); + assert_eq!( + create["Healthcheck"], + json!({"Test":["CMD","/bin/sh","-c","mysqladmin ping -h localhost"],"Retries":3}) + ); + let shell = json!({"Entrypoint":["/e"],"Shell":["/bin/bash","-o","pipefail","-c"]}); + let mut create = json!({"Healthcheck":{"Test":["CMD-SHELL","true"]}}); + apply_private(&mut create, &shell).unwrap(); + assert_eq!( + create["Healthcheck"]["Test"], + json!(["CMD", "/bin/bash", "-o", "pipefail", "-c", "true"]) + ); + // Exec-form tests, including relative names, are unchanged. + let mut create = json!({"Healthcheck":{"Test":["CMD","mysqladmin","ping"]}}); + apply_private(&mut create, &image).unwrap(); + assert_eq!( + create["Healthcheck"]["Test"], + json!(["CMD", "mysqladmin", "ping"]) + ); + for test in [ + json!(["CMD-SHELL"]), + json!(["CMD-SHELL", ""]), + json!(["CMD-SHELL", "a", "b"]), + ] { + let mut create = json!({"Healthcheck":{"Test":test}}); + let before = create.clone(); + let refused = apply_private(&mut create, &image).unwrap_err(); + assert_eq!(refused.field, ProcessField::Healthcheck); assert_eq!(create, before); } - let mut create = json!({"Entrypoint":[],"Cmd":["/explicit"]}); - let before = create.clone(); - assert!(apply_private(&mut create, &json!({"Entrypoint":["/image"]})).is_err()); - assert_eq!(create, before); + let mut create = json!({"Healthcheck":{"Test":["CMD-SHELL","true"]}}); + assert!(apply_private(&mut create, &json!({"Entrypoint":["/e"],"Shell":"sh"})).is_err()); } #[test] diff --git a/packages/runtime-core/src/provider/graph/launcher.rs b/packages/runtime-core/src/provider/graph/launcher.rs index 9d6a5c104..1baff55c7 100644 --- a/packages/runtime-core/src/provider/graph/launcher.rs +++ b/packages/runtime-core/src/provider/graph/launcher.rs @@ -1,5 +1,7 @@ //! Feature-gated static launcher; no resident wrapper or runtime credential provider. +use super::process_refusals::Refused; use super::*; +use crate::error::ProcessField; #[cfg(feature = "environment-launcher")] use base64::Engine as _; #[cfg(feature = "environment-launcher")] @@ -99,25 +101,40 @@ pub(super) fn identity(config: &Value) -> Result<(u32, u32), CandidateError> { Ok((parse(uid)?, parse(gid)?)) } -pub(super) fn validate(config: &Value) -> Result<(), CandidateError> { - identity(config)?; - let entry = config["Entrypoint"] +/// Wrapper admission for one materialized process (see +/// `image_process::apply_private`). Executables may be absolute, relative to +/// the working directory, or bare names: the wrapper resolves a bare name +/// through the container's PATH at exec time, as Docker does. Each refusal +/// names the field it concerns. +pub(super) fn validate(config: &Value) -> Result<(), Refused> { + identity(config).map_err(|error| Refused { + field: ProcessField::User, + error, + })?; + let executable = config["Entrypoint"] .as_array() - .filter(|a| !a.is_empty()) - .ok_or_else(|| { - error( - "environment_entrypoint", - "Environment delivery requires an explicit absolute entrypoint.", - ) - })?; - if !entry[0].as_str().is_some_and(|s| s.starts_with('/')) - || config["StopSignal"] - .as_str() - .is_some_and(|s| !["SIGTERM", "SIGINT", "SIGQUIT", "SIGKILL"].contains(&s)) + .and_then(|entry| entry.first()) + .and_then(Value::as_str) + .is_some_and(super::image_process::executable); + if !executable + || !config["Entrypoint"] + .as_array() + .is_some_and(|entry| entry.iter().all(Value::is_string)) { - return Err(error( + return Err(Refused::new( + ProcessField::Entrypoint, "environment_entrypoint", - "Environment delivery requires an absolute entrypoint ; unsupported stop signals remain gated.", + "Environment delivery requires an explicit entrypoint that names an executable.", + )); + } + if config["StopSignal"] + .as_str() + .is_some_and(|s| !["SIGTERM", "SIGINT", "SIGQUIT", "SIGKILL"].contains(&s)) + { + return Err(Refused::new( + ProcessField::StopSignal, + "environment_entrypoint", + "Environment delivery supports only SIGTERM, SIGINT, SIGQUIT or SIGKILL stop signals.", )); } validate_health(config)?; @@ -128,7 +145,8 @@ pub(super) fn validate(config: &Value) -> Result<(), CandidateError> { { let target = mount["Target"].as_str().unwrap_or(""); if ["/run", "/"].contains(&target) || target.starts_with("/run/hack-environment") { - return Err(error( + return Err(Refused::new( + ProcessField::Mounts, "environment_mount", "A mount overlaps reserved environment paths.", )); @@ -136,12 +154,13 @@ pub(super) fn validate(config: &Value) -> Result<(), CandidateError> { } Ok(()) } -fn validate_health(config: &Value) -> Result<(), CandidateError> { +fn validate_health(config: &Value) -> Result<(), Refused> { let Some(health) = config.get("Healthcheck") else { return Ok(()); }; let test = health["Test"].as_array().ok_or_else(|| { - error( + Refused::new( + ProcessField::Healthcheck, "environment_health", "An explicit CMD health check is required.", ) @@ -151,18 +170,21 @@ fn validate_health(config: &Value) -> Result<(), CandidateError> { } if test.len() < 2 || test[0] != "CMD" - || !test[1].as_str().is_some_and(|s| s.starts_with('/')) + || !test[1] + .as_str() + .is_some_and(super::image_process::executable) || !test.iter().all(Value::is_string) { - return Err(error( + return Err(Refused::new( + ProcessField::Healthcheck, "environment_health", - "Environment health delivery requires CMD with an absolute executable.", + "Environment health delivery requires an exec-form CMD test that names an executable.", )); } Ok(()) } pub(super) fn attach(config: &mut Value, path: &str, launcher: &str) -> Result<(), CandidateError> { - validate(config)?; + validate(config).map_err(Refused::into_error)?; // Preserve Docker's numeric UID-only group lookup and supplemental groups. identity(config)?; if config["Healthcheck"]["Test"][0] == "CMD" { @@ -339,16 +361,50 @@ mod tests { assert_eq!(value["Healthcheck"]["Interval"], 1000000000); assert_eq!(value["Healthcheck"]["Timeout"], 2000000000); assert_eq!(value["Healthcheck"]["Retries"], 3); + // apply_private turns CMD-SHELL into the exec form Docker runs first. for test in [ json!(["CMD-SHELL", "echo ok"]), json!(["CMD"]), json!(["CMD", "/bin/check", 5]), + json!(["CMD", ""]), ] { value["Healthcheck"]["Test"] = test; - assert!(validate(&value).is_err()); + let refused = validate(&value).unwrap_err(); + assert_eq!(refused.field, ProcessField::Healthcheck); + assert_eq!(refused.error.code, "environment_health"); } } #[test] + fn relative_entrypoint_and_health_names_are_kept_for_container_path_lookup() { + let mut value = config(); + value["Entrypoint"] = json!(["docker-entrypoint.sh"]); + value["Cmd"] = json!(["mysqld"]); + value["Healthcheck"] = json!({"Test":["CMD","mysqladmin","ping"]}); + attach(&mut value, "/run/slot/values.json", "/storage/launcher").unwrap(); + assert_eq!( + value["Entrypoint"], + json!([ + "/run/hack-environment-launcher", + "/run/hack-environment.json", + "/run/hack-environment.expires", + "docker-entrypoint.sh", + "mysqld" + ]) + ); + assert_eq!( + value["Healthcheck"]["Test"], + json!([ + "CMD", + "/run/hack-environment-launcher", + "--health", + "/run/hack-environment.json", + "/run/hack-environment.expires", + "mysqladmin", + "ping" + ]) + ); + } + #[test] fn uid_only_keeps_engine_group_resolution_while_private_files_use_owner_uid() { let mut value = config(); value["User"] = json!("1001"); @@ -429,12 +485,34 @@ mod tests { value["HostConfig"]["Mounts"] = json!([{"Target":target}]); assert!(validate(&value).is_err()); } + for (field, entry) in [ + (ProcessField::Entrypoint, json!([])), + (ProcessField::Entrypoint, json!([""])), + (ProcessField::Entrypoint, json!(["/"])), + (ProcessField::Entrypoint, json!(["app", 5])), + (ProcessField::Entrypoint, json!(null)), + ] { + let mut value = config(); + value["Entrypoint"] = entry; + assert_eq!(validate(&value).unwrap_err().field, field); + } let mut value = config(); - value["Healthcheck"] = json!({"Test":["CMD","true"]}); - assert!(validate(&value).is_err()); + value["StopSignal"] = json!("SIGUSR1"); + assert_eq!( + validate(&value).unwrap_err().field, + ProcessField::StopSignal + ); + let mut value = config(); + value["User"] = json!("bun"); + assert_eq!(validate(&value).unwrap_err().field, ProcessField::User); + let mut value = config(); + value["HostConfig"]["Mounts"] = json!([{"Target":"/run"}]); + assert_eq!(validate(&value).unwrap_err().field, ProcessField::Mounts); + // Relative names are accepted for container PATH lookup. let mut value = config(); value["Entrypoint"] = json!(["app"]); - assert!(validate(&value).is_err()); + value["Healthcheck"] = json!({"Test":["CMD","true"]}); + assert!(validate(&value).is_ok()); } } diff --git a/packages/runtime-core/src/provider/graph/mod.rs b/packages/runtime-core/src/provider/graph/mod.rs index c5a3faba4..1fb72729c 100644 --- a/packages/runtime-core/src/provider/graph/mod.rs +++ b/packages/runtime-core/src/provider/graph/mod.rs @@ -124,6 +124,7 @@ pub use endpoints::GuestEndpoint; mod environment_crash_test; mod launcher; mod probes; +mod process_refusals; #[cfg(all(test, feature = "environment-launcher"))] mod startup_test; pub use environment::stage_environment; @@ -1371,17 +1372,44 @@ pub(super) fn validate_private_delivery( configs: &BTreeMap, environments: &BTreeMap, ) -> Result<(), CandidateError> { + let mut refusals = process_refusals::Collector::default(); for (name, pending) in environments { let config = &configs[name]; if pending.wraps_process() { - launcher::validate(config)?; + refusals.check(name, launcher::validate(config)); } else { - launcher::identity(config)?; + refusals.check( + name, + launcher::identity(config) + .map(|_| ()) + .map_err(|error| process_refusals::Refused { + field: crate::error::ProcessField::User, + error, + }), + ); } - launcher::validate_files(config, &pending.file_targets())?; + } + refusals.finish()?; + for (name, pending) in environments { + launcher::validate_files(&configs[name], &pending.file_targets())?; } Ok(()) } + +/// Wrapper admission for host-dependency services, before any runtime state is +/// reserved: their release barrier runs the same materialized process. +pub(super) fn validate_dependency_processes( + configs: &BTreeMap, + services: &BTreeSet, +) -> Result<(), CandidateError> { + let mut refusals = process_refusals::Collector::default(); + for name in services { + if let Some(config) = configs.get(name) { + refusals.check(name, launcher::validate(config)); + } + } + refusals.finish() +} /// Services whose process the private launcher wraps, and files-only services whose /// image user must still be materialized for exact file ownership. pub(super) fn private_services( @@ -1495,8 +1523,17 @@ fn run_inputs( super::environment::preflight_capacity(engine.guest(), environments.len())?; } // Resolve inherited process defaults only from the exact verified image, - // before the private launcher takes ownership of Entrypoint and Cmd. - let (wrapped, file_owners) = private_services(&environments); + // before the private launcher or the dependency release barrier takes + // ownership of Entrypoint and Cmd. + let (mut wrapped, mut file_owners) = private_services(&environments); + let dependency_processes = startup + .as_ref() + .map(|driver| driver.process_services()) + .unwrap_or_default(); + for name in &dependency_processes { + file_owners.remove(name); + wrapped.insert(name.clone()); + } let expected_environment = verify_images( &engine, &prepared.resources, @@ -1505,6 +1542,7 @@ fn run_inputs( &file_owners, )?; validate_private_delivery(&prepared.configs, &environments)?; + validate_dependency_processes(&prepared.configs, &dependency_processes)?; if let Some(driver) = startup.as_ref() { driver.check_cancelled()?; } @@ -2521,6 +2559,7 @@ fn verify_images( file_owners: &BTreeSet, ) -> Result>, CandidateError> { let mut expected = BTreeMap::new(); + let mut refusals = process_refusals::Collector::default(); for resource in resources.values().filter(|r| r.kind == Kind::Container) { let image = engine.request( Method::GET, @@ -2547,24 +2586,32 @@ fn verify_images( resources, )?; if private_services.contains(&resource.key) { - image_process::apply_private( - configs - .get_mut(&resource.key) - .expect("compiled service config"), - &image["Config"], - )?; + refusals.check( + &resource.key, + image_process::apply_private( + configs + .get_mut(&resource.key) + .expect("compiled service config"), + &image["Config"], + ), + ); } else if file_owners.contains(&resource.key) { - image_process::apply_file_owner( - configs - .get_mut(&resource.key) - .expect("compiled service config"), - &image["Config"], - )?; + refusals.check( + &resource.key, + image_process::apply_file_owner( + configs + .get_mut(&resource.key) + .expect("compiled service config"), + &image["Config"], + ), + ); } expected.insert( resource.key.clone(), image_environment::compose(&image["Config"]["Env"], &configs[&resource.key]["Env"])?, ); } + // Every wrapped service is checked before the first refusal is reported. + refusals.finish()?; Ok(expected) } diff --git a/packages/runtime-core/src/provider/graph/native/runtime.rs b/packages/runtime-core/src/provider/graph/native/runtime.rs index b53e7cc70..ad76d7916 100644 --- a/packages/runtime-core/src/provider/graph/native/runtime.rs +++ b/packages/runtime-core/src/provider/graph/native/runtime.rs @@ -1088,7 +1088,8 @@ pub(super) fn run_guarded( verify_native_images(&engine, &config.resources, &mut config.configs, &private)? }; for name in &private { - launcher::validate(&config.configs[name])?; + launcher::validate(&config.configs[name]) + .map_err(super::super::process_refusals::Refused::into_error)?; } check_startup(startup_guard)?; selected.assert_fresh(candidate)?; @@ -1792,7 +1793,8 @@ fn verify_native_images( } } if private.contains(&resource.key) { - image_process::apply_private(config, &image["Config"])?; + image_process::apply_private(config, &image["Config"]) + .map_err(super::super::process_refusals::Refused::into_error)?; } expected.insert( resource.key.clone(), diff --git a/packages/runtime-core/src/provider/graph/plan_reconcile.rs b/packages/runtime-core/src/provider/graph/plan_reconcile.rs index f1d5d4e3e..b99031b11 100644 --- a/packages/runtime-core/src/provider/graph/plan_reconcile.rs +++ b/packages/runtime-core/src/provider/graph/plan_reconcile.rs @@ -13,18 +13,36 @@ //! (Compose also keeps project volumes of removed declarations); explicit //! data removal still reaches it; //! * containers and networks of removed services are dropped from the receipt only -//! because their absence was confirmed before this projection. +//! because their absence was confirmed before this projection; +//! * a dependency cache whose freshly resolved identity equals the recorded one is +//! kept with its provenance. Its identity binds only the initializer's own +//! definition, cache declaration, image and input files, so edits to other +//! services keep it. A changed identity selects that identity's shared cache +//! exactly as a fresh admission would (adopt by labels or create, then the +//! initializer runs); the old cache is shared, never run-owned data, so only +//! this run's reference to it is released. A retained run-owned volume is never +//! reselected as a cache. //! //! Ownership never moves: run, owner, namespace and the ownership plan recorded on //! every label are unchanged, so another instance's data cannot be selected. -//! Dependency caches, live source and source-device rebinding keep their narrower -//! dedicated proofs and are not eligible here. +//! Live source, a cache scope carried across a source-device rebind and pending +//! initializer cache-release effects keep their narrower dedicated proofs and are +//! not eligible here. use super::*; +use crate::error::PlanChangeField; fn refused(message: impl Into) -> CandidateError { error("graph_plan_reconcile", message) } +/// Run-owned data is never reselected as a shared, content-addressed cache. +fn cache_over_retained_data(service: &str) -> CandidateError { + refused( + "A retained named volume is now declared as a dependency cache; its data was not reselected as a shared cache.", + ) + .with_plan_change(service, PlanChangeField::DependencyCacheVolume) +} + /// Retained named data is any recorded volume that may have been created. A /// volume still `reserved` never received a create intent, so it holds no data. pub(super) fn retained_volume(resource: &Resource) -> bool { @@ -45,45 +63,51 @@ fn active_volume_keys(plan: &project::PlanData) -> BTreeSet { .collect() } -/// Whether the plan or receipt uses machinery that has its own narrower proof. -fn eligible(receipt: &Receipt, plan: &project::PlanData) -> bool { +/// Whether the receipt uses machinery that has its own narrower proof. +fn eligible(receipt: &Receipt) -> bool { receipt.normalized_input.is_some() && receipt.initializer_cache_release.is_empty() - && receipt - .resources - .values() - .all(|resource| resource.cache.is_none() && resource.cache_provenance.is_none()) && receipt .source .as_ref() .is_none_or(|binding| binding.cache_scope.is_none() && binding.live.is_none()) - && plan - .services - .values() - .all(|service| !service.active || service.dependency_cache.is_none()) } -/// The ordinary strict replay derives volume names from the sorted declared set. +/// The ordinary strict replay derives run-owned volume names from the sorted +/// declared set (a dependency cache takes an index but keeps its content name). /// Once a reconciliation reattached data out of that order, or retains data no /// longer declared, only this projection can restore the graph again. +/// A cache with a writable consumer is scoped to this instance; a receipt recorded +/// before that scoping names a shared cache that may hold another worktree's +/// writes. Such graphs always take this path, which keeps the cache only when its +/// freshly resolved instance-scoped identity equals the recorded binding exactly. fn ordinary_layout(receipt: &Receipt, plan: &project::PlanData) -> bool { + let scoped = dependency_cache::instance_scoped_volumes(plan); + if receipt.resources.values().any(|resource| { + resource.kind == Kind::Volume && resource.cache.is_some() && scoped.contains(&resource.key) + }) { + return false; + } let declared = active_volume_keys(plan); let recorded = receipt .resources .values() .filter(|resource| resource.kind == Kind::Volume) - .map(|resource| (resource.key.clone(), resource.name.clone())) + .map(|resource| (resource.key.clone(), resource)) .collect::>(); recorded.len() == declared.len() && declared.iter().enumerate().all(|(index, key)| { - recorded.get(key) == Some(&format!("{}{index}", volume_prefix(&receipt.run))) + recorded.get(key).is_some_and(|resource| { + resource.cache.is_some() + || resource.name == format!("{}{index}", volume_prefix(&receipt.run)) + }) }) } /// Selects reconciliation for a reviewed plan that the strict replay path cannot -/// restore. Unchanged reviews with the ordinary layout keep strict replay; -/// graphs with dependency caches, live source or a pending source-device rebind -/// keep their dedicated hostname/shared-source proofs. +/// restore. Unchanged reviews with the ordinary layout keep strict replay; live +/// source, a carried cache scope or a pending source-device rebind keep their +/// dedicated hostname/shared-source proofs. pub(super) fn applies( receipt: &Receipt, plan_id: &str, @@ -92,7 +116,7 @@ pub(super) fn applies( source_rebind: bool, ) -> bool { !source_rebind - && eligible(receipt, plan) + && eligible(receipt) && !(receipt.plan_id == plan_id && receipt.normalized_input.as_ref() == Some(next) && ordinary_layout(receipt, plan)) @@ -126,9 +150,9 @@ pub(super) fn prepare( .normalized_input .clone() .ok_or_else(|| refused("Only normalized graphs can be reconciled."))?; - if !eligible(receipt, plan) { + if !eligible(receipt) { return Err(refused( - "Dependency caches, live source and cache-scoped source use their dedicated restore proofs.", + "Live source, a carried cache scope and pending initializer cache release use their dedicated restore proofs.", )); } if !next.matches_plan(plan) @@ -156,17 +180,31 @@ pub(super) fn prepare( if resource.kind != Kind::Volume { continue; } - if !resource - .name - .strip_prefix(&prefix) - .is_some_and(|index| index.parse::().is_ok_and(|n| n < 64)) - || !matches!(resource.phase.as_str(), "created" | "reserved") - { + let owned_name = resource.cache.as_ref().map_or_else( + || { + resource + .name + .strip_prefix(&prefix) + .is_some_and(|index| index.parse::().is_ok_and(|n| n < 64)) + }, + |cache| cache.valid() && resource.name == cache.name(), + ); + if !owned_name || !matches!(resource.phase.as_str(), "created" | "reserved") { return Err(refused( "Retained data is not this run's settled owned volume; it was not reattached.", )); } } + for (service, declared) in plan.services.iter().filter(|(_, service)| service.active) { + if let Some(cache) = &declared.dependency_cache + && receipt + .resources + .get(&format!("volume:{}", cache.volume)) + .is_some_and(|old| old.cache.is_none() && retained_volume(old)) + { + return Err(cache_over_retained_data(service)); + } + } Ok(Verified { run: receipt.run.clone(), owner: receipt.owner.clone(), @@ -219,11 +257,14 @@ impl Verified { if !self.matches_receipt(receipt) || prepared.namespace != receipt.namespace { return Err(refused("Reconciliation ownership changed.")); } + config::retain_recorded_legacy_caches(prepared, receipt); let prefix = volume_prefix(&receipt.run); + // Run-owned retained data. Dependency caches are shared, content-addressed + // volumes selected separately below; they are never reattached as data. let retained = receipt .resources .iter() - .filter(|(_, resource)| retained_volume(resource)) + .filter(|(_, resource)| retained_volume(resource) && resource.cache.is_none()) .collect::>(); let mut used = retained .values() @@ -236,12 +277,42 @@ impl Verified { resources.insert(key.clone(), resource.clone()); continue; } - if resource.cache.is_some() { - return Err(refused( - "Dependency cache volumes use their dedicated restore proof.", - )); - } - let selected = if let Some(old) = retained.get(key) { + let selected = if let Some(cache) = &resource.cache { + if retained.contains_key(key) { + let service = prepared + .cache_initializers + .iter() + .find(|(_, volume)| *volume == &resource.key) + .map_or(resource.key.as_str(), |(service, _)| service.as_str()); + return Err(cache_over_retained_data(service)); + } + match receipt.resources.get(key) { + // Same identity: keep the cache and its provenance fence. + Some(old) + if old.cache.as_ref() == Some(cache) + && old.name == resource.name + && matches!(old.phase.as_str(), "created" | "reserved") => + { + let mut kept = old.clone(); + if let Some(provenance) = kept.cache_provenance.as_mut() { + provenance.retain_initializers(|service| { + prepared + .resources + .contains_key(&format!("container:{service}")) + }); + } + kept + } + // New or changed identity: fresh-admission selection of the + // shared cache; the recorded reference (if any) is released. + _ => Resource { + id: None, + cache_provenance: None, + phase: "reserved".into(), + ..resource.clone() + }, + } + } else if let Some(old) = retained.get(key) { if old.key != resource.key || old.disposable_caches != resource.disposable_caches { return Err(refused( "A retained volume's disposable-cache declaration changed; it was not reattached.", @@ -270,6 +341,7 @@ impl Verified { resources.insert(key.clone(), selected); } // Retained data a changed plan no longer declares stays owned and recorded. + // A dependency cache is not run-owned data: an undeclared one is released. for (key, resource) in &retained { resources .entry((*key).clone()) @@ -342,7 +414,7 @@ impl Verified { #[cfg(test)] mod tests { use super::*; - use std::path::Path; + use std::{os::unix::fs::DirBuilderExt, path::Path}; const RUN: &str = "abababababababababababababababab"; @@ -397,6 +469,20 @@ mod tests { project::PlanReport, NormalizedInputIdentity, config::Prepared, + ) { + self.prepared_with(document, false) + } + + /// Dependency-cache graphs resolve their identities from a reviewed + /// source snapshot, as a fresh admission's publication supplies it. + fn prepared_with( + &self, + document: Value, + cached: bool, + ) -> ( + project::PlanReport, + NormalizedInputIdentity, + config::Prepared, ) { let (report, identity) = self.review(document); let inputs = project::inputs::compile( @@ -416,9 +502,35 @@ mod tests { .services .iter() .filter(|(_, service)| service.active) - .map(|(name, _)| (name.clone(), Condition::Started)) + .map(|(name, service)| { + let goal = if service.dependency_cache.is_some() { + Condition::Completed + } else { + Condition::Started + }; + (name.clone(), goal) + }) .collect(); - let prepared = config::prepare(inputs, &readiness, RUN, &"b".repeat(32), None).unwrap(); + let source = cached.then(|| { + let snapshot = project::snapshot::capture_plan(&report.plan).unwrap(); + let manifest = snapshot.receipt().clone(); + source::Inputs { + current_manifest: None, + binding: SourceBinding { + cache_scope: None, + shared: None, + shared_contract: None, + live: None, + revision: manifest.revision.clone(), + archive_sha256: "6".repeat(64), + selection_sha256: manifest.selection_sha256.clone(), + }, + manifest, + paths: BTreeMap::new(), + } + }); + let prepared = + config::prepare(inputs, &readiness, RUN, &"b".repeat(32), source.as_ref()).unwrap(); (report, identity, prepared) } } @@ -802,4 +914,318 @@ mod tests { super::super::restore::observed_volumes_with(&next, |_| Ok(None)).unwrap_err(); assert_eq!(missing.code, "graph_data_missing"); } + + /// An initializer populating a dependency cache, beside an app that mounts + /// the cache read-write (so the cache is instance-scoped) and its own data. + fn cached(marker: &str, installer: &str) -> Value { + cached_with(marker, installer, "deps:/deps") + } + + /// The same graph whose app only reads the cache, so the cache is shared. + fn cached_shared(marker: &str, installer: &str) -> Value { + cached_with(marker, installer, "deps:/deps:ro") + } + + fn cached_with(marker: &str, installer: &str, consumer: &str) -> Value { + json!({ + "services": { + "deps": { + "image": image('a'), + "network_mode": "none", + "entrypoint": [installer], + "volumes": ["deps:/deps"], + "labels": { + "hack.dependencies.bootstrap": "true", + "hack.dependencies.cache-volume": "deps", + "hack.dependencies.lockfiles": "lock.txt", + }, + }, + "app": { + "image": image('a'), + "network_mode": "none", + "entrypoint": ["/bin/app"], + "environment": {"MARKER": marker}, + "volumes": [consumer, "data:/data"], + }, + }, + "volumes": {"deps": {}, "data": {}}, + }) + } + + fn cached_project() -> (Project, Receipt, config::Prepared) { + cached_project_with(cached("one", "/bin/install")) + } + + fn cached_project_with(document: Value) -> (Project, Receipt, config::Prepared) { + let project = Project::new(); + fs::write(project.root.join("lock.txt"), "v1").unwrap(); + let shared = document["services"]["app"]["volumes"][0] == "deps:/deps:ro"; + let (report, identity, prepared) = project.prepared_with(document, true); + let mut receipt = stopped(&prepared, &identity); + // A completed first admission recorded the cache's provenance fence. + let provenance: cache_provenance::CacheProvenance = serde_json::from_value(json!({ + "version": 1, "origin": "created_here", "boot": "boot", + "identity": {"created_at": "2026-11-06T00:00:00Z", "directory": "1:2"}, + "initializers": ["deps"], "completed": {}, + })) + .unwrap(); + receipt + .resources + .get_mut("volume:deps") + .unwrap() + .cache_provenance = Some(provenance); + assert!(receipt.resources["volume:deps"].cache.is_some()); + // A shared cache keeps strict replay; an instance-scoped one always + // re-proves its identity through reconciliation. + assert_eq!( + applies(&receipt, &report.plan_id, &report.plan, &identity, false), + !shared + ); + (project, receipt, prepared) + } + + fn reconcile_cached( + project: &Project, + receipt: &Receipt, + next: Value, + ) -> (config::Prepared, Projection) { + let (report, identity, mut prepared) = project.prepared_with(next, true); + assert!(applies( + receipt, + &report.plan_id, + &report.plan, + &identity, + false + )); + let proof = prepare(&report.plan, receipt, &identity).unwrap(); + let projection = proof.project(&mut prepared, receipt).unwrap(); + let mut applied = receipt.clone(); + let snapshot = Projection { + resources: projection.resources.clone(), + }; + let readiness = projection + .resources + .values() + .filter(|resource| resource.kind == Kind::Container) + .map(|resource| (resource.key.clone(), Condition::Started)) + .collect(); + proof + .apply(&mut applied, snapshot, &readiness, None) + .unwrap(); + // The reconciled receipt still loads: every cache name and provenance is valid. + let fixture = super::super::tests::Fixture::new(); + let root = fixture.0.join(RUN); + fs::DirBuilder::new().mode(0o700).create(&root).unwrap(); + state::write(&root.join("state.json"), &applied).unwrap(); + super::super::load_at(root, RUN, &applied.owner).unwrap(); + (prepared, projection) + } + + #[test] + fn env_edit_to_another_service_reconciles_and_keeps_the_dependency_cache() { + let (project, receipt, original) = cached_project(); + let cache = receipt.resources["volume:deps"].clone(); + let (prepared, projection) = + reconcile_cached(&project, &receipt, cached("two", "/bin/install")); + // The same shared cache, with its provenance, and the same run-owned data. + assert!(same(&projection.resources["volume:deps"], &cache)); + assert!(same( + &projection.resources["volume:data"], + &receipt.resources["volume:data"] + )); + assert!(added(&projection).is_empty()); + assert_eq!(mounts(&prepared, "deps"), mounts(&original, "deps")); + assert_eq!(mounts(&prepared, "app"), mounts(&original, "app")); + assert_eq!(projection.resources["container:app"].phase, "reserved"); + } + + #[test] + fn changed_cache_inputs_select_a_fresh_cache_and_keep_run_data() { + let (project, receipt, _) = cached_project(); + let old = receipt.resources["volume:deps"].clone(); + // The initializer's own definition is a cache input. + let (prepared, projection) = + reconcile_cached(&project, &receipt, cached("one", "/bin/install-v2")); + let fresh = &projection.resources["volume:deps"]; + assert_ne!(fresh.cache, old.cache); + assert_ne!(fresh.name, old.name); + assert_eq!(fresh.name, fresh.cache.as_ref().unwrap().name()); + assert_eq!(fresh.phase, "reserved"); + assert!(fresh.cache_provenance.is_none()); + assert!(mounts(&prepared, "app").contains(&fresh.name)); + assert!(!mounts(&prepared, "app").contains(&old.name)); + assert!(same( + &projection.resources["volume:data"], + &receipt.resources["volume:data"] + )); + // So is a lockfile's content. + fs::write(project.root.join("lock.txt"), "v2").unwrap(); + let (_, projection) = reconcile_cached(&project, &receipt, cached("two", "/bin/install")); + let rebuilt = &projection.resources["volume:deps"]; + assert_ne!(rebuilt.cache, old.cache); + assert_eq!(rebuilt.phase, "reserved"); + } + + #[test] + fn adding_a_cache_writer_selects_a_fresh_cache() { + let (project, receipt, _) = cached_project(); + let old = receipt.resources["volume:deps"].clone(); + // A new service that mounts the shared cache read-write can change what + // every run sharing it reuses, so it is part of the cache identity. + let mut next = cached("one", "/bin/install"); + next["services"]["worker"] = json!({ + "image": image('a'), + "network_mode": "none", + "entrypoint": ["/bin/worker"], + "volumes": ["deps:/deps"], + }); + let (_, projection) = reconcile_cached(&project, &receipt, next); + let fresh = &projection.resources["volume:deps"]; + assert_ne!(fresh.cache, old.cache); + assert_eq!(fresh.phase, "reserved"); + assert!(same( + &projection.resources["volume:data"], + &receipt.resources["volume:data"] + )); + } + + #[test] + fn retained_run_data_is_never_reselected_as_a_dependency_cache() { + let project = Project::new(); + fs::write(project.root.join("lock.txt"), "v1").unwrap(); + let mut plain = cached("one", "/bin/install"); + plain["services"]["deps"] + .as_object_mut() + .unwrap() + .remove("labels"); + let (_, identity, prepared) = project.prepared(plain); + let receipt = stopped(&prepared, &identity); + assert!(receipt.resources["volume:deps"].cache.is_none()); + let (report, identity, mut next) = + project.prepared_with(cached("one", "/bin/install"), true); + let failure = prepare(&report.plan, &receipt, &identity).err().unwrap(); + assert_eq!(failure.code, "graph_plan_reconcile"); + let change = failure.plan_change().unwrap(); + assert_eq!(change.service, "deps"); + assert_eq!(change.field, PlanChangeField::DependencyCacheVolume); + // Admission repeats the check on the projection itself. + let proof = Verified { + run: receipt.run.clone(), + owner: receipt.owner.clone(), + namespace: receipt.namespace.clone(), + ownership_plan: receipt.plan_id.clone(), + previous: receipt.normalized_input.clone().unwrap(), + next: identity, + }; + let failure = proof.project(&mut next, &receipt).err().unwrap(); + assert_eq!(failure.plan_change().unwrap().service, "deps"); + } + + #[test] + fn dropping_a_cache_declaration_releases_the_reference_and_creates_run_data() { + let (project, receipt, _) = cached_project(); + let mut plain = cached("one", "/bin/install"); + plain["services"]["deps"] + .as_object_mut() + .unwrap() + .remove("labels"); + let (report, identity, mut prepared) = project.prepared(plain); + let proof = prepare(&report.plan, &receipt, &identity).unwrap(); + let projection = proof.project(&mut prepared, &receipt).unwrap(); + let deps = &projection.resources["volume:deps"]; + assert!(deps.cache.is_none()); + assert_eq!(deps.phase, "reserved"); + assert!(deps.name.starts_with(&volume_prefix(RUN))); + assert_ne!(deps.name, receipt.resources["volume:data"].name); + } + + #[test] + fn legacy_cache_identity_is_kept_only_while_its_inputs_still_match() { + let (project, mut receipt, original) = + cached_project_with(cached_shared("one", "/bin/install")); + let legacy = original.legacy_caches["deps"].clone(); + assert_ne!( + Some(&legacy), + original.resources["volume:deps"].cache.as_ref() + ); + let deps = receipt.resources.get_mut("volume:deps").unwrap(); + deps.cache = Some(legacy.clone()); + deps.name = legacy.name(); + // Unchanged review: strict replay keeps the recorded legacy cache. + let (report, identity, mut prepared) = + project.prepared_with(cached_shared("one", "/bin/install"), true); + // The review's source metadata is fresh; the receipt is that admission's. + receipt.plan_id = report.plan_id.clone(); + assert!(!applies( + &receipt, + &report.plan_id, + &report.plan, + &identity, + false + )); + config::retain_replay_ownership(&mut prepared, &receipt).unwrap(); + assert_eq!(prepared.resources["volume:deps"].name, legacy.name()); + assert!(mounts(&prepared, "app").contains(&legacy.name())); + // An edit changes the legacy whole-document identity: the current one is + // selected once, after which edits elsewhere keep it. + let (_, projection) = + reconcile_cached(&project, &receipt, cached_shared("two", "/bin/install")); + let fresh = &projection.resources["volume:deps"]; + assert_ne!(fresh.name, legacy.name()); + assert_eq!(fresh.cache, original.resources["volume:deps"].cache); + assert_eq!(fresh.phase, "reserved"); + } + + #[test] + fn an_earlier_shared_cache_with_a_writable_consumer_moves_to_this_instance() { + let (project, mut receipt, original) = cached_project(); + // No earlier identity is ever matched for a writable consumer. + assert!(!original.legacy_caches.contains_key("deps")); + let scoped = original.resources["volume:deps"].cache.clone().unwrap(); + // A receipt from an earlier build names a cache shared across worktrees. + let shared = dependency_cache::CacheBinding { + scope: dependency_cache::scope(&project.root).unwrap(), + fingerprint: "c".repeat(64), + image: scoped.image.clone(), + }; + assert_ne!(shared.scope, scoped.scope); + let deps = receipt.resources.get_mut("volume:deps").unwrap(); + deps.cache = Some(shared.clone()); + deps.name = shared.name(); + // Even the unchanged review re-proves identity and selects this + // instance's cache once; the shared one is never mounted again. + let (prepared, projection) = + reconcile_cached(&project, &receipt, cached("one", "/bin/install")); + let fresh = &projection.resources["volume:deps"]; + assert_eq!(fresh.cache.as_ref(), Some(&scoped)); + assert_eq!(fresh.phase, "reserved"); + assert!(!mounts(&prepared, "app").contains(&shared.name())); + } + + #[test] + fn reserved_shared_caches_may_exist_but_reserved_run_names_may_not() { + let (project, receipt, _) = cached_project(); + let (_, projection) = + reconcile_cached(&project, &receipt, cached("one", "/bin/install-v2")); + let mut next = receipt.clone(); + next.resources = projection.resources; + let present = |_: &Resource| { + Ok(Some( + json!({"inspected": {"Name": "x"}, "directory": "1:2"}), + )) + }; + // The fresh cache may already exist as another graph's shared cache. + let selected = super::super::restore::observed_volumes_with(&next, present).unwrap(); + assert!(!selected.contains_key("volume:deps")); + assert!(selected.contains_key("volume:data")); + let mut owned = next.clone(); + let data = owned.resources.get_mut("volume:data").unwrap(); + data.phase = "reserved".into(); + assert_eq!( + super::super::restore::observed_volumes_with(&owned, present) + .unwrap_err() + .code, + "graph_resource_exists" + ); + } } diff --git a/packages/runtime-core/src/provider/graph/process_refusals.rs b/packages/runtime-core/src/provider/graph/process_refusals.rs new file mode 100644 index 000000000..79696bab2 --- /dev/null +++ b/packages/runtime-core/src/provider/graph/process_refusals.rs @@ -0,0 +1,138 @@ +//! Named, collected refusals of a private process wrapper's selected process. +//! +//! A wrapper (private environment delivery or the host-dependency release +//! barrier) runs the exact process Docker would run, so every wrapped service is +//! checked before any runtime state is reserved. Each refusal keeps its fixed +//! code and value-free message and also names the service and the closed field +//! to change. All wrapped services are checked in one pass so a project with +//! several affected stock images learns about all of them from one start. +use crate::error::{CandidateError, ProcessField, ProcessRefusal}; + +/// One wrapper refusal before it is attributed to a service. +#[derive(Debug)] +pub(super) struct Refused { + pub(super) field: ProcessField, + pub(super) error: CandidateError, +} + +impl Refused { + pub(super) fn new(field: ProcessField, code: &'static str, message: &'static str) -> Self { + Self { + field, + error: CandidateError::new(code, message), + } + } + + /// The plain refusal, for a caller that checks one process only. + pub(super) fn into_error(self) -> CandidateError { + self.error + } +} + +/// Bounded like other value-free per-service diagnostics. +const MAX_REFUSALS: usize = 32; + +/// Collects refusals across services; the first refusal's code is reported. +#[derive(Default)] +pub(super) struct Collector { + first: Option, + refusals: Vec, +} + +impl Collector { + pub(super) fn check(&mut self, service: &str, result: Result<(), Refused>) { + let Err(refused) = result else { + return; + }; + if self.refusals.len() < MAX_REFUSALS + && !self + .refusals + .iter() + .any(|known| known.service == service && known.field == refused.field) + { + self.refusals.push(ProcessRefusal { + service: service.into(), + field: refused.field, + }); + } + self.first.get_or_insert(refused.error); + } + + pub(super) fn finish(self) -> Result<(), CandidateError> { + match self.first { + None => Ok(()), + Some(mut error) => { + error.message = format!( + "{} Refused process settings are named in process_refusals.", + error.message + ); + Err(error.with_process_refusals(self.refusals)) + } + } + } +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn collects_every_service_and_field_and_keeps_the_first_code() { + let mut collector = Collector::default(); + collector.check("web", Ok(())); + collector.check( + "db", + Err(Refused::new( + ProcessField::User, + "environment_user", + "Invalid.", + )), + ); + collector.check( + "api", + Err(Refused::new( + ProcessField::Healthcheck, + "environment_health", + "Invalid.", + )), + ); + collector.check( + "db", + Err(Refused::new( + ProcessField::User, + "environment_user", + "Again.", + )), + ); + let error = collector.finish().unwrap_err(); + assert_eq!(error.code, "environment_user"); + let detail = error.process_refusals().expect("named refusals"); + assert_eq!(detail.version, 1); + assert_eq!( + detail.refusals, + vec![ + ProcessRefusal { + service: "db".into(), + field: ProcessField::User + }, + ProcessRefusal { + service: "api".into(), + field: ProcessField::Healthcheck + }, + ] + ); + let reply = serde_json::to_value( + CandidateError::new("environment_user", "Invalid.") + .with_process_refusals(detail.refusals.clone()), + ) + .unwrap(); + assert_eq!( + reply["process_refusals"], + serde_json::json!({"version":1,"refusals":[ + {"service":"db","field":"user"}, + {"service":"api","field":"healthcheck"} + ]}) + ); + assert!(Collector::default().finish().is_ok()); + } +} diff --git a/packages/runtime-core/src/provider/graph/restore.rs b/packages/runtime-core/src/provider/graph/restore.rs index 2bd5cc14f..5168e6736 100644 --- a/packages/runtime-core/src/provider/graph/restore.rs +++ b/packages/runtime-core/src/provider/graph/restore.rs @@ -176,7 +176,9 @@ pub(super) fn observed_volumes_with( } if !plan_reconcile::retained_volume(resource) { // Never created: it holds no data, and an existing name is not adopted. - if observe(resource)?.is_some() { + // A not-yet-selected dependency cache is shared and content-addressed: + // admission adopts it by labels or creates it, as a fresh graph does. + if resource.cache.is_none() && observe(resource)?.is_some() { return Err(error( "graph_resource_exists", "A never-created volume name already resolves; it will not be adopted.", @@ -373,19 +375,20 @@ fn restore_inputs( #[cfg(not(target_os = "macos"))] let source_receipt = &receipt; let ordinary_source = if reconcile.is_some() { - // Exactly what a fresh admission of this review would select. Caches are - // not eligible here, so no retained cache publication may be named. - if options.source_revision.is_some() || options.live_source { + // Exactly what a fresh admission of this review would select: dependency + // caches name a publication that must match the current project contents + // (verified by the same mode checks); other graphs name none. + if options.live_source { return Err(error( "graph_plan_reconcile", - "Reconciliation selects no retained cache publication or live source.", + "Reconciliation selects no live source.", )); } source::prepare_mode( candidate, &engine, &inputs.review.plan, - None, + options.source_revision, false, options.shared_source, )? @@ -484,7 +487,15 @@ fn restore_inputs( if !environments.is_empty() { super::super::environment::preflight_capacity(engine.guest(), environments.len())?; } - let (wrapped, file_owners) = private_services(&environments); + let (mut wrapped, mut file_owners) = private_services(&environments); + let dependency_processes = startup + .as_ref() + .map(|driver| driver.process_services()) + .unwrap_or_default(); + for name in &dependency_processes { + file_owners.remove(name); + wrapped.insert(name.clone()); + } let expected_environment = verify_images( &engine, &prepared.resources, @@ -492,6 +503,7 @@ fn restore_inputs( &wrapped, &file_owners, )?; + super::validate_dependency_processes(&prepared.configs, &dependency_processes)?; for resource in receipt.resources.values() { let present = inspect_resource(&engine, &receipt, resource)?.is_some(); if resource.kind == Kind::Volume { @@ -501,7 +513,7 @@ fn restore_inputs( "Retained volume is missing; restore will not recreate it.", )); } - if !plan_reconcile::retained_volume(resource) && present { + if !plan_reconcile::retained_volume(resource) && resource.cache.is_none() && present { return Err(error( "graph_resource_exists", "A never-created volume name already resolves; it will not be adopted.", diff --git a/packages/runtime-core/src/provider/graph/startup.rs b/packages/runtime-core/src/provider/graph/startup.rs index 15a03aa2c..b4a31320b 100644 --- a/packages/runtime-core/src/provider/graph/startup.rs +++ b/packages/runtime-core/src/provider/graph/startup.rs @@ -188,6 +188,12 @@ pub(super) trait Driver { fn check_cancelled(&self) -> Result<(), CandidateError> { Ok(()) } + /// Services whose process this driver wraps behind its release barrier. + /// Their inherited image process is materialized before admission, exactly + /// as for private environment delivery, so the wrapper runs what Docker would. + fn process_services(&self) -> std::collections::BTreeSet { + std::collections::BTreeSet::new() + } fn validate_inputs( &self, inputs: &project::inputs::ExecutionInputs, diff --git a/packages/runtime-core/src/provider/graph/startup/runtime.rs b/packages/runtime-core/src/provider/graph/startup/runtime.rs index 5d4c25c5e..b8a8665fb 100644 --- a/packages/runtime-core/src/provider/graph/startup/runtime.rs +++ b/packages/runtime-core/src/provider/graph/startup/runtime.rs @@ -683,6 +683,13 @@ impl Driver for HostRelayRuntime { self.run.is_some() } + fn process_services(&self) -> std::collections::BTreeSet { + self.dependencies + .keys() + .map(|(service, _)| service.clone()) + .collect() + } + fn check_cancelled(&self) -> Result<(), CandidateError> { if self.startup_cancelled.is_some_and(|check| check()) { return Err(error( @@ -816,7 +823,8 @@ impl Driver for HostRelayRuntime { let config = configs .get(name) .ok_or_else(|| stage_refused("graph_startup_service_config"))?; - launcher::validate(config)?; + launcher::validate(config) + .map_err(super::super::process_refusals::Refused::into_error)?; if config["HostConfig"]["Init"] != true { return Err(stage_refused("graph_startup_init")); } diff --git a/scripts/lib/tla-runtime-models.ts b/scripts/lib/tla-runtime-models.ts index 8e5b2c1d5..227c136d8 100644 --- a/scripts/lib/tla-runtime-models.ts +++ b/scripts/lib/tla-runtime-models.ts @@ -163,6 +163,34 @@ const contracts: readonly ModelContract[] = [ }, ], }, + { + name: "native-cache-reconcile", + module: "CacheReconcile", + states: 2208, + invariant: "UnrelatedEditKeepsCache", + action: "UpCache", + fields: ["needlessRebuild = TRUE", "staleMounted = FALSE"], + additionalControls: [ + { + name: "stale", + negative: true, + invariant: "CacheMatchesInputs", + fields: ["staleMounted = TRUE", "needlessRebuild = FALSE"], + }, + { + name: "data-as-cache", + negative: true, + invariant: "NoDataAsCache", + fields: ["dataMounted = TRUE", "staleMounted = FALSE"], + }, + { + name: "poison", + negative: true, + invariant: "NoForeignWrites", + fields: ["foreignMounted = TRUE", "staleMounted = FALSE"], + }, + ], + }, { name: "pre-ready-retirement", module: "Retirement", diff --git a/src/backends/native-compose-interpolation.ts b/src/backends/native-compose-interpolation.ts new file mode 100644 index 000000000..f09feda33 --- /dev/null +++ b/src/backends/native-compose-interpolation.ts @@ -0,0 +1,199 @@ +import { COMPOSE_PROJECT_NAME_VARIABLE } from "./native-compose-project-name.ts"; + +/** + * Predict the value native execution compiles one Compose string to, before + * any runtime work. + * + * Native execution resolves Compose interpolation itself (runtime-core + * `project::inputs::Resolver`) from an explicit map: no process environment or + * dotenv file is consulted, and the only bound name is `COMPOSE_PROJECT_NAME` + * (the reviewed Compose project name). Every other reference is therefore + * unset: `${NAME:-word}` selects `word`, `${NAME:+word}` selects nothing, and + * `$NAME` or `${NAME}` refuses at execution. + * + * This mirrors that grammar exactly: `$$` escapes, braced `:-` and `:+` + * operators whose words hold literal text, `$$` and nested braced expressions + * (at most four levels), and unbraced references. It returns null whenever + * native execution would refuse the text or the value is not knowable yet: a + * required reference, unsupported syntax, a Hack-managed name (delivered + * privately and never interpolated), or `COMPOSE_PROJECT_NAME`, which is bound + * only after review. Callers must treat null as "unknown" and leave the final + * decision to native review and execution, which stay the authority. + */ +export function resolveNativeComposeText(opts: { + readonly text: string; + /** Names Hack delivers privately to the service; a reference refuses. */ + readonly managed?: ReadonlySet; +}): string | null { + if (opts.text.length > MAX_TEXT) { + return null; + } + try { + return resolve(opts.text, opts.managed ?? EMPTY); + } catch (error) { + if (error === UNKNOWN) { + return null; + } + throw error; + } +} + +const MAX_TEXT = 64 * 1024; +/** Compose nesting such as `${A:-${B:-${C:-${D:-word}}}}` is bounded at four levels. */ +const MAX_DEPTH = 4; +const EMPTY: ReadonlySet = new Set(); +const VARIABLE = /^[A-Za-z_][A-Za-z0-9_]*$/; +const NAME_BYTE = /[A-Za-z0-9_]/; +/** Thrown internally for any text whose native value is not knowable here. */ +const UNKNOWN = Symbol("unknown"); + +function unknown(): never { + throw UNKNOWN; +} + +/** + * Validate a referenced name exactly as the runtime does before lookup. A live + * reference to the project name is unknown here; elsewhere it is irrelevant. + */ +function checked( + name: string, + managed: ReadonlySet, + live: boolean +): void { + if ( + !VARIABLE.test(name) || + managed.has(name) || + (live && name === COMPOSE_PROJECT_NAME_VARIABLE) + ) { + unknown(); + } +} + +/** Index of the `}` closing a body that follows `${`, as the runtime counts it. */ +function bracedEnd(text: string): number { + let depth = 1; + let index = 0; + while (index < text.length) { + if (text.startsWith("$$", index)) { + index += 2; + continue; + } + if (text.startsWith("${", index)) { + depth += 1; + index += 2; + continue; + } + if (text[index] === "}") { + depth -= 1; + if (depth === 0) { + return index; + } + } + index += 1; + } + return unknown(); +} + +function resolve(text: string, managed: ReadonlySet): string { + let result = ""; + let rest = text; + for (;;) { + const index = rest.indexOf("$"); + if (index < 0) { + return result + rest; + } + result += rest.slice(0, index); + rest = rest.slice(index + 1); + if (rest.startsWith("$")) { + result += "$"; + rest = rest.slice(1); + continue; + } + if (rest.startsWith("{")) { + const body = rest.slice(1); + const end = bracedEnd(body); + result += expression(body.slice(0, end), 1, true, managed); + rest = body.slice(end + 1); + continue; + } + // An unbraced reference is always required, and no supplied value exists. + let end = 0; + while (end < rest.length && NAME_BYTE.test(rest[end] ?? "")) { + end += 1; + } + checked(rest.slice(0, end), managed, true); + return unknown(); + } +} + +/** + * One braced body. `live` is false inside a word the operator does not select: + * it is still validated, but a missing input there is not required. + */ +function expression( + body: string, + depth: number, + live: boolean, + managed: ReadonlySet +): string { + const colon = body.indexOf(":"); + if (colon < 0) { + checked(body, managed, live); + // A live bare reference is required; nothing is supplied. + return live ? unknown() : ""; + } + const name = body.slice(0, colon); + const suffix = body.slice(colon + 1); + if (suffix === "") { + return unknown(); + } + checked(name, managed, live); + const operator = suffix[0]; + if (operator !== "+" && operator !== "-") { + return unknown(); + } + // Every checked name is unset, so `:-` selects its word and `:+` does not. + const selected = operator === "-"; + const value = word(suffix.slice(1), depth, live && selected, managed); + return selected ? value : ""; +} + +/** Literal text, `$$` escapes and nested braced expressions only. */ +function word( + text: string, + depth: number, + live: boolean, + managed: ReadonlySet +): string { + let result = ""; + let rest = text; + for (;;) { + const found = rest.indexOf("$"); + const index = found < 0 ? rest.length : found; + const literal = rest.slice(0, index); + if (literal.includes("{") || literal.includes("}")) { + return unknown(); + } + if (live) { + result += literal; + } + if (index === rest.length) { + return result; + } + rest = rest.slice(index + 1); + if (rest.startsWith("$")) { + if (live) { + result += "$"; + } + rest = rest.slice(1); + continue; + } + if (!rest.startsWith("{") || depth >= MAX_DEPTH) { + return unknown(); + } + const body = rest.slice(1); + const end = bracedEnd(body); + result += expression(body.slice(0, end), depth + 1, live, managed); + rest = body.slice(end + 1); + } +} diff --git a/src/backends/native-plan-change.ts b/src/backends/native-plan-change.ts new file mode 100644 index 000000000..fb331e8d7 --- /dev/null +++ b/src/backends/native-plan-change.ts @@ -0,0 +1,70 @@ +import { isRecord } from "../lib/guards.ts"; + +/** + * Where a Compose edit that a retained native run cannot apply in place was + * made. The service is an admitted Compose identifier and the field a closed + * vocabulary, so both are safe to print; native messages never are. + */ +export interface NativePlanChange { + readonly service: string; + readonly field: NativePlanChangeField; +} + +const FIELDS = { + dependency_cache_volume: "`hack.dependencies.cache-volume`", + hostnames: "routed hostnames", +} as const; +type NativePlanChangeField = keyof typeof FIELDS; + +const SERVICE = /^[A-Za-z0-9_.-]{1,128}$/; + +/** Accept only the exact versioned diagnostic; anything else is dropped. */ +export function readNativePlanChange( + value: unknown +): NativePlanChange | undefined { + if ( + !isRecord(value) || + Object.keys(value).sort().join(",") !== "field,service,version" || + value.version !== 1 || + typeof value.service !== "string" || + !SERVICE.test(value.service) || + typeof value.field !== "string" || + !Object.hasOwn(FIELDS, value.field) + ) { + return; + } + return { + service: value.service, + field: value.field as NativePlanChangeField, + }; +} + +const APPLY = + "Revert the edit to keep the retained run as admitted, or run `hack down --remove-data` and then `hack up` to apply it with fresh data."; + +/** + * Closed guidance for native refusals of a Compose edit to a retained run. + * Returns undefined for every other code, which keeps the generic text. + */ +export function nativePlanChangeGuidance( + code: string, + change: NativePlanChange | undefined +): string | undefined { + const where = change + ? ` (service \`${change.service}\`, ${FIELDS[change.field]})` + : ""; + switch (code) { + case "graph_plan_reconcile": + return `this Compose edit${where} cannot be applied to the retained run. ${ + change?.field === "dependency_cache_volume" + ? "A retained named volume holds run data and is never reused as a shared dependency cache. " + : "" + }${APPLY}`; + case "graph_plan_change_unsupported": + return `this Compose edit cannot be applied in place: the retained run uses live source, a carried cache scope, a pending source-device rebind or a pending initializer cache release, so only routed hostname changes apply to it. ${APPLY}`; + case "graph_hostname_change": + return `the hostname change${where} cannot be applied to the retained run together with the other changes. Apply the hostname change on its own, or ${APPLY.charAt(0).toLowerCase()}${APPLY.slice(1)}`; + default: + return; + } +} diff --git a/src/backends/native-project-internal-hosts.ts b/src/backends/native-project-internal-hosts.ts index 2330a18cc..5e2504abb 100644 --- a/src/backends/native-project-internal-hosts.ts +++ b/src/backends/native-project-internal-hosts.ts @@ -1,11 +1,15 @@ import { HackCliError } from "../lib/cli-result.ts"; import { isRecord } from "../lib/guards.ts"; -import { resolveInternalExtraHosts } from "../lib/internal-extra-hosts.ts"; +import { + resolveExtraHostsPath, + resolveInternalExtraHosts, +} from "../lib/internal-extra-hosts.ts"; import { type ProjectConfig, type ProjectContext, readProjectConfig, } from "../lib/project.ts"; +import { resolveNativeComposeText } from "./native-compose-interpolation.ts"; import type { NativeHostDependency } from "./native-project-dependencies.ts"; import type { NativeProjectInput } from "./native-project-input.ts"; @@ -24,21 +28,98 @@ import type { NativeProjectInput } from "./native-project-input.ts"; const LABEL = /^[a-z0-9](?:[a-z0-9-]{0,61}[a-z0-9])?$/; const NUMERIC_HOST = /^[0-9.]+$/; -const SEPARATOR = /[:=]/; const MAX_ALIASES = 32; -/** Fixed text when a declared host alias has no reviewed dependency binding. */ +/** Leading text when a declared host alias has no reviewed dependency binding. */ export const NATIVE_UNBOUND_HOST_ALIAS_REFUSAL = - "Native host alias has no reviewed host dependency binding; configuration was not dropped. Select a host dependency binding for every service that declares the alias."; + "Native host alias has no reviewed host dependency binding; configuration was not dropped."; /** Fixed text when a project-wide host alias cannot be expressed natively. */ export const NATIVE_INTERNAL_HOST_ALIAS_REFUSAL = "Native up supports internal extra_hosts only as exact host-gateway aliases with reviewed host dependency bindings; configuration was not dropped."; -function unbound(): Error { +/** Leading text when a service's declared host aliases cannot be compiled natively. */ +export const NATIVE_HOST_ALIAS_CONFLICT_REFUSAL = + "Native host alias selection conflicts with declared configuration"; + +/** + * Where a project-wide alias comes from, so a refusal can point at the file to + * change. `inherited` is the main checkout's checkout-local map, which a linked + * worktree inherits unless it hides the entry. + */ +export type NativeHostAliasSource = "project" | "checkout" | "inherited"; + +const PRINTABLE_SERVICE = /^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$/; +/** Compose service names are printed only in their validated form. */ +function serviceLabel(name: string): string { + return PRINTABLE_SERVICE.test(name) ? `service ${name}` : "a service"; +} + +function describeSource(source: NativeHostAliasSource | undefined): string { + switch (source) { + case "project": + return "internal.extra_hosts in the project configuration"; + case "checkout": + return "this checkout's .hack/.internal/extra-hosts.json"; + case "inherited": + return "the main checkout's .hack/.internal/extra-hosts.json, which this linked worktree inherits"; + default: + return "the service's Compose extra_hosts"; + } +} + +function removalHint( + alias: string, + source: NativeHostAliasSource | undefined +): string { + switch (source) { + case "project": + return `remove ${alias} from internal.extra_hosts in the project configuration`; + case "checkout": + return `run \`hack internal extra-hosts unset ${alias}\``; + case "inherited": + return `run \`hack internal extra-hosts unset ${alias}\` in this worktree to stop inheriting it`; + default: + return `remove ${alias} from the service's extra_hosts`; + } +} + +type UnboundAlias = { + readonly alias: string; + readonly services: readonly string[]; + readonly source: NativeHostAliasSource | undefined; +}; + +/** At most this many aliases are listed; the rest are counted. */ +const MAX_LISTED = 8; + +/** + * Name every unbound (service, alias) pair, where each alias comes from, and + * the binding that would admit it. Aliases are validated DNS names and service + * names are printed only when they match Compose's name rule, so the line + * carries configuration names only: never targets, ports or listener values. + */ +function unbound(missing: readonly UnboundAlias[]): Error { + const listed = missing + .slice(0, MAX_LISTED) + .map( + (entry) => + `${entry.alias} for ${entry.services.map(serviceLabel).join(", ")} (from ${describeSource(entry.source)})` + ); + const more = + missing.length > MAX_LISTED + ? `; and ${missing.length - MAX_LISTED} more aliases` + : ""; + const first = missing[0]; + const aliases = JSON.stringify( + missing.slice(0, MAX_LISTED).map((entry) => entry.alias) + ); + const hint = first + ? ` Add a binding with "aliases": ${aliases} to the HACK_NATIVE_DEPENDENCIES selection (a version 2 entry without "services" applies to every service that declares them), or ${removalHint(first.alias, first.source)}.` + : ""; return new HackCliError({ code: "E_NATIVE_PROJECT_UNSUPPORTED", - message: NATIVE_UNBOUND_HOST_ALIAS_REFUSAL, + message: `${NATIVE_UNBOUND_HOST_ALIAS_REFUSAL} Unbound: ${listed.join("; ")}${more}.${hint}`, }); } function unsupported(): Error { @@ -50,8 +131,21 @@ function unsupported(): Error { function conflict(): Error { return new HackCliError({ code: "E_CONFIG_INVALID", - message: - "Native host alias selection conflicts with declared configuration; values omitted.", + message: `${NATIVE_HOST_ALIAS_CONFLICT_REFUSAL}; values omitted.`, + }); +} +/** A target other than `host-gateway` for a project-wide alias; the target is omitted. */ +function conflictingTarget(service: string, alias: string): Error { + return new HackCliError({ + code: "E_CONFIG_INVALID", + message: `${NATIVE_HOST_ALIAS_CONFLICT_REFUSAL}: ${serviceLabel(service)} maps project-wide host alias ${alias} to another target; native supports only host-gateway for it. Target value omitted.`, + }); +} +/** Two declarations compile to one runtime alias, which native execution refuses. */ +function duplicateAlias(service: string, alias: string): Error { + return new HackCliError({ + code: "E_CONFIG_INVALID", + message: `${NATIVE_HOST_ALIAS_CONFLICT_REFUSAL}: ${serviceLabel(service)} declares host alias ${alias} more than once after Compose interpolation and name normalization; keep one declaration. Nothing was started.`, }); } @@ -70,12 +164,11 @@ function exactAlias(value: string): boolean { const ASCII = /^[\x20-\x7e]*$/; /** - * The name native review compiles a service-declared extra host to, with the + * The name native review compiles a resolved extra host name to, with the * runtime's `normalize_hostname` semantics: strip one trailing dot, lowercase * ASCII, and require an exact DNS name (not an address). Returns null for a - * name this check cannot canonicalize (for example an `${...}` reference the - * runtime resolves later); native admission still enforces exact alias and - * binding equality for those. + * name this check cannot canonicalize; native admission still enforces exact + * alias and binding equality for those. */ function canonicalHostName(value: string): string | null { if (!ASCII.test(value)) { @@ -93,10 +186,87 @@ function canonicalHostName(value: string): string | null { return name; } +/** + * The runtime alias one declared `extra_hosts` entry compiles to: Compose + * interpolation as native execution resolves it (see + * {@link resolveNativeComposeText}), then name canonicalization. Null when + * the value is not knowable before execution or is not an exact DNS name. + */ +function compiledHostName( + value: string, + managed: ReadonlySet +): string | null { + const resolved = resolveNativeComposeText({ text: value, managed }); + return resolved === null ? null : canonicalHostName(resolved); +} + +/** A declared target as native execution resolves it; null when unknown. */ +function compiledTarget( + value: string, + managed: ReadonlySet +): string | null { + return resolveNativeComposeText({ text: value, managed }); +} + +/** + * The (compiled name, compiled target) of each declaration that resolves + * before execution, refusing a service that declares one alias twice: the + * runtime refuses that only after the VM boots. Unknown declarations are left + * to review and execution, keyed by their literal text. + */ +function compiledEntries(opts: { + readonly service: string; + readonly entries: readonly (readonly [string, string])[]; + readonly managed: ReadonlySet; +}): Map { + const declared = new Map(); + for (const [name, target] of opts.entries) { + const compiled = compiledHostName(name, opts.managed); + if (compiled !== null && declared.has(compiled)) { + throw duplicateAlias(opts.service, compiled); + } + declared.set(compiled ?? name, compiledTarget(target, opts.managed)); + } + return declared; +} + +function managedNames( + managedEnvironment: + | Readonly>>> + | undefined, + service: string +): ReadonlySet { + const values = managedEnvironment + ? Object.getOwnPropertyDescriptor(managedEnvironment, service)?.value + : undefined; + return new Set(isRecord(values) ? Object.keys(values) : []); +} + +/** + * The first `:` or `=` outside `{...}`, exactly as native review splits a list + * entry, so the `:` of an interpolation such as `${PREFIX:+x-}` stays in the + * name. -1 when there is none. + */ +function separatorIndex(entry: string): number { + let depth = 0; + for (let index = 0; index < entry.length; index += 1) { + const c = entry[index]; + if (c === "{") { + depth += 1; + } else if (c === "}") { + depth = Math.max(0, depth - 1); + } else if ((c === ":" || c === "=") && depth === 0) { + return index; + } + } + return -1; +} + /** * Split one Compose `extra_hosts` value into (name, target) pairs, using the - * first `:` or `=` like the native review does. Returns undefined for a shape - * native review would refuse; callers leave that refusal to review. + * first `:` or `=` outside braces like the native review does. Returns + * undefined for a shape native review would refuse; callers leave that + * refusal to review. */ function extraHostEntries( value: unknown @@ -122,11 +292,11 @@ function extraHostEntries( if (typeof entry !== "string") { return undefined; } - const match = SEPARATOR.exec(entry); - if (!match) { + const index = separatorIndex(entry); + if (index < 0) { return undefined; } - entries.push([entry.slice(0, match.index), entry.slice(match.index + 1)]); + entries.push([entry.slice(0, index), entry.slice(index + 1)]); } return entries; } @@ -145,27 +315,31 @@ function internalAliasNames(hosts: Readonly>): string[] { return names.sort(); } -function declareOnService( - service: Record, - aliases: readonly string[] -): void { +function declareOnService(opts: { + readonly name: string; + readonly service: Record; + readonly aliases: readonly string[]; + readonly managed: ReadonlySet; +}): void { + const { service, aliases } = opts; const entries = extraHostEntries(service.extra_hosts); if (!entries) { throw conflict(); } // Compare by the compiled name: `API.Example.` and `api.example` are one - // runtime alias, so adding the second spelling would duplicate it. - const declared = new Map(); - for (const [name, target] of entries) { - declared.set(canonicalHostName(name) ?? name, target); - } + // runtime alias, and so is `${UNSET:+x-}api.example` once execution + // resolves it; adding a second spelling would duplicate the alias. + const declared = compiledEntries({ + service: opts.name, + entries, + managed: opts.managed, + }); const added: string[] = []; for (const alias of aliases) { - const target = declared.get(alias); - if (target === undefined) { + if (!declared.has(alias)) { added.push(alias); - } else if (target !== "host-gateway") { - throw conflict(); + } else if (declared.get(alias) !== "host-gateway") { + throw conflictingTarget(opts.name, alias); } } if (added.length === 0) { @@ -204,28 +378,54 @@ function declareOnService( export function applyNativeInternalHostAliases(opts: { readonly input: NativeProjectInput; readonly hosts?: Readonly>; + /** Where each project-wide alias comes from, for refusal text only. */ + readonly sources?: Readonly>; readonly hasDependencySelection: boolean; }): NativeProjectInput { if (opts.hosts === undefined || Object.keys(opts.hosts).length === 0) { return opts.input; } const aliases = internalAliasNames(opts.hosts); - if (!opts.hasDependencySelection) { - throw unbound(); - } const compose: unknown = JSON.parse(opts.input.normalizedComposeJson); if (!(isRecord(compose) && isRecord(compose.services))) { throw conflict(); } - for (const service of Object.values(compose.services)) { + if (!opts.hasDependencySelection) { + const services = Object.keys(compose.services).sort(); + throw unbound( + aliases.map((alias) => ({ + alias, + services, + source: sourceOf(opts.sources, alias), + })) + ); + } + for (const [name, service] of Object.entries(compose.services)) { if (!isRecord(service)) { throw conflict(); } - declareOnService(service, aliases); + declareOnService({ + name, + service, + aliases, + managed: managedNames(opts.input.managedEnvironment, name), + }); } return { ...opts.input, normalizedComposeJson: JSON.stringify(compose) }; } +function sourceOf( + sources: Readonly> | undefined, + alias: string +): NativeHostAliasSource | undefined { + const value: unknown = sources + ? Object.getOwnPropertyDescriptor(sources, alias)?.value + : undefined; + return value === "project" || value === "checkout" || value === "inherited" + ? value + : undefined; +} + function activeService( spec: Readonly>, profiles: readonly string[] @@ -245,17 +445,23 @@ function activeService( /** * The `host-gateway` aliases each profile-active service declares, keyed by - * service, in the canonical form native review compiles them to (see - * {@link canonicalHostName}), so a declared `API.Example.` matches a binding - * for `api.example`. Inactive services are never scheduled and receive no - * bindings, so they need none. Malformed or not-yet-resolved declarations are - * left to review and admission. + * service, in the form native execution compiles them to: Compose + * interpolation resolved as the runtime resolves it, then canonicalized (see + * {@link compiledHostName}), so a declared `API.Example.` or + * `${UNSET:-api.example}` matches a binding for `api.example`. A service that + * declares one alias twice refuses here, before any runtime work. Inactive + * services are never scheduled and receive no bindings, so they need none. + * Malformed or not-yet-knowable declarations are left to review and admission. */ export function nativeServiceHostAliases(opts: { readonly services: Readonly< Record>> >; readonly profiles: readonly string[]; + /** Names delivered privately per service; references to them are unknown. */ + readonly managedEnvironment?: Readonly< + Record>> + >; }): Record { const result: Record = {}; for (const [name, spec] of Object.entries(opts.services)) { @@ -263,9 +469,12 @@ export function nativeServiceHostAliases(opts: { continue; } const aliases = new Set(); - for (const [alias, target] of extraHostEntries(spec.extra_hosts) ?? []) { + const entries = extraHostEntries(spec.extra_hosts) ?? []; + const managed = managedNames(opts.managedEnvironment, name); + const declared = compiledEntries({ service: name, entries, managed }); + for (const [alias, target] of declared) { const canonical = canonicalHostName(alias); - if (target === "host-gateway" && canonical !== null) { + if (target === "host-gateway" && canonical === alias) { aliases.add(canonical); } } @@ -282,25 +491,38 @@ export function nativeServiceHostAliases(opts: { /** * Fail closed when an active service declares a host-gateway alias that no * selected binding of that service covers. Native admission later enforces - * exact equality as well; this refuses earlier, before runtime effects, with - * a value-free line. + * exact equality as well; this refuses earlier, before runtime effects. The + * refusal names every unbound (service, alias) pair and where each alias + * comes from, but never listener values. */ export function requireNativeHostAliasBindings(opts: { readonly dependencies: readonly NativeHostDependency[]; readonly hostAliases: Readonly>; + /** Where each project-wide alias comes from, for refusal text only. */ + readonly sources?: Readonly>; }): void { const bound = new Set( opts.dependencies.flatMap((dependency) => dependency.aliases.map((alias) => `${dependency.service}\0${alias}`) ) ); + const missing = new Map(); for (const [service, aliases] of Object.entries(opts.hostAliases)) { for (const alias of aliases) { if (!bound.has(`${service}\0${alias}`)) { - throw unbound(); + missing.set(alias, [...(missing.get(alias) ?? []), service]); } } } + if (missing.size > 0) { + throw unbound( + [...missing.keys()].sort().map((alias) => ({ + alias, + services: (missing.get(alias) ?? []).sort(), + source: sourceOf(opts.sources, alias), + })) + ); + } } /** Fixed text when lifecycle hooks changed project-wide aliases after review input. */ @@ -335,18 +557,49 @@ export async function resolveNativeInternalExtraHosts(opts: { readonly project: ProjectContext; readonly cfg?: ProjectConfig; }): Promise> { + return (await resolveNativeInternalExtraHostSelection(opts)).hosts; +} + +/** + * {@link resolveNativeInternalExtraHosts} plus where each effective alias + * comes from: `internal.extra_hosts` in project configuration, this + * checkout's local map, or the main checkout's local map a linked worktree + * inherits. Sources are used only to explain refusals. + */ +export async function resolveNativeInternalExtraHostSelection(opts: { + readonly project: ProjectContext; + readonly cfg?: ProjectConfig; +}): Promise<{ + readonly hosts: Record; + readonly sources: Record; +}> { try { const cfg = opts.cfg ?? (await readProjectConfig(opts.project)); if (cfg.parseError) { throw new Error("invalid project configuration"); } - return ( - await resolveInternalExtraHosts({ - projectRoot: opts.project.projectRoot, - projectDir: opts.project.projectDir, - staticHosts: cfg.internal?.extraHosts, - }) - ).hosts; + const resolved = await resolveInternalExtraHosts({ + projectRoot: opts.project.projectRoot, + projectDir: opts.project.projectDir, + staticHosts: cfg.internal?.extraHosts, + }); + const local = resolveExtraHostsPath(opts.project.projectDir); + const sources: Record = {}; + for (const [alias, origin] of Object.entries(resolved.origins)) { + let source: NativeHostAliasSource = "inherited"; + if (origin === "internal.extra_hosts") { + source = "project"; + } else if (origin === local) { + source = "checkout"; + } + Object.defineProperty(sources, alias, { + value: source, + enumerable: true, + writable: true, + configurable: true, + }); + } + return { hosts: resolved.hosts, sources }; } catch { throw new HackCliError({ code: "E_CONFIG_INVALID", diff --git a/src/backends/native-project-process.ts b/src/backends/native-project-process.ts index 1ae88872e..9167ecefb 100644 --- a/src/backends/native-project-process.ts +++ b/src/backends/native-project-process.ts @@ -1,6 +1,7 @@ import { isAbsolute } from "node:path"; import { ClosedCliError } from "../lib/cli-result.ts"; import { isRecord } from "../lib/guards.ts"; +import type { NativeProcessRefusal } from "../lib/native-process-wrappers.ts"; import { type NativeAuthoredReceipt, type NativeAuthoredReview, @@ -30,6 +31,8 @@ export type NativeExitDiagnostic = { readonly nativeCode?: string; /** Bounded cause the runtime allows with `nativeCode`, such as a wrapped stopped runtime. */ readonly nativeCauseCode?: string; + /** Every service and closed field a process wrapper refused; value-free. */ + readonly processRefusals?: readonly NativeProcessRefusal[]; }; type ProcessOptions = { @@ -481,6 +484,7 @@ async function serveGraphProcess( const failure = await failureCodes; const nativeCode = failure?.code; const nativeCauseCode = failure?.causeCode; + const processRefusals = failure?.processRefusals; clearTimeout(drainTimer); drain.abort(); try { @@ -488,6 +492,7 @@ async function serveGraphProcess( exitCode, ...(nativeCode ? { nativeCode } : {}), ...(nativeCauseCode ? { nativeCauseCode } : {}), + ...(processRefusals ? { processRefusals } : {}), }); } catch { // Observation must never replace the startup result or cleanup authority. diff --git a/src/backends/native-project-restart-preflight.ts b/src/backends/native-project-restart-preflight.ts index cc68860a6..9cfe15944 100644 --- a/src/backends/native-project-restart-preflight.ts +++ b/src/backends/native-project-restart-preflight.ts @@ -4,6 +4,7 @@ import { tmpdir } from "node:os"; import { dirname, join } from "node:path"; import { ClosedCliError } from "../lib/cli-result.ts"; import { isRecord } from "../lib/guards.ts"; +import { requireWrappableNativeProcesses } from "../lib/native-process-wrappers.ts"; import { adaptNativeAwsEnvironment } from "./native-aws-environment.ts"; import type { NativeDependencyOwnerResolver } from "./native-dependency-owners.ts"; import { adaptNativeGcloudEnvironment } from "./native-gcloud-environment.ts"; @@ -30,6 +31,7 @@ import { prepareNativeProjectInput } from "./native-project-input.ts"; import { inspectNativeProjectGraph } from "./native-project-inspect.ts"; import { applyNativeInternalHostAliases, + type NativeHostAliasSource, nativeServiceHostAliases, requireNativeHostAliasBindings, } from "./native-project-internal-hosts.ts"; @@ -378,6 +380,10 @@ export async function preflightNativeRestart(opts: { readonly internalExtraHosts?: Readonly>; /** Resolves owner-scoped dependency selections against the running instance. */ readonly dependencyOwners?: NativeDependencyOwnerResolver; + /** Where each project-wide alias comes from, for refusal text only. */ + readonly internalExtraHostSources?: Readonly< + Record + >; readonly allowedHosts?: readonly string[]; /** Same public metadata selection the replacement startup will review. */ readonly runtimeMetadata?: NativeRuntimeMetadataSelection; @@ -459,6 +465,7 @@ export async function preflightNativeRestart(opts: { input = applyNativeInternalHostAliases({ input, hosts: opts.internalExtraHosts, + sources: opts.internalExtraHostSources, hasDependencySelection: opts.dependencyFile !== undefined, }); const reviewedBranch = await prepareNativeReviewBranch({ @@ -512,6 +519,7 @@ export async function preflightNativeRestart(opts: { const hostAliases = nativeServiceHostAliases({ services: specs, profiles: selected.profiles, + managedEnvironment: input.managedEnvironment, }); const dependencies = stoppedRetry ? await readNativeRestartDependencyIntent({ @@ -541,8 +549,21 @@ export async function preflightNativeRestart(opts: { invoke: deps.invoke, }), }); - requireNativeHostAliasBindings({ dependencies, hostAliases }); + requireNativeHostAliasBindings({ + dependencies, + hostAliases, + sources: opts.internalExtraHostSources, + }); prepareNativeDependencyServices({ dependencies, services: specs }); + // The replacement startup would refuse these before its VM work; refuse + // before the current graph is stopped. + requireWrappableNativeProcesses({ + services: specs, + profiles: selected.profiles, + managedEnvironment: input.managedEnvironment, + managedFiles: input.managedFiles, + dependencyServices: dependencies.map((binding) => binding.service), + }); const activeImages = stoppedRetry ? undefined : await deps.activeImages({ diff --git a/src/backends/native-project-restore.ts b/src/backends/native-project-restore.ts index 42d532b70..db77b873a 100644 --- a/src/backends/native-project-restore.ts +++ b/src/backends/native-project-restore.ts @@ -88,8 +88,15 @@ export async function verifyNativeSourceCompatibility(opts: { `Native shared-source compatibility changed; ${NOT_ADOPTED}` ); } - const cached = nativeCacheInitializers(opts.review.report.plan).size > 0; - if ((result.source_revision === null) !== !cached) { + // Reconciliation recreates compute exactly as a fresh admission: it never + // names a retained cache publication, so a cached review publishes afresh. + const cached = + result.reconcile !== true && + nativeCacheInitializers(opts.review.report.plan).size > 0; + if ( + (result.reconcile !== undefined && result.reconcile !== true) || + (result.source_revision === null) !== !cached + ) { throw new ClosedCliError( `Native shared-source compatibility changed; ${NOT_ADOPTED}` ); diff --git a/src/backends/native-project-retained.ts b/src/backends/native-project-retained.ts index 577d49d29..91a50d8ff 100644 --- a/src/backends/native-project-retained.ts +++ b/src/backends/native-project-retained.ts @@ -44,15 +44,7 @@ export function confirmedNativeRetainedGraph( return false; } expected.add(key); - const observed = observations[key]; - // A `reserved` volume never received a create intent: it holds no data and - // must be absent. Every other volume is settled retained data. - const settledData = - resource.kind === "volume" && resource.phase !== "reserved"; - if ( - !isRecord(observed) || - observed.state !== (settledData ? "present" : "absent") - ) { + if (!observedAsRecorded(resource, observations[key])) { return false; } } @@ -61,3 +53,27 @@ export function confirmedNativeRetainedGraph( Object.keys(observations).every((key) => expected.has(key)) ); } + +/** + * Compute must be absent and settled data present. A `reserved` volume never + * received a create intent: it holds no data and must be absent. A reserved + * dependency cache is shared and content-addressed: admission adopts an + * existing one by labels or creates it, so either observation is valid. + */ +function observedAsRecorded( + resource: Record, + observed: unknown +): boolean { + if (!isRecord(observed)) { + return false; + } + if (resource.kind !== "volume") { + return observed.state === "absent"; + } + if (resource.phase !== "reserved") { + return observed.state === "present"; + } + return isRecord(resource.cache) + ? observed.state === "present" || observed.state === "absent" + : observed.state === "absent"; +} diff --git a/src/backends/native-project-start.ts b/src/backends/native-project-start.ts index bd9f07ef3..1f022cab7 100644 --- a/src/backends/native-project-start.ts +++ b/src/backends/native-project-start.ts @@ -8,6 +8,11 @@ import { withComposeServiceEnvFingerprints, } from "../lib/compose-env-fingerprint.ts"; import { isRecord } from "../lib/guards.ts"; +import { + describeNativeProcessRefusals, + type NativeProcessRefusal, + requireWrappableNativeProcesses, +} from "../lib/native-process-wrappers.ts"; import { adaptNativeAwsEnvironment } from "./native-aws-environment.ts"; import { scheduleNativeAwsExpiryWarnings } from "./native-aws-expiry.ts"; import type { NativeDependencyOwnerResolver } from "./native-dependency-owners.ts"; @@ -76,6 +81,7 @@ import { import { inspectNativeProjectGraph } from "./native-project-inspect.ts"; import { applyNativeInternalHostAliases, + type NativeHostAliasSource, nativeServiceHostAliases, requireNativeHostAliasBindings, } from "./native-project-internal-hosts.ts"; @@ -733,6 +739,74 @@ export async function verifyHttpsRoutes( } } +const PRINTABLE_NAME = /^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$/; +const PRINTABLE_HOST = /^[a-z0-9](?:[a-z0-9.-]{0,251}[a-z0-9])?$/; +/** + * What a ready foreground graph serves on this host, from the reviewed plan + * only: each routed service's reviewed hostnames as HTTPS URLs when the + * native HTTPS frontend runs (with its port unless 443), and the published + * loopback ports. Names that fail their closed patterns are skipped. + */ +export function nativeReadySummary(opts: { + readonly plan: unknown; + readonly routed: ReadonlySet; + readonly httpsPort: number | null; +}): string[] { + const lines: string[] = []; + const services = + isRecord(opts.plan) && isRecord(opts.plan.services) + ? opts.plan.services + : {}; + const suffix = + opts.httpsPort === null || opts.httpsPort === 443 + ? "" + : `:${opts.httpsPort}`; + let unserved = false; + for (const name of [...opts.routed].sort()) { + const service = services[name]; + const hostnames = + isRecord(service) && + isRecord(service.routing) && + Array.isArray(service.routing.hostnames) + ? service.routing.hostnames.filter( + (host): host is string => + typeof host === "string" && PRINTABLE_HOST.test(host) + ) + : []; + if (!PRINTABLE_NAME.test(name) || hostnames.length === 0) { + continue; + } + if (opts.httpsPort === null) { + unserved = true; + continue; + } + lines.push( + ` ${name}: ${hostnames.map((host) => `https://${host}${suffix}`).join(" ")}` + ); + } + if (unserved) { + lines.push( + " Routes are not served on this host: the native HTTPS frontend is not selected (HACK_NATIVE_CADDY_BINARY, HACK_NATIVE_CADDY_SHA256 and HACK_NATIVE_HTTPS_PORT)." + ); + } + const ports = reviewedNativeLoopbackPorts(opts.plan); + if (ports.length > 0) { + lines.push( + ` Published ports: ${ports.map((port) => `127.0.0.1:${port}`).join(" ")}` + ); + } + return lines; +} +/** One confirmed-stop line for an interrupted ready graph, from its final receipt. */ +export function nativeStoppedNotice(final: unknown): string { + const removed = + isRecord(final) && + isRecord(final.receipt) && + final.receipt.phase === "removed"; + return removed + ? "Native project stopped; owned cleanup confirmed and no runtime data was retained." + : "Native project stopped; owned cleanup confirmed. Named volumes and the stopped run mapping are kept for the next hack up."; +} function foregroundExitCode(opts: { aborted: boolean; interruptedExit: number; @@ -905,41 +979,79 @@ function cleanupUnconfirmed( `Native foreground exit cleanup is unconfirmed; runtime and bridge state may be retained. Any published run mapping is retained; inspect owned state before retrying.${startupDiagnostic(startupFailure)}${nativeCode ? ` Native exit diagnostic: ${nativeCode}.` : ""}${recorded}` ); } +/** Runtime process-wrapper refusals: service names and closed fields only. */ +function refusedProcesses( + refusals: readonly NativeProcessRefusal[] | undefined +): string { + return refusals && refusals.length > 0 + ? ` Native cannot wrap these services' processes: ${describeNativeProcessRefusals(refusals)}.` + : ""; +} +/** + * A fresh startup whose graph owner exited before reserving any runtime state. + * `graph inspect` refuses `graph_unreserved` only when the run's journals are + * exactly absent and no foreground owner is published; deps.serve settles only + * after the owned child exited, so nothing can still reserve state. + */ +const GRAPH_UNRESERVED = "graph_unreserved"; /** - * The settled foreground start failed and fresh inspection proves this run never - * reserved runtime state (no journal, no published owner). Nothing is retained, - * so report the startup failure itself instead of uncertain cleanup. + * The report when final inspection of a settled startup fails. For a fresh run + * that failed, never attempted a mapping and whose inspection reports + * {@link GRAPH_UNRESERVED}, nothing was created: report the startup failure + * itself (with its runtime reason) and mark graph cleanup confirmed. Every + * other case keeps the unconfirmed-cleanup report. */ -function unreservedStartupFailure( - startupFailure: unknown, - inspectionFailure: unknown -): Error | undefined { +function finalInspectionFailure(opts: { + readonly serveFailure: unknown; + readonly inspectionFailure: unknown; + readonly restored: boolean; + readonly mappingAttempted: boolean; + readonly nativeCode: string | undefined; + readonly processRefusals?: readonly NativeProcessRefusal[]; +}): { readonly error: Error; readonly nothingReserved: boolean } { if ( - startupFailure === undefined || - !( - inspectionFailure instanceof NativeRuntimeRequestError && - inspectionFailure.nativeCode === "graph_unreserved" - ) + opts.serveFailure === undefined || + opts.restored || + opts.mappingAttempted || + !(opts.inspectionFailure instanceof NativeRuntimeRequestError) || + opts.inspectionFailure.nativeCode !== GRAPH_UNRESERVED ) { - return; + return { + error: cleanupUnconfirmed( + opts.serveFailure ?? opts.inspectionFailure, + opts.nativeCode + ), + nothingReserved: false, + }; } - return new ClosedCliError( - `Native graph startup failed before any runtime state was reserved; nothing was retained.${startupDiagnostic(startupFailure)}` - ); + return { + error: new ClosedCliError( + `Native graph startup failed before any runtime state was reserved; nothing was created and no cleanup is needed.${startupDiagnostic(opts.serveFailure)}${refusedProcesses(opts.processRefusals)}` + ), + nothingReserved: true, + }; } -/** - * Error for a failed final inspection. deps.serve settles only after the owned - * child exited, so the consumer that could still reserve state is gone. - */ -function finalInspectionFailure( - serveFailure: unknown, - inspectionFailure: unknown, - nativeExitCode: string | undefined -): Error { - return ( - unreservedStartupFailure(serveFailure, inspectionFailure) ?? - cleanupUnconfirmed(serveFailure ?? inspectionFailure, nativeExitCode) - ); +/** The ready line's event: run id and what the ready graph serves on this host. */ +function readyProgress(opts: { + readonly services: readonly string[]; + readonly run: string; + readonly plan: unknown; + readonly routed: ReadonlySet; + readonly httpsPort: number | null; +}): NativeStartupProgress { + return { + phase: "ready", + services: opts.services, + run: RUN.test(opts.run) ? opts.run : "unknown", + endpoints: nativeReadySummary(opts), + }; +} +/** The HTTPS port routes are served on: only while the frontend lease runs. */ +function servedHttpsPort( + lease: unknown, + selection: NativeHttpsSelection | undefined +): number | null { + return lease === undefined ? null : (selection?.httpsPort ?? null); } function requireConfirmedCleanup( final: unknown, @@ -1395,12 +1507,24 @@ export type NativeStartupProgress = | "runtime" | "images" | "all-stopped" - | "stopping" - | "stopped"; + | "stopping"; + } + | { + readonly phase: "stopped"; + /** The confirmed-stop retention line when a run mapping was published. */ + readonly summary?: string; + } + | { + readonly phase: "services"; + readonly services: readonly string[]; } | { - readonly phase: "services" | "ready"; + readonly phase: "ready"; readonly services: readonly string[]; + /** The run id, or `unknown` when it fails the run-id pattern. */ + readonly run: string; + /** Indented route, unserved-route and published-port lines. */ + readonly endpoints: readonly string[]; } | ({ readonly phase: "service-exited" } & NativeServiceExit); @@ -1447,13 +1571,20 @@ function watchServiceExits(opts: { } /** After confirmed cleanup, an interrupted foreground graph has stopped. */ -function reportStopped( - progress: (event: NativeStartupProgress) => void, - signal: AbortSignal, - infrastructureFailure: Error | undefined -): void { - if (signal.aborted && !infrastructureFailure) { - progress({ phase: "stopped" }); +function reportStopped(opts: { + readonly progress: (event: NativeStartupProgress) => void; + readonly signal: AbortSignal; + readonly infrastructureFailure: Error | undefined; + /** Whether this run's mapping was published; only then is retention reported. */ + readonly published: boolean; + readonly final: unknown; +}): void { + if (opts.signal.aborted && !opts.infrastructureFailure) { + opts.progress( + opts.published + ? { phase: "stopped", summary: nativeStoppedNotice(opts.final) } + : { phase: "stopped" } + ); } } @@ -1483,6 +1614,10 @@ export async function startNativeProject(opts: { * override; each active pair then requires a reviewed dependency binding. */ readonly internalExtraHosts?: Readonly>; + /** Where each project-wide alias comes from, for refusal text only. */ + readonly internalExtraHostSources?: Readonly< + Record + >; readonly allowedHosts?: readonly string[]; readonly preparedBase?: NativePreparedBase; readonly https?: NativeHttpsSelection; @@ -1611,6 +1746,7 @@ export async function startNativeProject(opts: { input = applyNativeInternalHostAliases({ input, hosts: opts.internalExtraHosts, + sources: opts.internalExtraHostSources, hasDependencySelection: opts.dependencyFile !== undefined, }); input = ( @@ -1697,6 +1833,7 @@ export async function startNativeProject(opts: { const hostAliases = nativeServiceHostAliases({ services: specs, profiles, + managedEnvironment: input.managedEnvironment, }); const hostDependencies = await readNativeHostDependencies({ path: selectedDependencyFile, @@ -1732,11 +1869,20 @@ export async function startNativeProject(opts: { requireNativeHostAliasBindings({ dependencies: hostDependencies, hostAliases, + sources: opts.internalExtraHostSources, }); prepareNativeDependencyServices({ dependencies: hostDependencies, services: specs, }); + // Refuse Compose settings no process wrapper can run before the VM starts. + requireWrappableNativeProcesses({ + services: specs, + profiles, + managedEnvironment: input.managedEnvironment, + managedFiles: input.managedFiles, + dependencyServices: hostDependencies.map((binding) => binding.service), + }); await verifyNativeRetainedMapping({ run: retained, scope: opts.scope, @@ -1963,6 +2109,9 @@ export async function startNativeProject(opts: { let code = 1; let serveFailure: unknown; let nativeExitCode: string | undefined; + let nativeProcessRefusals: + | readonly NativeProcessRefusal[] + | undefined; // Redacted at receipt; the raw tail is never retained or persisted. let startupLogReport = ""; const services = Object.keys(specs).sort(); @@ -2004,6 +2153,7 @@ export async function startNativeProject(opts: { ...stop?.serve, onExitDiagnostic: (diagnostic) => { nativeExitCode = diagnostic.nativeCode; + nativeProcessRefusals = diagnostic.processRefusals; }, onStartupLogs: (logs) => { startupLogReport = formatNativeStartupLogReport({ @@ -2110,7 +2260,15 @@ export async function startNativeProject(opts: { } await hooks?.ready?.(); await opts.onReady?.(); - progress({ phase: "ready", services }); + progress( + readyProgress({ + services, + run, + plan: review.report.plan, + routed: routes.services, + httpsPort: servedHttpsPort(https, opts.https), + }) + ); stopWatching = watchServiceExits({ runtime: opts.runtime, projectRoot: opts.scope.projectRoot, @@ -2138,11 +2296,16 @@ export async function startNativeProject(opts: { try { final = await invokeInspect(); } catch (inspectionFailure) { - throw finalInspectionFailure( + const settled = finalInspectionFailure({ serveFailure, inspectionFailure, - nativeExitCode - ); + restored: restore !== undefined, + mappingAttempted, + nativeCode: nativeExitCode, + processRefusals: nativeProcessRefusals, + }); + graphCleanupConfirmed = settled.nothingReserved; + throw settled.error; } const confirmed = await confirmCleanupOrRecordOwner({ confirm: { @@ -2192,7 +2355,13 @@ export async function startNativeProject(opts: { mappingAttempted, invoke: deps.invoke, }); - reportStopped(progress, controller.signal, httpsFailure); + reportStopped({ + progress, + signal: controller.signal, + infrastructureFailure: httpsFailure, + published: mapping !== undefined, + final, + }); return foregroundExitCode({ aborted: controller.signal.aborted, interruptedExit: stopSignals.interruptedExit(), diff --git a/src/backends/native-runtime-client.ts b/src/backends/native-runtime-client.ts index 6c6c3d53e..099a2ba55 100644 --- a/src/backends/native-runtime-client.ts +++ b/src/backends/native-runtime-client.ts @@ -1,7 +1,16 @@ import { isAbsolute } from "node:path"; import { ClosedCliError, HackCliError } from "../lib/cli-result.ts"; import { isRecord } from "../lib/guards.ts"; +import { + type NativeProcessRefusal, + readNativeProcessRefusals, +} from "../lib/native-process-wrappers.ts"; import { nativeAuthoredExecOptions } from "./native-authored-exec-options.ts"; +import { + type NativePlanChange, + nativePlanChangeGuidance, + readNativePlanChange, +} from "./native-plan-change.ts"; import { NATIVE_STARTUP_FAILURE_REPLY_BYTES, type NativeStartupLogTail, @@ -27,6 +36,9 @@ interface NativeFailure { readonly service?: string; readonly stopFailures?: readonly NativeStopFailure[]; readonly startupLogs?: readonly NativeStartupLogTail[]; + /** Services and closed fields a process wrapper refused; value-free. */ + readonly processRefusals?: readonly NativeProcessRefusal[]; + readonly planChange?: NativePlanChange; } const FIXED_MESSAGE = /^[\x20-\x7e]{1,400}$/; @@ -541,7 +553,10 @@ function nativeFailureMessage(failure: NativeFailure | undefined): string { const code = failure ? ` (${failure.code}${failure.causeCode ? `: ${failure.causeCode}` : ""})` : ""; - return `Native runtime request failed${code}; inspect owned state before retrying.${failure ? nativeFailureNote(failure) : ""}${nativeStopFailureSummary(failure?.stopFailures)}`; + const guidance = failure + ? nativePlanChangeGuidance(failure.code, failure.planChange) + : undefined; + return `Native runtime request failed${code}; ${guidance ?? "inspect owned state before retrying."}${failure ? nativeFailureNote(failure) : ""}${nativeStopFailureSummary(failure?.stopFailures)}`; } function completionResponse( @@ -690,10 +705,12 @@ function nativeFailureFromReply( code: value.code, cause: value.cause_code, }); + const planChange = readNativePlanChange(value.plan_change); return { code: value.code, ...(causeCode ? { causeCode } : {}), ...nativeFailureDetail(value), + ...(planChange ? { planChange } : {}), ...(["engine_protocol", "engine_rejected", "engine_not_found"].includes( value.code === "graph_owner_recovery" && typeof value.cause_code === "string" @@ -705,8 +722,16 @@ function nativeFailureFromReply( ...(startupLogs && value.startup_logs !== undefined ? { startupLogs: readNativeStartupLogs(value.startup_logs) } : {}), + ...processRefusalsOf(value.process_refusals), }; } +function processRefusalsOf(value: unknown): { + readonly processRefusals?: readonly NativeProcessRefusal[]; +} { + const processRefusals = + value === undefined ? undefined : readNativeProcessRefusals(value); + return processRefusals ? { processRefusals } : {}; +} /** Only bounded native error codes leave this boundary, never stderr messages. */ async function readNativeFailure( @@ -800,6 +825,7 @@ export async function readNativeStartupFailure( readonly message?: string; readonly service?: string; readonly startupLogs?: readonly NativeStartupLogTail[]; + readonly processRefusals?: readonly NativeProcessRefusal[]; } | undefined > { @@ -816,6 +842,9 @@ export async function readNativeStartupFailure( ...(failure.message ? { message: failure.message } : {}), ...(failure.service ? { service: failure.service } : {}), ...(failure.startupLogs ? { startupLogs: failure.startupLogs } : {}), + ...(failure.processRefusals + ? { processRefusals: failure.processRefusals } + : {}), }; } diff --git a/src/backends/native-startup-logs.ts b/src/backends/native-startup-logs.ts index 2c8f58ce4..347b58954 100644 --- a/src/backends/native-startup-logs.ts +++ b/src/backends/native-startup-logs.ts @@ -128,8 +128,10 @@ function needles(values: Iterable): { * JSON-escaped form and each line of a multi-line value that is long enough, * with and without surrounding whitespace. Longer values are replaced first. * - A shorter value (one to three characters) is replaced only where it stands as - * a whole token, not next to a letter, digit or underscore, so a value such as - * `1` or `on` cannot shred unrelated output. + * a whole token: not next to a letter, digit or underscore, and not joined to + * one by a `.` or `-`. So a value such as `1`, `0` or `on` cannot shred + * unrelated output such as a version string (`8.0.46`), a decimal or a + * hyphenated word, while a value ending a sentence (`mode on.`) still matches. * - Remaining control characters except tab and newline become U+FFFD. */ export function redactNativeStartupLog( @@ -143,7 +145,7 @@ export function redactNativeStartupLog( } for (const value of tokens) { const pattern = new RegExp( - `(? + checkNativeProcessWrappers({ startDir: opts.startDir }) + ) + ); } progress.stop(`Native doctor checks complete (${results.length} checks)`); const hasError = results.some((result) => result.status === "error"); @@ -1385,6 +1399,8 @@ export type DoctorJsonCheck = { readonly amd64Emulation?: Amd64EmulationReport; /** Present only on the native settings check: origins and closed warnings. */ readonly nativeSettings?: NativeSettingsReport; + /** Present only on the native process-wrapper check: service names and closed fields. */ + readonly nativeProcess?: NativeProcessReport; }; export type DoctorJsonData = { @@ -1430,6 +1446,7 @@ export function buildDoctorJsonData(opts: { ...(result.nativeSettings ? { nativeSettings: result.nativeSettings } : {}), + ...(result.nativeProcess ? { nativeProcess: result.nativeProcess } : {}), })), summary: buildDoctorSummaryLines({ results: opts.results }), counts, @@ -3162,6 +3179,36 @@ async function checkNativeProjectSettings(opts: { return await checkNativeSettings({ project: ctx, cfg, env: process.env }); } +/** + * Predicts the Compose-declared settings native up's process wrappers refuse + * (HACK-1939), for the services it may wrap: every service when project-wide + * host aliases exist, services with host-gateway extra hosts, and services + * with bare environment keys. Read-only and value-free; image-inherited + * settings are checked by native up against the pinned image. + */ +async function checkNativeProcessWrappers(opts: { + readonly startDir: string; +}): Promise { + const ctx = await findProjectContext(opts.startDir); + if (!ctx) { + return { + name: NATIVE_PROCESS_CHECK, + status: "warn", + message: `Skipped (no ${HACK_PROJECT_DIR_PRIMARY}/ found)`, + }; + } + const cfg = await readProjectConfig(ctx); + const aliases = await resolveInternalExtraHosts({ + projectRoot: ctx.projectRoot, + projectDir: ctx.projectDir, + staticHosts: cfg.internal?.extraHosts, + }).catch(() => ({ hosts: {} })); + return nativeProcessCheckFromCompose({ + compose: await readNativeComposeDocument(ctx.composeFile), + projectWideAliases: Object.keys(aliases.hosts).length > 0, + }); +} + /** * Read-only key-strength and exposure audit (HACK-1127). `--fix` never acts on * it; each finding suggests the exact command instead. It also catches a key diff --git a/src/commands/project.ts b/src/commands/project.ts index fb03ce218..582c0ecc0 100644 --- a/src/commands/project.ts +++ b/src/commands/project.ts @@ -50,7 +50,7 @@ import { } from "../backends/native-project-down.ts"; import { requireNativeProjectInternalExtraHostsUnchanged, - resolveNativeInternalExtraHosts, + resolveNativeInternalExtraHostSelection, } from "../backends/native-project-internal-hosts.ts"; import { adoptNativeLifecycleCleanup } from "../backends/native-project-lifecycle.ts"; import { @@ -5862,7 +5862,10 @@ function reportNativeStartupProgress(event: NativeStartupProgress): void { return; case "ready": logger.success({ - message: `Project is ready (${event.services.join(", ")}). Use hack ps and hack logs from another terminal; press Ctrl-C to stop.`, + message: [ + `Project is ready (${event.services.join(", ")}; run ${event.run}). Use hack ps and hack logs from another terminal; press Ctrl-C to stop.`, + ...event.endpoints, + ].join("\n"), }); return; case "service-exited": @@ -5879,7 +5882,7 @@ function reportNativeStartupProgress(event: NativeStartupProgress): void { }); return; case "stopped": - logger.info({ message: "Native project stopped." }); + logger.info({ message: event.summary ?? "Native project stopped." }); return; default: return; @@ -5968,10 +5971,8 @@ async function handleNativeUp({ }; const envFingerprints = createNativeEnvFingerprintSession(envFingerprintScope); - const internalExtraHosts = await resolveNativeInternalExtraHosts({ - project, - cfg, - }); + const { hosts: internalExtraHosts, sources: internalExtraHostSources } = + await resolveNativeInternalExtraHostSelection({ project, cfg }); const startup: Parameters[0] = { runtime: native, scope: { @@ -5994,6 +5995,7 @@ async function handleNativeUp({ adaptationFile: adaptation.file, adaptationFileByEnv: adaptation.fileByEnv, internalExtraHosts, + internalExtraHostSources, allowedHosts: host.allowedHosts, preparedBase: host.preparedBase, https: host.https, @@ -6195,6 +6197,7 @@ async function restartNativeImageProject({ dependencyFileByEnv: startup.dependencyFileByEnv, internalExtraHosts: startup.internalExtraHosts, dependencyOwners: startup.dependencyOwners, + internalExtraHostSources: startup.internalExtraHostSources, allowedHosts: startup.allowedHosts, runtimeMetadata: startup.runtimeMetadata, httpsPort: startup.https?.httpsPort ?? null, diff --git a/src/lib/native-process-wrappers.ts b/src/lib/native-process-wrappers.ts new file mode 100644 index 000000000..5d4c7087c --- /dev/null +++ b/src/lib/native-process-wrappers.ts @@ -0,0 +1,533 @@ +import { resolveNativeComposeText } from "../backends/native-compose-interpolation.ts"; +import { HackCliError } from "./cli-result.ts"; +import { isRecord } from "./guards.ts"; + +/** + * Native wraps some services' processes: private environment delivery runs + * the selected process through a launcher, and host-dependency services run it + * behind a release barrier. Both run exactly what Docker would (image or + * Compose `entrypoint` + `command`, bare executable names resolved through the + * container's PATH, shell-form health checks through `/bin/sh -c`), but a few + * settings still cannot be wrapped. Native up checks every wrapped service + * before any runtime state is reserved and names each refused service and + * field; this module predicts the Compose-declared cases before `up` starts + * anything, and parses and describes the runtime's named refusals. + * + * Settings inherited from an image (its ENTRYPOINT, CMD or USER) are not read + * here; native up checks those against the pinned image and reports every + * refused service at once. + */ +export const NATIVE_PROCESS_FIELDS = [ + "entrypoint", + "healthcheck", + "user", + "stop_signal", + "mounts", +] as const; + +export type NativeProcessField = (typeof NATIVE_PROCESS_FIELDS)[number]; + +/** One refused (service, field) pair; value-free. */ +export type NativeProcessRefusal = { + readonly service: string; + readonly field: NativeProcessField; +}; + +const SERVICE = /^[A-Za-z0-9][A-Za-z0-9._-]{0,127}$/; +const MAX_REFUSALS = 32; + +/** + * Accept only the runtime's versioned, bounded `process_refusals` contract; + * anything else is dropped whole. + */ +export function readNativeProcessRefusals( + value: unknown +): readonly NativeProcessRefusal[] | undefined { + if ( + !isRecord(value) || + Object.keys(value).length !== 2 || + value.version !== 1 || + !Array.isArray(value.refusals) || + value.refusals.length < 1 || + value.refusals.length > MAX_REFUSALS + ) { + return; + } + const result: NativeProcessRefusal[] = []; + for (const entry of value.refusals) { + if ( + !isRecord(entry) || + Object.keys(entry).length !== 2 || + typeof entry.service !== "string" || + !SERVICE.test(entry.service) || + !NATIVE_PROCESS_FIELDS.includes(entry.field as NativeProcessField) + ) { + return; + } + result.push({ + service: entry.service, + field: entry.field as NativeProcessField, + }); + } + return result; +} + +const FIELD_LABELS: Readonly> = { + entrypoint: "entrypoint or command", + healthcheck: "healthcheck test", + user: "user", + stop_signal: "stop_signal", + mounts: "volume target", +}; + +const FIELD_ADVICE: Readonly> = { + entrypoint: "the process must name an executable", + healthcheck: "use an exec-form CMD test that names an executable", + user: "use a numeric UID or UID:GID", + stop_signal: "use SIGTERM, SIGINT, SIGQUIT or SIGKILL", + mounts: "a volume target overlaps a path the wrapper reserves", +}; + +/** `service web (user: use a numeric UID or UID:GID); ...`, fields in a fixed order. */ +export function describeNativeProcessRefusals( + refusals: readonly NativeProcessRefusal[] +): string { + return refusals + .map( + (refusal) => + `service ${refusal.service} ${FIELD_LABELS[refusal.field]} (${FIELD_ADVICE[refusal.field]})` + ) + .join("; "); +} + +const SIGNALS = new Set(["SIGTERM", "SIGINT", "SIGQUIT", "SIGKILL"]); +const NUMERIC_USER = /^(\d{1,10})(?::(\d{1,10}))?$/; +const MAX_ID = 2_147_483_647; +const ENVIRONMENT_RESERVED = ["/run/hack-environment"]; +const DEPENDENCY_RESERVED = [ + "/run/hack-relay-guest", + "/run/hack-startup", + "/run/hack-dependencies", +]; + +/** A Compose string as native execution resolves it; null when not knowable. */ +function compiled(value: string): string | null { + return resolveNativeComposeText({ text: value }); +} + +function numericUser(value: unknown): boolean | undefined { + if (value === undefined || value === null) { + return true; + } + if (typeof value === "number") { + return Number.isInteger(value) && value >= 0 && value <= MAX_ID; + } + if (typeof value !== "string") { + return false; + } + const user = compiled(value); + if (user === null) { + return; + } + if (user === "") { + return true; + } + const match = NUMERIC_USER.exec(user); + return ( + match !== null && + Number(match[1]) <= MAX_ID && + (match[2] === undefined || Number(match[2]) <= MAX_ID) + ); +} + +function emptyArgv(value: unknown): boolean { + if (Array.isArray(value)) { + return value.length === 0; + } + return typeof value === "string" && value.trim() === ""; +} + +/** True when the declared entrypoint and command select no executable. */ +function emptyProcess(service: Readonly>): boolean { + const entrypoint = service.entrypoint; + if (Array.isArray(entrypoint) && entrypoint.length > 0) { + const first = entrypoint[0]; + return typeof first === "string" && (first === "" || first === "/"); + } + if (!emptyArgv(entrypoint)) { + return false; + } + // An explicit empty entrypoint suppresses the image's CMD as well. + const command = service.command; + if (Array.isArray(command) && command.length > 0) { + const first = command[0]; + return typeof first === "string" && (first === "" || first === "/"); + } + return command === undefined || command === null || emptyArgv(command); +} + +function unwrappableHealth(health: unknown): boolean { + // A native HTTP probe replaces the container health check entirely. + if ( + !isRecord(health) || + health.disable === true || + Object.hasOwn(health, "x-hack-http") + ) { + return false; + } + const test = health.test; + if (test === undefined || test === null) { + return false; + } + if (typeof test === "string") { + return test.trim() === ""; + } + if (!Array.isArray(test) || test.length === 0) { + return true; + } + const [kind, ...rest] = test; + if (kind === "NONE") { + return false; + } + if (kind === "CMD") { + return !( + typeof rest[0] === "string" && + rest[0] !== "" && + rest.every((part) => typeof part === "string") + ); + } + if (kind === "CMD-SHELL") { + return !( + rest.length === 1 && + typeof rest[0] === "string" && + rest[0] !== "" + ); + } + return true; +} + +function volumeTargets(volumes: unknown): string[] { + if (!Array.isArray(volumes)) { + return []; + } + const targets: string[] = []; + for (const volume of volumes) { + if (typeof volume === "string") { + const parts = volume.split(":"); + const target = parts.length >= 2 ? parts[1] : parts[0]; + if (target) { + targets.push(target); + } + } else if (isRecord(volume) && typeof volume.target === "string") { + targets.push(volume.target); + } + } + return targets; +} + +/** The same reserved targets the environment launcher and release barrier refuse. */ +function reservedTarget(target: string, dependency: boolean): boolean { + const normalized = + target.length > 1 && target.endsWith("/") ? target.slice(0, -1) : target; + return ( + normalized === "/" || + normalized === "/run" || + ENVIRONMENT_RESERVED.some((prefix) => normalized.startsWith(prefix)) || + (dependency && + DEPENDENCY_RESERVED.some( + (prefix) => normalized === prefix || normalized.startsWith(`${prefix}/`) + )) + ); +} + +/** + * Compose-declared settings native cannot wrap, for every wrapped service: + * `wrapped` services have their process wrapped, `fileOwners` receive private + * files only (their user must be numeric). `dependencies` are the wrapped + * services that also run behind the host-dependency release barrier. + * Interpolated values are compiled like native execution; unknown ones are + * left to the runtime check. + */ +export function predictNativeProcessRefusals(opts: { + readonly services: Readonly< + Record>> + >; + readonly wrapped: ReadonlySet; + readonly dependencies?: ReadonlySet; + readonly fileOwners?: ReadonlySet; +}): NativeProcessRefusal[] { + const refusals: NativeProcessRefusal[] = []; + for (const name of Object.keys(opts.services).sort()) { + const service = opts.services[name]; + const wrapped = opts.wrapped.has(name); + if ( + !( + service && + SERVICE.test(name) && + (wrapped || opts.fileOwners?.has(name)) + ) + ) { + continue; + } + const add = (field: NativeProcessField) => { + if (refusals.length < MAX_REFUSALS) { + refusals.push({ service: name, field }); + } + }; + if (wrapped && emptyProcess(service)) { + add("entrypoint"); + } + if (wrapped && unwrappableHealth(service.healthcheck)) { + add("healthcheck"); + } + if (numericUser(service.user) === false) { + add("user"); + } + if ( + wrapped && + typeof service.stop_signal === "string" && + !SIGNALS.has(service.stop_signal) + ) { + add("stop_signal"); + } + const dependency = opts.dependencies?.has(name) === true; + if ( + wrapped && + volumeTargets(service.volumes).some((target) => + reservedTarget(target, dependency) + ) + ) { + add("mounts"); + } + } + return refusals; +} + +/** + * Compose profile activation: a service without `profiles` (or with an empty + * list) always runs; otherwise it runs only when one of its profiles is + * selected. Inactive services are never scheduled, so nothing wraps them. + */ +export function activeNativeService( + service: Readonly>, + profiles: readonly string[] +): boolean { + const declared = service.profiles; + if (declared === undefined || declared === null) { + return true; + } + return ( + Array.isArray(declared) && + (declared.length === 0 || + declared.some( + (profile) => typeof profile === "string" && profiles.includes(profile) + )) + ); +} + +function activeServices( + services: Readonly>>>, + profiles: readonly string[] +): Record>> { + const result: Record>> = {}; + for (const [name, service] of Object.entries(services)) { + if (activeNativeService(service, profiles)) { + Object.defineProperty(result, name, { + value: service, + enumerable: true, + writable: true, + configurable: true, + }); + } + } + return result; +} + +/** Leading text of the refusal for Compose settings no process wrapper can run. */ +export const NATIVE_PROCESS_WRAPPER_REFUSAL = + "Native up cannot wrap these services' processes; nothing was started."; + +/** + * Refuse, before the VM starts, the Compose-declared settings the runtime's + * process wrappers refuse. Services with private environment values or host + * dependency bindings have their process wrapped, and services with private + * files only need a numeric user. Settings inherited from images are checked + * by the runtime against the pinned image before any state is reserved. + */ +export function requireWrappableNativeProcesses(opts: { + readonly services: Readonly< + Record>> + >; + /** Selected profiles: services outside them never run and are not checked. */ + readonly profiles: readonly string[]; + readonly managedEnvironment: Readonly< + Record>> + >; + readonly managedFiles?: Readonly>; + readonly dependencyServices: readonly string[]; +}): void { + const dependencies = new Set(opts.dependencyServices); + const wrapped = new Set(dependencies); + for (const [service, values] of Object.entries(opts.managedEnvironment)) { + if (Object.keys(values).length > 0) { + wrapped.add(service); + } + } + const fileOwners = new Set( + Object.entries(opts.managedFiles ?? {}) + .filter(([service, files]) => files.length > 0 && !wrapped.has(service)) + .map(([service]) => service) + ); + const refusals = predictNativeProcessRefusals({ + services: activeServices(opts.services, opts.profiles), + wrapped, + dependencies, + fileOwners, + }); + if (refusals.length > 0) { + throw new HackCliError({ + code: "E_NATIVE_PROJECT_UNSUPPORTED", + message: `${NATIVE_PROCESS_WRAPPER_REFUSAL} Refused: ${describeNativeProcessRefusals(refusals)}.`, + }); + } +} + +/** Doctor check id for the process-wrapper prediction. */ +export const NATIVE_PROCESS_CHECK = "native process wrappers"; + +/** Value-free JSON payload: service names and closed fields only. */ +export type NativeProcessReport = { + readonly refusals: readonly NativeProcessRefusal[]; + /** Services behind a profile: refused only when that profile is selected. */ + readonly profiled?: readonly NativeProcessRefusal[]; +}; + +function declaresHostGateway(extraHosts: unknown): boolean { + if (isRecord(extraHosts)) { + return Object.values(extraHosts).some( + (target) => target === "host-gateway" + ); + } + return ( + Array.isArray(extraHosts) && + extraHosts.some( + (entry) => typeof entry === "string" && entry.endsWith("host-gateway") + ) + ); +} + +/** Bare or null environment keys are where Hack delivers managed values. */ +function declaresManagedKeys(environment: unknown): boolean { + if (isRecord(environment)) { + return Object.values(environment).some((value) => value === null); + } + return ( + Array.isArray(environment) && + environment.some( + (entry) => typeof entry === "string" && !entry.includes("=") + ) + ); +} + +/** + * The services native up may wrap, from the Compose file alone: every service + * when project-wide host aliases exist (each then needs a host dependency + * binding), services that declare a host-gateway extra host, and services + * that declare bare environment keys Hack can deliver privately. + */ +export function nativeWrappedServiceCandidates(opts: { + readonly compose: unknown; + readonly projectWideAliases: boolean; +}): { + readonly services: Record>>; + readonly wrapped: ReadonlySet; + readonly dependencies: ReadonlySet; +} { + const services: Record>> = {}; + const wrapped = new Set(); + const dependencies = new Set(); + if (!(isRecord(opts.compose) && isRecord(opts.compose.services))) { + return { services, wrapped, dependencies }; + } + for (const [name, service] of Object.entries(opts.compose.services)) { + if (!(isRecord(service) && SERVICE.test(name))) { + continue; + } + Object.defineProperty(services, name, { + value: service, + enumerable: true, + writable: true, + configurable: true, + }); + if (opts.projectWideAliases || declaresHostGateway(service.extra_hosts)) { + dependencies.add(name); + wrapped.add(name); + } + if (declaresManagedKeys(service.environment)) { + wrapped.add(name); + } + } + return { services, wrapped, dependencies }; +} + +/** + * Doctor result for the process-wrapper prediction. Warning when a + * Compose-declared setting would be refused, otherwise ok. Read-only and + * value-free apart from service names. + */ +export function nativeProcessCheckResult( + refusals: readonly NativeProcessRefusal[], + /** Refusals of services that run only when one of their profiles is selected. */ + profiled: readonly NativeProcessRefusal[] = [] +): { + readonly name: string; + readonly status: "ok" | "warn"; + readonly message: string; + readonly nativeProcess: NativeProcessReport; +} { + const nativeProcess = { + refusals: refusals.map((entry) => ({ ...entry })), + ...(profiled.length > 0 + ? { profiled: profiled.map((entry) => ({ ...entry })) } + : {}), + }; + const note = + profiled.length > 0 + ? ` Only when their profile is selected: ${describeNativeProcessRefusals(profiled)}.` + : ""; + if (refusals.length === 0) { + return { + name: NATIVE_PROCESS_CHECK, + status: "ok", + message: `No Compose setting blocks native process wrapping for the services that always run; native up checks image entrypoints, commands and users before it reserves anything and names every refused service at once.${note}`, + nativeProcess, + }; + } + return { + name: NATIVE_PROCESS_CHECK, + status: "warn", + message: `Native up would refuse to wrap these services' processes: ${describeNativeProcessRefusals(refusals)}.${note}`, + nativeProcess, + }; +} + +/** + * Doctor's prediction, split by profile activation: services that always run + * can block `up` (warning); services behind a profile block it only when that + * profile is selected, so they are reported as information only. + */ +export function nativeProcessCheckFromCompose(opts: { + readonly compose: unknown; + readonly projectWideAliases: boolean; +}): ReturnType { + const candidates = nativeWrappedServiceCandidates(opts); + const refusals = predictNativeProcessRefusals(candidates); + const always = refusals.filter((refusal) => { + const service = candidates.services[refusal.service]; + return service !== undefined && activeNativeService(service, []); + }); + return nativeProcessCheckResult( + always, + refusals.filter((refusal) => !always.includes(refusal)) + ); +} diff --git a/tests/internal-extra-hosts-worktree.test.ts b/tests/internal-extra-hosts-worktree.test.ts index 4ee15c06c..02f075a5b 100644 --- a/tests/internal-extra-hosts-worktree.test.ts +++ b/tests/internal-extra-hosts-worktree.test.ts @@ -2,6 +2,7 @@ import { afterEach, beforeEach, expect, test } from "bun:test"; import { mkdir, mkdtemp, realpath, rm, symlink } from "node:fs/promises"; import { tmpdir } from "node:os"; import { join } from "node:path"; +import { resolveNativeInternalExtraHostSelection } from "../src/backends/native-project-internal-hosts.ts"; import { resolveInternalExtraHosts } from "../src/lib/internal-extra-hosts.ts"; import { resolvePrimaryLocalProjectDir } from "../src/lib/worktree-local-config.ts"; import { fixtureGitEnv } from "./helpers/fixture-git.ts"; @@ -255,3 +256,38 @@ test("explicit local static aliases override inherited aliases; local dynamic wi "search.test": "127.0.0.3", }); }); + +test("native alias sources distinguish project config, this checkout and the inherited main checkout", async () => { + const f = await fixture(); + await f.write(f.primary, { "inherited.example.test": "host-gateway" }); + await f.write(f.branch, { "local.example.test": "host-gateway" }); + const configFile = join(f.context.projectDir, "hack.config.json"); + await Bun.write( + configFile, + JSON.stringify({ + name: "sources-fixture", + internal: { extra_hosts: { "project.example.test": "host-gateway" } }, + }) + ); + const selection = await resolveNativeInternalExtraHostSelection({ + project: { + ...f.context, + projectDirName: ".hack", + composeFile: join(f.context.projectDir, "docker-compose.yml"), + envFile: join(f.context.projectDir, ".env"), + configFile, + }, + }); + expect(selection).toEqual({ + hosts: { + "inherited.example.test": "host-gateway", + "local.example.test": "host-gateway", + "project.example.test": "host-gateway", + }, + sources: { + "inherited.example.test": "inherited", + "local.example.test": "checkout", + "project.example.test": "project", + }, + }); +}); diff --git a/tests/models/tla/README.md b/tests/models/tla/README.md index dcfb6fbe4..efa53e1cc 100644 --- a/tests/models/tla/README.md +++ b/tests/models/tla/README.md @@ -793,9 +793,50 @@ Assumptions: admission through startup holds the provider mutation lease, so no replacement happens inside it. A replacement with copied ownership labels before selection is indistinguishable from the recorded volume by name and labels, as in the implementation; the generation fence protects only the selection-to-admission -window. Dependency caches, live source and source-device rebinding keep their own -proofs and are outside this model. It does not prove engine mount behavior, crash -durability of the receipt, image resolution or application data contents. +window. Dependency caches are checked by the next model; live source and +source-device rebinding keep their own proofs and are outside this model. It does +not prove engine mount behavior, crash durability of the receipt, image resolution +or application data contents. + +## Dependency caches across plan changes + +`native-cache-reconcile/CacheReconcile.tla` checks which volume backs a dependency +cache key when a stopped graph is reconciled after a Compose edit. A cache is a +volume named by an identity of its inputs; run-owned data uses the run's own names. +Edits change the cache's own inputs (initializer definition, declaration, image, +lockfiles, and every service mount of the cache volume), anything else in the +Compose file, whether a non-initializer service mounts the cache read-write, or +whether the key is declared as a cache. Such a writable consumer can change the +contents with inputs the identity does not hash, so its cache is scoped to this +checkout or branch instance; another instance with its own writable consumer may +write into the caches it selects (`ForeignWrite`). Two values for each explore +2,208 distinct states. + +| Model action | Implementation boundary | +| --- | --- | +| `EditInputs` / `EditOther` / `EditConsumer` / `EditDeclaration` | Compose or lockfile edit between ups; `dependency_cache::resolve_with_legacy` computes the current identity and `instance_scoped_volumes` the scope | +| `ForeignWrite` | A linked worktree or branch instance whose writable consumer changes the cache it mounts | +| `UpCache` | `plan_reconcile::prepare` refuses run data declared as a cache; `Verified::project` keeps a cache of the same identity or selects the new identity's cache, then `Session::create_resources` adopts it by labels or creates it | +| `UpData` | A key that is no longer a cache gets a fresh run-owned volume; the cache reference is released, never deleted | +| `Stop` | Ctrl-C or `down` keeps every recorded volume | + +The `negative` control binds the whole Compose document into the identity (the +earlier formula) and must fail `UnrelatedEditKeepsCache` in an `UpCache` state with +`needlessRebuild = TRUE`. The `stale` control reattaches the recorded cache whatever +its identity and must fail `CacheMatchesInputs` with `staleMounted = TRUE`. The +`data-as-cache` control mounts retained run data newly declared as a cache and must +fail `NoDataAsCache` with `dataMounted = TRUE`. The `poison` control keeps caches +with a writable consumer shared across instances and must fail `NoForeignWrites` +with `foreignMounted = TRUE`. + +Assumptions: identities are exact (no collisions), the initializer populates a newly +selected cache before dependents start, read-only consumers and initializers only +produce what the identity describes, and admission holds the provider lease. Within +one instance the same author controls every input, so writes there are trusted. The +compatibility path that keeps a receipt recorded under the earlier identity while +its inputs still match (never for a writable consumer) is covered by +`plan_reconcile.rs` tests, not this model. It does not prove initializer +idempotence, cache content, capacity limits or cache-wide collection. ## Graph admission diff --git a/tests/models/tla/native-cache-reconcile/CacheReconcile.tla b/tests/models/tla/native-cache-reconcile/CacheReconcile.tla new file mode 100644 index 000000000..8dc981632 --- /dev/null +++ b/tests/models/tla/native-cache-reconcile/CacheReconcile.tla @@ -0,0 +1,129 @@ +------------------------- MODULE CacheReconcile ------------------------- +(* Dependency-cache selection when a stopped native graph is reconciled after a + Compose edit. A cache is a volume named by an identity of its inputs; run-owned + data lives under the run's own names. Compute is always recreated, so only + which volume backs the cache key is modeled, for one trusted instance. `inputs` + are the cache's own inputs (initializer definition, declaration, image, + lockfiles and every service mount of the cache volume); `other` is every other + part of the Compose file. `writable` says a non-initializer service mounts the + cache read-write: it can change the contents with inputs the identity does not + hash, so the cache is scoped to this instance. A linked worktree or branch + instance with its own writable consumer may write into the caches it selects. + Broken constants are negative controls. *) +EXTENDS Naturals, TLC +CONSTANTS Values, + WholeDocument, \* identity also binds `other` (the earlier formula) + KeepStale, \* reattach the recorded cache whatever its identity + DataAsCache, \* reselect retained run data newly declared as a cache + ShareWritable \* keep a cache with a writable consumer shared +Entry(k, id, i, w) == [kind |-> k, id |-> id, rin |-> i, rw |-> w] +None == Entry("none", <<0, 0, "none">>, 0, FALSE) +Scope(w) == IF w /\ ~ShareWritable THEN "here" ELSE "shared" +Id(i, o, w) == <> +VARIABLES phase, inputs, other, writable, declared, recorded, caches, tainted, + mounted, staleMounted, dataMounted, needlessRebuild, foreignMounted +vars == <> +flags == <> + +\* A first admission populated the cache for the initial inputs; it was shared. +Init == /\ phase = "stopped" + /\ inputs = 1 /\ other = 1 /\ writable = FALSE /\ declared = "cache" + /\ recorded = Entry("cache", Id(1, 1, FALSE), 1, FALSE) + /\ caches = {Id(1, 1, FALSE)} + /\ tainted = {} + /\ mounted = None + /\ staleMounted = FALSE /\ dataMounted = FALSE + /\ needlessRebuild = FALSE /\ foreignMounted = FALSE + +\* Edits between ups: the cache's own inputs, anything else, a writable consumer +\* added or removed, or whether the key is declared as a cache at all. +EditInputs(v) == /\ phase = "stopped" /\ v # inputs /\ inputs' = v + /\ UNCHANGED <> +EditOther(v) == /\ phase = "stopped" /\ v # other /\ other' = v + /\ UNCHANGED <> +EditConsumer == /\ phase = "stopped" /\ writable' = ~writable + /\ UNCHANGED <> +EditDeclaration == /\ phase = "stopped" + /\ declared' = IF declared = "cache" THEN "data" ELSE "cache" + /\ UNCHANGED <> + +\* Another instance whose writable consumer runs with any inputs writes into the +\* cache it selects. Its own instance-scoped identity is never "here". +ForeignWrite(v, o) == + LET id == <> IN + /\ tainted' = tainted \cup {id} + /\ caches' = caches \cup {id} + /\ UNCHANGED <> + +\* plan_reconcile::Verified::project then Session::create_resources: keep a +\* recorded cache of the same identity; otherwise select that identity's cache +\* (adopt or create; the initializer populates it) and release the old +\* reference. Retained run data declared as a cache is refused. +UpCache == + LET want == Id(inputs, other, writable) + entry == Entry("cache", want, inputs, writable) IN + /\ phase = "stopped" /\ declared = "cache" + /\ IF recorded.kind = "data" + THEN IF DataAsCache + THEN /\ phase' = "running" /\ mounted' = recorded + /\ dataMounted' = TRUE + /\ UNCHANGED <> + ELSE UNCHANGED <> + ELSE IF recorded.kind = "cache" /\ (recorded.id = want \/ KeepStale) + THEN /\ phase' = "running" /\ mounted' = recorded + /\ staleMounted' = (staleMounted \/ recorded.rin # inputs) + /\ foreignMounted' = (foreignMounted \/ recorded.id \in tainted) + /\ UNCHANGED <> + ELSE /\ phase' = "running" /\ mounted' = entry + /\ recorded' = entry + /\ caches' = caches \cup {want} + /\ needlessRebuild' = (needlessRebuild \/ + (recorded.kind = "cache" /\ recorded.rin = inputs + /\ recorded.rw = writable /\ want \notin caches)) + /\ foreignMounted' = (foreignMounted \/ want \in tainted) + /\ UNCHANGED <> + /\ UNCHANGED <> + +\* The key is ordinary run data: reattach recorded data, or create fresh run data +\* when it was a cache (whose shared volume is only released, never deleted). +UpData == + /\ phase = "stopped" /\ declared = "data" + /\ phase' = "running" + /\ recorded' = IF recorded.kind = "data" THEN recorded + ELSE Entry("data", <<0, 0, "none">>, 0, FALSE) + /\ mounted' = recorded' + /\ UNCHANGED <> + +Stop == /\ phase = "running" /\ phase' = "stopped" /\ mounted' = None + /\ UNCHANGED <> + +Next == \/ \E v \in Values : EditInputs(v) \/ EditOther(v) + \/ \E v, o \in Values : ForeignWrite(v, o) + \/ EditConsumer \/ EditDeclaration \/ UpCache \/ UpData \/ Stop +Spec == Init /\ [][Next]_vars + +TypeOK == /\ phase \in {"stopped", "running"} + /\ inputs \in Values /\ other \in Values /\ writable \in BOOLEAN + /\ declared \in {"cache", "data"} + /\ recorded.kind \in {"none", "cache", "data"} + /\ mounted.kind \in {"none", "cache", "data"} + +\* A running graph's cache was populated for exactly its current inputs. +CacheMatchesInputs == /\ ~staleMounted + /\ (phase = "running" /\ mounted.kind = "cache") => + mounted.rin = inputs +\* Run-owned data is never mounted as a shared cache. +NoDataAsCache == ~dataMounted +\* An edit that leaves the cache's own inputs and consumers unchanged keeps the +\* cache: it never selects a not-yet-populated cache for the same inputs. +UnrelatedEditKeepsCache == ~needlessRebuild +\* A cache that another instance's writable consumer wrote is never mounted here. +NoForeignWrites == ~foreignMounted +============================================================================= diff --git a/tests/models/tla/native-cache-reconcile/data-as-cache.cfg b/tests/models/tla/native-cache-reconcile/data-as-cache.cfg new file mode 100644 index 000000000..4dd13de17 --- /dev/null +++ b/tests/models/tla/native-cache-reconcile/data-as-cache.cfg @@ -0,0 +1,4 @@ +SPECIFICATION Spec +CONSTANTS Values = {1, 2} + WholeDocument = FALSE KeepStale = FALSE DataAsCache = TRUE ShareWritable = FALSE +INVARIANTS TypeOK CacheMatchesInputs NoDataAsCache UnrelatedEditKeepsCache NoForeignWrites diff --git a/tests/models/tla/native-cache-reconcile/negative.cfg b/tests/models/tla/native-cache-reconcile/negative.cfg new file mode 100644 index 000000000..bdd5118ab --- /dev/null +++ b/tests/models/tla/native-cache-reconcile/negative.cfg @@ -0,0 +1,4 @@ +SPECIFICATION Spec +CONSTANTS Values = {1, 2} + WholeDocument = TRUE KeepStale = FALSE DataAsCache = FALSE ShareWritable = FALSE +INVARIANTS TypeOK CacheMatchesInputs NoDataAsCache UnrelatedEditKeepsCache NoForeignWrites diff --git a/tests/models/tla/native-cache-reconcile/poison.cfg b/tests/models/tla/native-cache-reconcile/poison.cfg new file mode 100644 index 000000000..f2aea6d89 --- /dev/null +++ b/tests/models/tla/native-cache-reconcile/poison.cfg @@ -0,0 +1,4 @@ +SPECIFICATION Spec +CONSTANTS Values = {1, 2} + WholeDocument = FALSE KeepStale = FALSE DataAsCache = FALSE ShareWritable = TRUE +INVARIANTS TypeOK CacheMatchesInputs NoDataAsCache UnrelatedEditKeepsCache NoForeignWrites diff --git a/tests/models/tla/native-cache-reconcile/positive.cfg b/tests/models/tla/native-cache-reconcile/positive.cfg new file mode 100644 index 000000000..1235af9d0 --- /dev/null +++ b/tests/models/tla/native-cache-reconcile/positive.cfg @@ -0,0 +1,4 @@ +SPECIFICATION Spec +CONSTANTS Values = {1, 2} + WholeDocument = FALSE KeepStale = FALSE DataAsCache = FALSE ShareWritable = FALSE +INVARIANTS TypeOK CacheMatchesInputs NoDataAsCache UnrelatedEditKeepsCache NoForeignWrites diff --git a/tests/models/tla/native-cache-reconcile/stale.cfg b/tests/models/tla/native-cache-reconcile/stale.cfg new file mode 100644 index 000000000..086641289 --- /dev/null +++ b/tests/models/tla/native-cache-reconcile/stale.cfg @@ -0,0 +1,4 @@ +SPECIFICATION Spec +CONSTANTS Values = {1, 2} + WholeDocument = FALSE KeepStale = TRUE DataAsCache = FALSE ShareWritable = FALSE +INVARIANTS TypeOK CacheMatchesInputs NoDataAsCache UnrelatedEditKeepsCache NoForeignWrites diff --git a/tests/native-compose-interpolation.test.ts b/tests/native-compose-interpolation.test.ts new file mode 100644 index 000000000..5bdc0c889 --- /dev/null +++ b/tests/native-compose-interpolation.test.ts @@ -0,0 +1,49 @@ +/* biome-ignore-all lint/suspicious/noTemplateCurlyInString: literal Compose interpolation fixtures. */ +import { expect, test } from "bun:test"; +import { resolveNativeComposeText } from "../src/backends/native-compose-interpolation.ts"; + +function resolved(text: string, managed: readonly string[] = []) { + return resolveNativeComposeText({ text, managed: new Set(managed) }); +} + +test("unset references select defaults and drop alternatives, as native execution does", () => { + expect(resolved("api.example.test")).toBe("api.example.test"); + expect(resolved("${UNSET_PREFIX:+x-}api.example.test")).toBe( + "api.example.test" + ); + expect(resolved("${API_HOST:-api.example.test}")).toBe("api.example.test"); + expect(resolved("${A:-${B:-${C:-${D:-deep}}}}.example.test")).toBe( + "deep.example.test" + ); + // An unselected word is validated but its references are not required. + expect(resolved("${A:+${B}}api.example.test")).toBe("api.example.test"); + expect(resolved("cost$$5")).toBe("cost$5"); + expect(resolved("${A:-a$$b}")).toBe("a$b"); +}); + +test("text native execution would refuse, or cannot know yet, is unknown", () => { + for (const text of [ + "${REQUIRED}", + "$REQUIRED.example.test", + "${A:-${REQUIRED}}", + "${A:?message}", + "${A-word}", + "${A:}", + "${1BAD:-x}", + "${A:-{literal}}", + "${A:-$B}", + "${unclosed", + "trailing$", + "${A:-${B:-${C:-${D:-${E:-x}}}}}", + // Bound to the reviewed project name only after review. + "${COMPOSE_PROJECT_NAME:-api}.example.test", + "${COMPOSE_PROJECT_NAME}", + ]) { + expect(resolved(text)).toBeNull(); + } + // A Hack-managed name is delivered privately and never interpolated. + expect(resolved("${SECRET:-api.example.test}", ["SECRET"])).toBeNull(); + expect(resolved("${A:+${SECRET}}", ["SECRET"])).toBeNull(); + // The project name in an unselected word does not matter. + expect(resolved("${A:+${COMPOSE_PROJECT_NAME}}x")).toBe("x"); +}); diff --git a/tests/native-plan-change.test.ts b/tests/native-plan-change.test.ts new file mode 100644 index 000000000..72deecda6 --- /dev/null +++ b/tests/native-plan-change.test.ts @@ -0,0 +1,39 @@ +import { expect, test } from "bun:test"; +import { + nativePlanChangeGuidance, + readNativePlanChange, +} from "../src/backends/native-plan-change.ts"; + +test("only the exact versioned plan-change diagnostic is read", () => { + const valid = { version: 1, service: "deps", field: "hostnames" }; + expect(readNativePlanChange(valid)).toEqual({ + service: "deps", + field: "hostnames", + }); + for (const value of [ + undefined, + { ...valid, version: 2 }, + { ...valid, field: "environment" }, + { ...valid, field: "toString" }, + { ...valid, service: "two words" }, + { ...valid, extra: true }, + ]) { + expect(readNativePlanChange(value)).toBeUndefined(); + } +}); + +test("retained-run edit refusals get closed, accurate guidance", () => { + expect( + nativePlanChangeGuidance("graph_plan_change_unsupported", undefined) + ).toContain("only routed hostname changes apply to it"); + expect( + nativePlanChangeGuidance("graph_hostname_change", { + service: "web", + field: "hostnames", + }) + ).toContain("the hostname change (service `web`, routed hostnames)"); + expect(nativePlanChangeGuidance("graph_plan_reconcile", undefined)).toBe( + "this Compose edit cannot be applied to the retained run. Revert the edit to keep the retained run as admitted, or run `hack down --remove-data` and then `hack up` to apply it with fresh data." + ); + expect(nativePlanChangeGuidance("graph_budget", undefined)).toBeUndefined(); +}); diff --git a/tests/native-process-wrappers.test.ts b/tests/native-process-wrappers.test.ts new file mode 100644 index 000000000..81c48a8ed --- /dev/null +++ b/tests/native-process-wrappers.test.ts @@ -0,0 +1,279 @@ +/* biome-ignore-all lint/suspicious/noTemplateCurlyInString: literal Compose interpolation fixtures. */ +import { expect, test } from "bun:test"; +import { HackCliError } from "../src/lib/cli-result.ts"; +import { + describeNativeProcessRefusals, + NATIVE_PROCESS_CHECK, + NATIVE_PROCESS_WRAPPER_REFUSAL, + nativeProcessCheckFromCompose, + nativeProcessCheckResult, + nativeWrappedServiceCandidates, + predictNativeProcessRefusals, + readNativeProcessRefusals, + requireWrappableNativeProcesses, +} from "../src/lib/native-process-wrappers.ts"; + +function predict( + services: Record>, + dependencies: string[] = [] +) { + return predictNativeProcessRefusals({ + services, + wrapped: new Set(Object.keys(services)), + dependencies: new Set(dependencies), + }); +} + +test("stock images with relative entrypoints and health checks are wrappable", () => { + expect( + predict({ + db: { + image: "mysql:8.0", + healthcheck: { test: ["CMD", "mysqladmin", "ping", "-h", "localhost"] }, + }, + shell: { + image: "postgres:16", + healthcheck: { test: "pg_isready -U postgres" }, + }, + shellform: { + image: "redis:7", + healthcheck: { test: ["CMD-SHELL", "redis-cli ping"] }, + }, + relative: { + image: "alpine:3.21", + entrypoint: ["docker-entrypoint.sh"], + command: ["serve"], + }, + commandOnly: { image: "python:3", entrypoint: [], command: ["python3"] }, + numeric: { image: "node:22", user: "1000:1000" }, + root: { image: "node:22", user: 0 }, + probe: { + image: "nginx", + healthcheck: { test: ["CMD"], "x-hack-http": { path: "/" } }, + }, + disabled: { image: "nginx", healthcheck: { disable: true, test: [] } }, + interpolated: { image: "nginx", user: "${APP_UID:-1000}" }, + }) + ).toEqual([]); +}); + +test("Compose settings no wrapper can run are named by service and field", () => { + expect( + predict( + { + empty: { image: "alpine", entrypoint: [] }, + blank: { image: "alpine", entrypoint: "", command: [] }, + slash: { image: "alpine", entrypoint: ["/"] }, + health: { image: "alpine", healthcheck: { test: ["CMD"] } }, + shell: { image: "alpine", healthcheck: { test: "" } }, + named: { image: "postgres", user: "postgres" }, + group: { image: "postgres", user: "999:postgres" }, + signal: { image: "nginx", stop_signal: "SIGUSR1" }, + run: { image: "nginx", volumes: ["data:/run"] }, + launcher: { + image: "nginx", + volumes: [{ target: "/run/hack-environment.json" }], + }, + relay: { image: "nginx", volumes: ["./x:/run/hack-startup/x:ro"] }, + }, + ["relay"] + ) + ).toEqual([ + { service: "blank", field: "entrypoint" }, + { service: "empty", field: "entrypoint" }, + { service: "group", field: "user" }, + { service: "health", field: "healthcheck" }, + { service: "launcher", field: "mounts" }, + { service: "named", field: "user" }, + { service: "relay", field: "mounts" }, + { service: "run", field: "mounts" }, + { service: "shell", field: "healthcheck" }, + { service: "signal", field: "stop_signal" }, + { service: "slash", field: "entrypoint" }, + ]); + // Release-barrier paths are reserved only for dependency services. + expect( + predict({ app: { image: "nginx", volumes: ["./x:/run/hack-startup/x"] } }) + ).toEqual([]); +}); + +test("only wrapped services and private-file owners are checked", () => { + const services = { + plain: { image: "postgres", user: "postgres", entrypoint: [] }, + files: { image: "postgres", user: "postgres", entrypoint: [] }, + }; + expect( + predictNativeProcessRefusals({ + services, + wrapped: new Set(), + fileOwners: new Set(["files"]), + }) + ).toEqual([{ service: "files", field: "user" }]); +}); + +test("the up preflight refuses before runtime work, naming every service and field", () => { + let thrown: unknown; + try { + requireWrappableNativeProcesses({ + services: { + db: { image: "postgres", user: "postgres" }, + web: { image: "nginx", healthcheck: { test: ["CMD"] } }, + idle: { image: "nginx", user: "nginx" }, + }, + profiles: [], + managedEnvironment: { db: { PASSWORD: "synthetic" }, idle: {} }, + dependencyServices: ["web"], + }); + } catch (error) { + thrown = error; + } + expect(thrown).toBeInstanceOf(HackCliError); + expect(thrown).toHaveProperty("code", "E_NATIVE_PROJECT_UNSUPPORTED"); + expect((thrown as Error).message).toBe( + `${NATIVE_PROCESS_WRAPPER_REFUSAL} Refused: service db user (use a numeric UID or UID:GID); service web healthcheck test (use an exec-form CMD test that names an executable).` + ); + expect((thrown as Error).message).not.toContain("synthetic"); + expect((thrown as Error).message).not.toContain("postgres"); +}); + +test("the runtime refusal list is accepted only in its closed, bounded form", () => { + expect( + readNativeProcessRefusals({ + version: 1, + refusals: [{ service: "db", field: "stop_signal" }], + }) + ).toEqual([{ service: "db", field: "stop_signal" }]); + for (const value of [ + null, + { version: 1 }, + { version: 1, refusals: [{ service: "db", field: "env" }] }, + { version: 1, refusals: [{ service: "", field: "user" }] }, + { + version: 1, + refusals: Array.from({ length: 33 }, (_, i) => ({ + service: `s${i}`, + field: "user", + })), + }, + ]) { + expect(readNativeProcessRefusals(value)).toBeUndefined(); + } + expect( + describeNativeProcessRefusals([{ service: "db", field: "mounts" }]) + ).toBe( + "service db volume target (a volume target overlaps a path the wrapper reserves)" + ); +}); + +test("doctor predicts for every service when project-wide aliases exist", () => { + const compose = { + services: { + api: { image: "node", user: "node" }, + gateway: { + image: "nginx", + user: "nginx", + extra_hosts: ["api.example.test:host-gateway"], + }, + secret: { + image: "postgres", + user: "postgres", + environment: ["PASSWORD"], + }, + literal: { + image: "postgres", + user: "postgres", + environment: { MODE: "dev" }, + }, + }, + }; + const local = nativeWrappedServiceCandidates({ + compose, + projectWideAliases: false, + }); + expect([...local.wrapped].sort()).toEqual(["gateway", "secret"]); + expect([...local.dependencies]).toEqual(["gateway"]); + const wide = nativeWrappedServiceCandidates({ + compose, + projectWideAliases: true, + }); + const result = nativeProcessCheckResult(predictNativeProcessRefusals(wide)); + expect(result.name).toBe(NATIVE_PROCESS_CHECK); + expect(result.status).toBe("warn"); + expect(result.nativeProcess.refusals.map((entry) => entry.service)).toEqual([ + "api", + "gateway", + "literal", + "secret", + ]); + expect(result.message).toContain( + "service api user (use a numeric UID or UID:GID)" + ); + expect(nativeProcessCheckResult([]).status).toBe("ok"); +}); + +test("only profile-active services can block the up preflight", () => { + const services = { + web: { image: "nginx" }, + optional: { image: "postgres", user: "postgres", profiles: ["extra"] }, + }; + const opts = { + services, + managedEnvironment: { optional: { PASSWORD: "synthetic" } }, + dependencyServices: [], + }; + requireWrappableNativeProcesses({ ...opts, profiles: [] }); + requireWrappableNativeProcesses({ ...opts, profiles: ["other"] }); + expect(() => + requireWrappableNativeProcesses({ ...opts, profiles: ["extra"] }) + ).toThrow("Refused: service optional user (use a numeric UID or UID:GID)."); + // An empty profiles list always runs, like Compose. + expect(() => + requireWrappableNativeProcesses({ + ...opts, + services: { optional: { ...services.optional, profiles: [] } }, + profiles: [], + }) + ).toThrow("service optional user"); +}); + +test("doctor reports profile-gated services as information, never as a warning", () => { + const compose = { + services: { + web: { image: "nginx", environment: ["TOKEN"] }, + optional: { + image: "postgres", + user: "postgres", + environment: ["PASSWORD"], + profiles: ["extra"], + }, + }, + }; + const gated = nativeProcessCheckFromCompose({ + compose, + projectWideAliases: false, + }); + expect(gated.status).toBe("ok"); + expect(gated.nativeProcess).toEqual({ + refusals: [], + profiled: [{ service: "optional", field: "user" }], + }); + expect(gated.message).toContain( + "Only when their profile is selected: service optional user" + ); + const always = nativeProcessCheckFromCompose({ + compose: { + services: { + ...compose.services, + web: { ...compose.services.web, user: "nginx" }, + }, + }, + projectWideAliases: false, + }); + expect(always.status).toBe("warn"); + expect(always.nativeProcess.refusals).toEqual([ + { service: "web", field: "user" }, + ]); + expect(always.nativeProcess.profiled).toEqual([ + { service: "optional", field: "user" }, + ]); +}); diff --git a/tests/native-project-internal-hosts.test.ts b/tests/native-project-internal-hosts.test.ts index d9e67ca41..68b3ffd0b 100644 --- a/tests/native-project-internal-hosts.test.ts +++ b/tests/native-project-internal-hosts.test.ts @@ -122,16 +122,59 @@ test("project-wide aliases are declared on every service like the Compose overri }); test("project-wide aliases without any dependency selection refuse instead of being ignored", () => { - expectCodedThrow( + const error = expectCodedThrow( () => applyNativeInternalHostAliases({ - input: input({ web: { image: "nginx" } }), + input: input({ web: { image: "nginx" }, worker: { image: "nginx" } }), hosts: { "api.example.test": "host-gateway" }, + sources: { "api.example.test": "inherited" }, hasDependencySelection: false, }), "E_NATIVE_PROJECT_UNSUPPORTED", NATIVE_UNBOUND_HOST_ALIAS_REFUSAL ); + expect(error.message).toContain( + "Unbound: api.example.test for service web, service worker (from the main checkout's .hack/.internal/extra-hosts.json, which this linked worktree inherits)." + ); + expect(error.message).toContain( + 'Add a binding with "aliases": ["api.example.test"] to the HACK_NATIVE_DEPENDENCIES selection' + ); + expect(error.message).toContain( + "run `hack internal extra-hosts unset api.example.test` in this worktree" + ); +}); + +test("an unbound refusal names each service, alias and where the alias comes from", () => { + const hostAliases = { + web: ["api.example.test", "auth.example.test", "own.example.test"], + worker: ["api.example.test", "local.example.test"], + }; + const error = expectCodedThrow( + () => + requireNativeHostAliasBindings({ + dependencies: [binding("web", ["auth.example.test"])], + hostAliases, + sources: { + "api.example.test": "project", + "auth.example.test": "project", + "local.example.test": "checkout", + }, + }), + "E_NATIVE_PROJECT_UNSUPPORTED", + NATIVE_UNBOUND_HOST_ALIAS_REFUSAL + ); + expect(error.message).toContain( + "Unbound: api.example.test for service web, service worker (from internal.extra_hosts in the project configuration); local.example.test for service worker (from this checkout's .hack/.internal/extra-hosts.json); own.example.test for service web (from the service's Compose extra_hosts)." + ); + expect(error.message).toContain( + '"aliases": ["api.example.test","local.example.test","own.example.test"]' + ); + expect(error.message).toContain( + "remove api.example.test from internal.extra_hosts in the project configuration" + ); + // Listener values are never echoed. + expect(error.message).not.toContain("8443"); + expect(error.message).not.toContain("123"); }); test("project-wide aliases native cannot honor refuse value-free", () => { @@ -170,6 +213,21 @@ test("a service that maps the alias to another target conflicts", () => { ["api.example.test:10.0.0.5"], ["api.example.test=10.0.0.5"], { "api.example.test": "10.0.0.5" }, + ["${UNSET:-api.example.test}:10.0.0.5"], + ]) { + const error = expectCodedThrow( + () => + applyNativeInternalHostAliases({ + input: input({ web: { image: "nginx", extra_hosts } }), + hosts: { "api.example.test": "host-gateway" }, + hasDependencySelection: true, + }), + "E_CONFIG_INVALID", + "Native host alias selection conflicts with declared configuration: service web maps project-wide host alias api.example.test to another target" + ); + expect(error.message).not.toContain("10.0.0.5"); + } + for (const extra_hosts of [ "api.example.test:host-gateway", Array.from({ length: 32 }, (_, i) => `h${i}.example.test:host-gateway`), ]) { @@ -224,10 +282,7 @@ test("service aliases are canonicalized like native review before matching bindi services: { web: { extra_hosts: ["API.Example.:host-gateway"] }, map: { extra_hosts: { "Search.EXAMPLE.test.": "host-gateway" } }, - both: { - extra_hosts: ["api.example:host-gateway", "API.EXAMPLE.=host-gateway"], - }, - // Resolved by the runtime later; admission checks these exactly. + // A required reference refuses at execution; admission checks it there. later: { extra_hosts: ["${ALIAS}:host-gateway", "1.2.3.4:host-gateway"] }, }, profiles: [], @@ -235,14 +290,12 @@ test("service aliases are canonicalized like native review before matching bindi expect(hostAliases).toEqual({ web: ["api.example"], map: ["search.example.test"], - both: ["api.example"], later: [], }); requireNativeHostAliasBindings({ dependencies: [ binding("web", ["api.example"]), binding("map", ["search.example.test"]), - binding("both", ["api.example"]), ], hostAliases, }); @@ -258,6 +311,119 @@ test("service aliases are canonicalized like native review before matching bindi ); }); +test("an interpolated service alias is compiled like native execution before matching", () => { + // Native execution supplies no ambient environment: `${UNSET:+x-}` selects + // nothing and `${UNSET:-word}` selects its word. + const hostAliases = nativeServiceHostAliases({ + services: { + plus: { + extra_hosts: ["${UNSET_PREFIX:+x-}api.example.test:host-gateway"], + }, + dash: { + extra_hosts: { "${API_HOST:-API.example.test.}": "host-gateway" }, + }, + target: { + extra_hosts: ["api.example.test:${GATEWAY:-host-gateway}"], + }, + // Managed values are delivered privately and never interpolated. + managed: { extra_hosts: ["${SECRET:-api.example.test}:host-gateway"] }, + // The project name is bound only after review. + named: { + extra_hosts: ["${COMPOSE_PROJECT_NAME:-api}.example.test:host-gateway"], + }, + }, + profiles: [], + managedEnvironment: { managed: { SECRET: "synthetic" } }, + }); + expect(hostAliases).toEqual({ + plus: ["api.example.test"], + dash: ["api.example.test"], + target: ["api.example.test"], + managed: [], + named: [], + }); +}); + +test("one alias declared twice after interpolation refuses before runtime work, naming it", () => { + for (const extra_hosts of [ + ["api.example:host-gateway", "API.EXAMPLE.=host-gateway"], + [ + "api.example.test:host-gateway", + "${UNSET_PREFIX:+x-}api.example.test:host-gateway", + ], + { + "api.example.test": "host-gateway", + "${A:-api.example.test}": "host-gateway", + }, + ]) { + const alias = + Array.isArray(extra_hosts) && extra_hosts[0]?.startsWith("api.example:") + ? "api.example" + : "api.example.test"; + const error = expectCodedThrow( + () => + nativeServiceHostAliases({ + services: { web: { extra_hosts } }, + profiles: [], + }), + "E_CONFIG_INVALID", + `Native host alias selection conflicts with declared configuration: service web declares host alias ${alias} more than once` + ); + expect(error.message).not.toContain("host-gateway"); + } +}); + +test("HACK-1938: an interpolated alias equal to a project-wide alias is not added twice", () => { + const hosts = { "api.example.test": "host-gateway" }; + for (const declared of [ + "${UNSET_PREFIX:+x-}api.example.test:host-gateway", + "${API_HOST:-api.example.test}:host-gateway", + // Literal control: the same name without interpolation. + "api.example.test:host-gateway", + ]) { + const result = applyNativeInternalHostAliases({ + input: input({ app: { image: "alpine", extra_hosts: [declared] } }), + hosts, + hasDependencySelection: true, + }); + const app = services(result).app; + expect(app.extra_hosts).toEqual([declared]); + // What native execution then compiles is exactly one alias. + expect( + nativeServiceHostAliases({ services: { app }, profiles: [] }) + ).toEqual({ app: ["api.example.test"] }); + } + // The map form is compared the same way. + const map = applyNativeInternalHostAliases({ + input: input({ + app: { + image: "alpine", + extra_hosts: { "${UNSET_PREFIX:+x-}api.example.test": "host-gateway" }, + }, + }), + hosts, + hasDependencySelection: true, + }); + expect(services(map).app.extra_hosts).toEqual({ + "${UNSET_PREFIX:+x-}api.example.test": "host-gateway", + }); + // A name that interpolates to something else still receives the alias. + const other = applyNativeInternalHostAliases({ + input: input({ + app: { + image: "alpine", + extra_hosts: ["${UNSET_PREFIX:-x-}api.example.test:host-gateway"], + }, + }), + hosts, + hasDependencySelection: true, + }); + expect(services(other).app.extra_hosts).toEqual([ + "${UNSET_PREFIX:-x-}api.example.test:host-gateway", + "api.example.test:host-gateway", + ]); +}); + test("a project-wide alias another spelling already declares is not duplicated", () => { const result = applyNativeInternalHostAliases({ input: input({ @@ -287,7 +453,7 @@ test("a project-wide alias another spelling already declares is not duplicated", hasDependencySelection: true, }), "E_CONFIG_INVALID", - "Native host alias selection conflicts with declared configuration; values omitted." + "Native host alias selection conflicts with declared configuration: service web maps project-wide host alias api.example.test to another target" ); }); diff --git a/tests/native-project-process.test.ts b/tests/native-project-process.test.ts index 7f6c030cc..1b837edaa 100644 --- a/tests/native-project-process.test.ts +++ b/tests/native-project-process.test.ts @@ -662,3 +662,70 @@ test("a second stop request during beforeStop forces the owner to stop", async ( ); expect(canceled).toBe(true); }, 15_000); + +test("HACK-1939: a startup failure reports every named process refusal to the exit diagnostic", async () => { + const reply = JSON.stringify({ + code: "environment_user", + message: "synthetic-private-detail", + process_refusals: { + version: 1, + refusals: [ + { service: "db", field: "user" }, + { service: "web", field: "healthcheck" }, + ], + }, + }); + const opts = await fixture( + `console.error(${JSON.stringify(reply)}); process.exit(2);` + ); + const diagnostics: unknown[] = []; + const failure = await serveNativeProjectGraph({ + ...opts, + onExitDiagnostic: (diagnostic) => { + diagnostics.push(diagnostic); + }, + onReady: async () => { + throw new Error("must not become ready"); + }, + }).catch((error: unknown) => String(error)); + expect(failure).toContain("environment_user"); + expect(failure).not.toContain("synthetic-private-detail"); + expect(diagnostics).toEqual([ + { + exitCode: 2, + nativeCode: "environment_user", + processRefusals: [ + { service: "db", field: "user" }, + { service: "web", field: "healthcheck" }, + ], + }, + ]); +}); + +test("a malformed process refusal list is dropped whole", async () => { + for (const process_refusals of [ + { version: 2, refusals: [{ service: "db", field: "user" }] }, + { version: 1, refusals: [{ service: "db", field: "argv" }] }, + { version: 1, refusals: [{ service: "../db", field: "user" }] }, + { version: 1, refusals: [] }, + { version: 1, refusals: [{ service: "db", field: "user", value: "x" }] }, + ]) { + const reply = JSON.stringify({ + code: "environment_user", + process_refusals, + }); + const opts = await fixture( + `console.error(${JSON.stringify(reply)}); process.exit(2);` + ); + const diagnostics: Record[] = []; + await serveNativeProjectGraph({ + ...opts, + onExitDiagnostic: (diagnostic) => { + diagnostics.push(diagnostic); + }, + onReady: async () => {}, + }).catch(() => undefined); + expect(diagnostics[0]?.processRefusals).toBeUndefined(); + expect(diagnostics[0]?.nativeCode).toBe("environment_user"); + } +}); diff --git a/tests/native-project-restore.test.ts b/tests/native-project-restore.test.ts index 4eb41a5dd..6eed50433 100644 --- a/tests/native-project-restore.test.ts +++ b/tests/native-project-restore.test.ts @@ -223,3 +223,55 @@ test("an edited Compose plan restores the same run after the native reconcile pr flags: ["--expect-generation", selected.generation], }); }); + +test("a reconciled dependency-cache graph publishes fresh source instead of the retained revision", async () => { + const review = { + ...options.review, + planId: "f".repeat(64), + report: { + plan: { + services: { + deps: { active: true, dependency_cache: { volume: "deps" } }, + app: { active: true }, + }, + }, + }, + }; + const compatibility = { + run: saved.run, + owner: saved.owner, + namespace: saved.namespace, + plan: saved.planId, + reviewed_plan: review.planId, + source_revision: null, + reconcile: true, + }; + const restore = (reply: Record) => + selectNativeProjectRestore({ + ...options, + review, + restore: saved, + invoke: async ({ args }) => + args[1] === "restore-selection" ? selected : reply, + }); + // No retained revision: the start path publishes the current snapshot. + expect(await restore(compatibility)).toEqual({ + run: saved.run, + restoring: true, + flags: ["--expect-generation", selected.generation], + }); + // A reconciliation never names a retained cache publication. + await expectClosedRefusal( + restore({ ...compatibility, source_revision: "1".repeat(64) }), + "compatibility changed" + ); + // Without reconciliation the retained revision stays mandatory. + await expectClosedRefusal( + restore({ ...compatibility, reconcile: undefined }), + "compatibility changed" + ); + await expectClosedRefusal( + restore({ ...compatibility, reconcile: "true" }), + "compatibility changed" + ); +}); diff --git a/tests/native-project-retained-images.test.ts b/tests/native-project-retained-images.test.ts index 0cec3157e..5afcd7883 100644 --- a/tests/native-project-retained-images.test.ts +++ b/tests/native-project-retained-images.test.ts @@ -525,3 +525,43 @@ test("restart of a stopped graph points at up instead of reporting image drift", "stopped with data retained. Run hack up" ); }); + +test("a reconciled graph's reserved shared cache may already exist; reserved run data may not", () => { + const baseline = observation(); + const withCache = (state: string, cache: boolean) => ({ + ...baseline, + receipt: { + ...baseline.receipt, + resources: { + ...baseline.receipt.resources, + "volume:deps": { + kind: "volume", + key: "deps", + phase: "reserved", + ...(cache + ? { + cache: { + scope: "1".repeat(64), + fingerprint: "2".repeat(64), + image, + }, + } + : {}), + }, + }, + }, + observations: { ...baseline.observations, "volume:deps": { state } }, + }); + expect(confirmedNativeRetainedGraph(withCache("present", true), saved)).toBe( + true + ); + expect(confirmedNativeRetainedGraph(withCache("absent", true), saved)).toBe( + true + ); + expect(confirmedNativeRetainedGraph(withCache("unknown", true), saved)).toBe( + false + ); + expect(confirmedNativeRetainedGraph(withCache("present", false), saved)).toBe( + false + ); +}); diff --git a/tests/native-project-start.test.ts b/tests/native-project-start.test.ts index 7c703e9c8..b99ece5de 100644 --- a/tests/native-project-start.test.ts +++ b/tests/native-project-start.test.ts @@ -40,7 +40,10 @@ import { import { NATIVE_CONFIGURATION_REFUSAL, NATIVE_SHARED_SOURCE_REFUSAL, + type NativeStartupProgress, nativeReadinessArgs, + nativeReadySummary, + nativeStoppedNotice, parseNativeHttpsSelection, prepareNativeProjectServices, reviewedNativeHttpsRoutes, @@ -2793,7 +2796,7 @@ test("final inspection failure reports uncertainty without echoing arbitrary fai expect(events).not.toContain("remove"); }); -test("a refusal before admission reports that nothing was retained, with the runtime reason", async () => { +test("a refusal before admission reports that nothing was created, with the runtime reason", async () => { const { opts, events } = await fixture(); const invoke = opts.dependencies.invoke!; opts.dependencies.serve = async () => { @@ -2816,7 +2819,7 @@ test("a refusal before admission reports that nothing was retained, with the run ); expect(failure).toBeInstanceOf(ClosedCliError); expect((failure as Error).message).toBe( - "Native graph startup failed before any runtime state was reserved; nothing was retained. Startup diagnostic: Native graph startup was interrupted or failed (graph_image). Service web: Pinned image architecture or implicit volumes are incompatible." + "Native graph startup failed before any runtime state was reserved; nothing was created and no cleanup is needed. Startup diagnostic: Native graph startup was interrupted or failed (graph_image). Service web: Pinned image architecture or implicit volumes are incompatible." ); expect(events).not.toContain("save"); expect(events).not.toContain("remove"); @@ -5016,13 +5019,18 @@ test("a project-wide alias without a binding for every active service refuses be ...opts, dependencyFile: path, internalExtraHosts, + internalExtraHostSources: { "api.example.test": "inherited" }, }).catch((caught: unknown) => caught); expect(error).toBeInstanceOf(HackCliError); expect(error).toHaveProperty("code", "E_NATIVE_PROJECT_UNSUPPORTED"); expect(String((error as Error).message)).toContain( "Native host alias has no reviewed host dependency binding" ); - expect(String((error as Error).message)).not.toContain("example"); + // HACK-1939: the refusal names the service, the alias and its source. + expect(String((error as Error).message)).toContain( + "Unbound: api.example.test for service worker (from the main checkout's .hack/.internal/extra-hosts.json, which this linked worktree inherits)." + ); + expect(String((error as Error).message)).not.toContain("8443"); expect(events).toContain("before"); expect(events).toContain("cleanup"); expect(events).not.toContain("runtime up"); @@ -5045,3 +5053,259 @@ test("project-wide aliases without a dependency selection refuse before hooks in // Without project-wide aliases the same project still starts unchanged. expect(await startNativeProject({ ...opts, internalExtraHosts: {} })).toBe(0); }); + +test("HACK-1939: a startup that failed before reserving any runtime state does not report unconfirmed cleanup", async () => { + const { opts, events } = await fixture(); + const invoke = opts.dependencies.invoke!; + opts.dependencies.serve = async () => { + throw new Error( + "Native graph startup was interrupted or failed (invalid_extra_hosts); inspect owned state before retrying." + ); + }; + opts.dependencies.invoke = async (request) => { + if (request.args[1] === "inspect") { + events.push("graph inspect"); + throw new NativeRuntimeRequestError({ + message: "Native graph run is not reserved.", + nativeCode: "graph_unreserved", + }); + } + return await invoke(request); + }; + const error = await startNativeProject(opts).catch( + (caught: unknown) => caught + ); + expect(error).toBeInstanceOf(ClosedCliError); + expect((error as Error).message).toBe( + "Native graph startup failed before any runtime state was reserved; nothing was created and no cleanup is needed. Startup diagnostic: Native graph startup was interrupted or failed (invalid_extra_hosts)." + ); + expect(events).toContain("graph inspect"); + expect(events).not.toContain("save"); + expect(events.at(-1)).toBe("cleanup"); +}); + +test("an unreserved run after a startup that published a mapping still reports unconfirmed cleanup", async () => { + const { opts } = await fixture(); + const invoke = opts.dependencies.invoke!; + const serve = opts.dependencies.serve!; + let served = false; + opts.dependencies.serve = async (request) => { + await serve({ + ...request, + onReady: async () => { + await request.onReady(); + served = true; + }, + }); + throw new Error( + "Native graph startup was interrupted or failed (runtime_not_running); inspect owned state before retrying." + ); + }; + opts.dependencies.invoke = async (request) => { + if (served && request.args[1] === "inspect") { + throw new NativeRuntimeRequestError({ + message: "Native graph run is not reserved.", + nativeCode: "graph_unreserved", + }); + } + return await invoke(request); + }; + await expect(startNativeProject(opts)).rejects.toThrow( + "cleanup is unconfirmed" + ); +}); + +test("HACK-1939: foreground up prints a ready summary and a confirmed-stop line", async () => { + const { opts } = await fixture(); + const events: NativeStartupProgress[] = []; + const controller = new AbortController(); + expect( + await startNativeProject({ + ...opts, + signal: controller.signal, + onReady: async () => { + controller.abort(); + }, + onProgress: (event) => { + events.push(event); + }, + }) + ).toBe(130); + const ready = events.find((event) => event.phase === "ready"); + expect(ready).toMatchObject({ phase: "ready", services: ["web"] }); + expect((ready as { run: string }).run).toMatch(/^[a-f0-9]{32}$/); + expect(events.at(-1)).toEqual({ + phase: "stopped", + summary: + "Native project stopped; owned cleanup confirmed. Named volumes and the stopped run mapping are kept for the next hack up.", + }); +}); + +test("the ready summary lists reviewed HTTPS routes and published loopback ports only", () => { + const plan = { + services: { + web: { + active: true, + routing: { hostnames: ["app.example.test", "www.example.test"] }, + ports: [ + { protocol: "tcp", proposed_host_ip: "127.0.0.1", published: 5173 }, + ], + }, + api: { active: true, routing: { hostnames: ["api.example.test"] } }, + odd: { active: true, routing: { hostnames: ["Not A Host"] } }, + db: { + active: true, + ports: [ + { protocol: "tcp", proposed_host_ip: "127.0.0.1", published: 5432 }, + { protocol: "tcp", proposed_host_ip: "0.0.0.0", published: 6000 }, + ], + }, + }, + }; + const routed = new Set(["web", "api", "odd"]); + expect(nativeReadySummary({ plan, routed, httpsPort: 18_443 })).toEqual([ + " api: https://api.example.test:18443", + " web: https://app.example.test:18443 https://www.example.test:18443", + " Published ports: 127.0.0.1:5173 127.0.0.1:5432", + ]); + expect(nativeReadySummary({ plan, routed, httpsPort: 443 })[0]).toBe( + " api: https://api.example.test" + ); + expect(nativeReadySummary({ plan, routed, httpsPort: null })).toEqual([ + " Routes are not served on this host: the native HTTPS frontend is not selected (HACK_NATIVE_CADDY_BINARY, HACK_NATIVE_CADDY_SHA256 and HACK_NATIVE_HTTPS_PORT).", + " Published ports: 127.0.0.1:5173 127.0.0.1:5432", + ]); +}); + +test("the confirmed-stop line reflects whether data was retained", () => { + expect(nativeStoppedNotice({ receipt: { phase: "removed" } })).toBe( + "Native project stopped; owned cleanup confirmed and no runtime data was retained." + ); + expect( + nativeStoppedNotice({ receipt: { phase: "stopped-data-retained" } }) + ).toContain("Named volumes and the stopped run mapping are kept"); +}); + +test("HACK-1939: a wrapped service whose Compose user cannot be wrapped refuses before the VM starts", async () => { + const { opts, events } = await fixture(); + const prepare = opts.dependencies.prepare!; + opts.dependencies.prepare = async (request) => { + const input = await prepare(request); + const compose = JSON.parse(input.normalizedComposeJson); + compose.services.web.user = "postgres"; + return { ...input, normalizedComposeJson: JSON.stringify(compose) }; + }; + const error = await startNativeProject(opts).catch( + (caught: unknown) => caught + ); + expect(error).toBeInstanceOf(HackCliError); + expect(error).toHaveProperty("code", "E_NATIVE_PROJECT_UNSUPPORTED"); + expect((error as Error).message).toBe( + "Native up cannot wrap these services' processes; nothing was started. Refused: service web user (use a numeric UID or UID:GID)." + ); + expect(events).not.toContain("runtime up"); + expect(events.at(-1)).toBe("cleanup"); +}); + +test("HACK-1939: runtime process refusals are named in the nothing-reserved report", async () => { + const { opts } = await fixture(); + const invoke = opts.dependencies.invoke!; + opts.dependencies.serve = async (request) => { + request.onExitDiagnostic?.({ + exitCode: 2, + nativeCode: "environment_entrypoint", + processRefusals: [ + { service: "web", field: "entrypoint" }, + { service: "worker", field: "user" }, + ], + }); + throw new Error( + "Native graph startup was interrupted or failed (environment_entrypoint); inspect owned state before retrying." + ); + }; + opts.dependencies.invoke = async (request) => { + if (request.args[1] === "inspect") { + throw new NativeRuntimeRequestError({ + message: "Native graph run is not reserved.", + nativeCode: "graph_unreserved", + }); + } + return await invoke(request); + }; + const error = await startNativeProject(opts).catch( + (caught: unknown) => caught + ); + expect((error as Error).message).toBe( + "Native graph startup failed before any runtime state was reserved; nothing was created and no cleanup is needed. Startup diagnostic: Native graph startup was interrupted or failed (environment_entrypoint). Native cannot wrap these services' processes: service web entrypoint or command (the process must name an executable); service worker user (use a numeric UID or UID:GID)." + ); +}); + +/** A fixture whose inactive-by-default `optional` service cannot be wrapped. */ +async function optionalProfileFixture() { + const fixtureResult = await fixture(); + const prepare = fixtureResult.opts.dependencies.prepare!; + fixtureResult.opts.dependencies.prepare = async (request) => { + const input = await prepare(request); + const compose = JSON.parse(input.normalizedComposeJson); + compose.services.optional = { + image: `sha256:${"e".repeat(64)}`, + user: "postgres", + profiles: ["extra"], + }; + return { + ...input, + normalizedComposeJson: JSON.stringify(compose), + managedEnvironment: { + ...input.managedEnvironment, + optional: { PASSWORD: "synthetic-secret" }, + }, + serviceNames: [...input.serviceNames, "optional"], + }; + }; + return fixtureResult; +} + +test("an inactive-profile service a wrapper cannot run does not block up; an active one does", async () => { + const inactive = await optionalProfileFixture(); + expect(await startNativeProject(inactive.opts)).toBe(0); + const active = await optionalProfileFixture(); + const error = await startNativeProject({ + ...active.opts, + profiles: ["extra"], + }).catch((caught: unknown) => caught); + expect((error as Error).message).toBe( + "Native up cannot wrap these services' processes; nothing was started. Refused: service optional user (use a numeric UID or UID:GID)." + ); + expect(active.events).not.toContain("runtime up"); +}); + +test("restart preflight ignores inactive-profile services a wrapper cannot run", async () => { + const { opts } = await optionalProfileFixture(); + const preflight = (profiles: string[]) => + preflightNativeRestart({ + runtime: opts.runtime, + scope: opts.scope, + composeFile: opts.composeFile, + run: { + run: "1".repeat(32), + owner: "c".repeat(32), + namespace: "b".repeat(64), + planId: "a".repeat(64), + effectiveEnvName: null, + profiles, + aws: null, + }, + dependencies: { + prepare: opts.dependencies.prepare, + review: opts.dependencies.review, + dependencies: async () => [], + invoke: restartPreflightInvoke, + }, + }); + const refusal = await preflight(["extra"]).catch((caught: unknown) => caught); + expect((refusal as Error).message).toContain( + "Refused: service optional user (use a numeric UID or UID:GID)." + ); + // The same preflight without the profile completes. + await preflight([]); +}); diff --git a/tests/native-runtime-client.test.ts b/tests/native-runtime-client.test.ts index 7489ca38d..77533b37e 100644 --- a/tests/native-runtime-client.test.ts +++ b/tests/native-runtime-client.test.ts @@ -149,6 +149,8 @@ async function fixture() { if (process.argv.includes("hang")) await Bun.sleep(60_000); if (process.argv.includes("fixed-message")) { console.error(JSON.stringify({code:"graph_image",message:"Pinned image architecture or implicit volumes are incompatible.",message_fixed:true,service:"web"})); process.exit(2); } if (process.argv.includes("unsafe-fixed")) { console.error(JSON.stringify({code:"graph_image",message:"synthetic-secret-diagnostic\\nsecond line",message_fixed:true,service:"../synthetic-secret-diagnostic"})); process.exit(2); } +if (process.argv.includes("plan-change")) { console.error(JSON.stringify({code:"graph_plan_reconcile",message:"synthetic-secret-diagnostic",plan_change:{version:1,service:"deps",field:"dependency_cache_volume"}})); process.exit(23); } +if (process.argv.includes("plan-change-unsafe")) { console.error(JSON.stringify({code:"graph_hostname_change",message:"synthetic-secret-diagnostic",plan_change:{version:1,service:"synthetic secret diagnostic",field:"hostnames"}})); process.exit(23); } if (process.argv.includes("structured")) { console.error(JSON.stringify({code:"source_conflict",message:"synthetic-secret-diagnostic"})); process.exit(23); } if (process.argv.includes("foreground-stopped")) { console.error(JSON.stringify({code:"native_graph_foreground",cause_code:"runtime_not_running",message:"synthetic-secret-diagnostic"})); process.exit(23); } if (process.argv.includes("foreground-other")) { console.error(JSON.stringify({code:"native_graph_foreground",cause_code:"engine_protocol",message:"synthetic-secret-diagnostic"})); process.exit(23); } @@ -769,6 +771,35 @@ test("native failure preserves only its validated structured code", async () => } }); +test("a refused Compose edit names its service and field and how to apply it", async () => { + const runtime = await fixture(); + const reconcile: unknown = await invokeNativeRuntime({ + runtime, + cwd: runtime.home, + args: ["plan-change"], + }).catch((caught: unknown) => caught); + expect(reconcile).toBeInstanceOf(NativeRuntimeRequestError); + expect(reconcile).toMatchObject({ nativeCode: "graph_plan_reconcile" }); + const message = String(reconcile); + expect(message).toContain( + "Native runtime request failed (graph_plan_reconcile); this Compose edit (service `deps`, `hack.dependencies.cache-volume`) cannot be applied to the retained run." + ); + expect(message).toContain("`hack down --remove-data` and then `hack up`"); + expect(message).not.toContain("synthetic-secret-diagnostic"); + // An unbounded service name is dropped; the closed guidance remains. + const hostname = String( + await invokeNativeRuntime({ + runtime, + cwd: runtime.home, + args: ["plan-change-unsafe"], + }).catch((caught: unknown) => caught) + ); + expect(hostname).toContain( + "Native runtime request failed (graph_hostname_change); the hostname change cannot be applied" + ); + expect(hostname).not.toContain("synthetic"); +}); + test("failures omit subprocess diagnostics and timeouts are bounded", async () => { const runtime = await fixture(); await expect( diff --git a/tests/native-startup-logs.test.ts b/tests/native-startup-logs.test.ts index 0a7ab3606..b3e25ceea 100644 --- a/tests/native-startup-logs.test.ts +++ b/tests/native-startup-logs.test.ts @@ -76,6 +76,19 @@ test("values shorter than four characters are redacted only as whole tokens", () ); }); +test("HACK-1939: short values never shred version strings, decimals or hyphenated words", () => { + expect( + redactNativeStartupLog( + "mysqld 8.0.46 ready; ratio 0.5; build 1-2; flag v-1; DEBUG=0; exit -1; mode 0.", + ["0", "1"] + ) + ).toBe( + "mysqld 8.0.46 ready; ratio 0.5; build 1-2; flag v-1; DEBUG=[REDACTED]; exit -[REDACTED]; mode [REDACTED]." + ); + // Long values are still redacted wherever they occur. + expect(redactNativeStartupLog("v8.0.46-x", ["8.0.46"])).toBe("v[REDACTED]-x"); +}); + test("remaining control characters cannot reach the terminal", () => { expect(redactNativeStartupLog("bell\u0007 tab\tback\b", [])).toBe( "bell� tab\tback�"