diff --git a/.github/workflows/tests-golang.yml b/.github/workflows/tests-golang.yml index 62bd9453a..014884fd7 100644 --- a/.github/workflows/tests-golang.yml +++ b/.github/workflows/tests-golang.yml @@ -61,6 +61,8 @@ on: - scripts/stack-conformance.sh - scripts/stack-conformance-selection.mjs - scripts/stack-eql-conformance.mjs + - scripts/stack-conformance-evidence.mjs + - scripts/stack-lifecycle-conformance.mjs - Cargo.toml - Cargo.lock - .cargo/** @@ -101,6 +103,8 @@ on: - scripts/stack-conformance.sh - scripts/stack-conformance-selection.mjs - scripts/stack-eql-conformance.mjs + - scripts/stack-conformance-evidence.mjs + - scripts/stack-lifecycle-conformance.mjs - Cargo.toml - Cargo.lock - .cargo/** @@ -272,6 +276,17 @@ jobs: echo "RLIMIT_MEMLOCK: $(ulimit -l) KiB" mise run go:test + - name: Required lifecycle and key-I/O conformance + env: + STACK_CONFORMANCE_REPORT_DIR: ${{ runner.temp }}/lifecycle-conformance + run: mise run test:conformance:lifecycle + + - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4 + with: + name: lifecycle-conformance + path: ${{ runner.temp }}/lifecycle-conformance/*.json + if-no-files-found: error + - name: Required EQL and JSON conformance env: STACK_CONFORMANCE_REPORT_DIR: ${{ runner.temp }}/eql-conformance-fast diff --git a/conformance/stack-encrypt/README.md b/conformance/stack-encrypt/README.md index 635a99185..e895fdecc 100644 --- a/conformance/stack-encrypt/README.md +++ b/conformance/stack-encrypt/README.md @@ -41,6 +41,7 @@ these rules. | `record-context-boundaries-v1.cases.json` | cases | `tests/context_conformance.rs` | `context_conformance_test.go` | | `eql-json-v1.cases.json` | cases | `tests/json_operators.rs`; EQL's `tests/encryption/tests/json_tree_data_plan.rs` | `json_conformance_test.go` | | `eql-json-v1.enrollment.json` | enrollment | `scripts/stack-eql-conformance.mjs` runs the tests it names | `scripts/stack-eql-conformance.mjs` runs the tests it names | +| `lifecycle-v1.enrollment.json` | enrollment | `scripts/stack-lifecycle-conformance.mjs` runs the tests it names | `scripts/stack-lifecycle-conformance.mjs` runs the tests it names | | `fixture-enrollment.json` | index | `tests/common/mod.rs` | `fixture_conformance_test.go` | Rust readers are under `packages/stack-encrypt/`; Go readers are under @@ -75,6 +76,14 @@ is `languages/golang/encrypt/conformance_test.go`. Go calls the record interface used by generated code. Testing public SDK declarations is a separate layer. Python and TypeScript runners are pending; their absence is not a passing result. +## Lifecycle, cleanup and key I/O + +The [lifecycle contract](lifecycle-contract.md) maps native/Go ownership, +cancellation, partial-output cleanup, diagnostics, validation-before-I/O and +batching evidence. Run `mise run test:conformance:lifecycle` on a supported Unix +host with the normal and deterministic guests built. Reports retain known +cleanup gaps and unobserved behavior alongside the passing requirements. + ## EQL and JSON operations The [EQL/JSON contract](eql-json-contract.md) enrolls resolver capabilities, diff --git a/conformance/stack-encrypt/coverage.md b/conformance/stack-encrypt/coverage.md index 37160f81e..39b356b78 100644 --- a/conformance/stack-encrypt/coverage.md +++ b/conformance/stack-encrypt/coverage.md @@ -122,3 +122,17 @@ reports make enrollment observable. This consolidates data and requirements; it does not replace adapter tests or claim whole-database authentication. Fresh JSON artifact exchange, released-version matrices and Python/TypeScript runners remain open. + +## Lifecycle checkpoint + +The [lifecycle contract](lifecycle-contract.md) enrolls native pending/KMS tests, +heap observations, guest ABI cleanup, Go cancellation/memory tests and public SDK +diagnostics. New probes cover unpolled and in-flight drops, provider failure, +cleanup after a late authenticated type failure, and chunk concurrency/drop. +Existing runtime/read/context vectors retain their provider-call counters. + +This adds required execution and explicit observation limits; it removes no +backend-specific tests. The known Go post-decryption shadow-stack gap and native +heap-observer exclusions stay visible in reports. Logical batching is distinct +from downstream chunking and unobserved live HTTP behavior. Python and TypeScript +lifecycle evidence is pending. diff --git a/conformance/stack-encrypt/lifecycle-contract.md b/conformance/stack-encrypt/lifecycle-contract.md new file mode 100644 index 000000000..2cf10ab12 --- /dev/null +++ b/conformance/stack-encrypt/lifecycle-contract.md @@ -0,0 +1,107 @@ +# Lifecycle, cleanup and key-I/O conformance + +`lifecycle-v1` enrolls observable native Rust and Go WASI behavior. The +[index](v1/lifecycle-v1.enrollment.json) names existing compiled tests and focused +new lifetime probes. These are backend-specific observations, not another +semantic value corpus and not a promise to erase every plaintext copy in a +process. Python and TypeScript are explicitly not enrolled. + +## Required evidence + +| Requirement | Native evidence | Go evidence and limits | +|---|---|---| +| `lifecycle-ownership-v1` | Actual record operations release fulfilment captures exactly once on success, provider failure and unpolled drop; an unpolled encrypt or decrypt makes no key request. Zipped fulfilments do not share key material. | Client keys are consumed/redacted; explicit close and unreachable-client cleanup release guest memory; calls on a closed client fail. GC cleanup is eventual, so callers should close explicitly. | +| `lifecycle-cancellation-v1` | Dropping an encrypt future paused in key generation, or a decrypt future paused in key retrieval, drops the provider future and operation captures; subsequent work on the cipher succeeds. Guest ABI abort spends the handle and drops the operation. | Cancellation before a call sends no request; cancellation between steps or during token fetch leaves unrelated operations usable. A guest step that overruns cancellation closes the client. Discarded late output is wiped. | +| `lifecycle-partial-output-v1` | A batch with a valid first row and a later authenticated type mismatch releases its decoded prefix without marked heap residue. Existing context cases reject late authentication failures without returning partial output. | Generated and raw batch APIs return no partial rows on failure; interrupted HTTP bodies fail; cancelled output is wiped. SDK conversion diagnostics contain no decrypted values. This does not assert erasure of partially populated caller-owned Go objects. | +| `lifecycle-buffer-cleanup-v1` | The existing allocator observer checks selected write/read/JSON parse paths and has a negative control. Guest ABI tests check buffer-registry cleanup, error-buffer release and protected text drop. | Request bodies survive until their owner closes them, then wipe; refused requests wipe; heap growth wipes abandoned backing arrays; guest runtime close releases memory. Post-encryption memory scans are supplementary observations. | +| `lifecycle-diagnostics-v1` | Guest errors discard untrusted foreign cause messages; failed/successful exports do not hand over stale diagnostics. | Structured errors remain available after failures; SDK errors and redacted display/logging expose types or fields rather than plaintext values. | +| `lifecycle-validation-before-io-v1` | Existing runtime/read/context cases count provider calls. Wrong shape, independently detectable context mismatch and oversized descriptors fail before key I/O; lazy assembly stops after local failure. | Host refusals happen before entering the guest, and shared runtime/read cases exercise the uninitialised guest's preflight boundary. Go does not yet expose native provider counters. | +| `lifecycle-logical-batching-v1` | Instrumented `DataKeySource` calls establish one logical generation batch, retrieval grouping by keyset, descriptor ordering and no work for empty batches. | Go uses the same engine, but this profile does not claim independently instrumented Go key-request counts. | +| `lifecycle-provider-chunks-v1` | The actual KMS chunking helper preserves order, handles a final short chunk, fails the whole call on a chunk error, bounds in-flight chunks and releases their futures on cancellation. | This is a native algorithm observation, not a measurement of live HTTP request counts. | + +The drop witnesses belong to fulfilment closures and provider futures in real +record operations. They establish ownership release. They do **not** establish +that arbitrary captured values implement zeroization. The separate heap and +buffer observations provide the narrower byte-cleanup evidence. + +A validation failure detectable only after authentication necessarily retrieves +keys. The profile distinguishes that from shape and context checks that can run +without keys. An error does not imply that no provider request ever occurred. + +## Logical calls and physical requests + +A `Pending` combines requests before awaiting them. For supported assemblies, +generation dispatches one logical `generate_keys` call; retrieval dispatches one +logical `retrieve_keys` call per keyset. Existing instrumented providers observe +those calls and their ordered payloads. + +`StackKms::send_chunked` subsequently splits a logical call according to +`max_keys_per_req` and bounds concurrent chunks by `max_concurrent_reqs`. +The chunk tests observe this helper with controlled callbacks: ten inputs with +chunk size three and concurrency two start only two three-item chunks while +both are pending; dropping the aggregate releases both. This is distinct from +“one HTTP request per batch.” Live service limits, retries and transport timing +are not inferred from the logical counters. + +## Limits carried in every report + +These entries in the index are limitations, not successful requirements: + +- **`go-post-decrypt-shadow-stack` — known gap.** + `TestPlaintextDoesNotRemainInGuestMemoryAfterDecrypt` in + `languages/golang/encrypt/roundtrip_test.go` is explicitly skipped because + decrypted plaintext can remain in the WASM shadow stack in underlying cipher + temporaries. It is not enrolled as passing evidence. The post-encryption scan + cannot establish the post-decryption guarantee. +- **`native-heap-observer-exclusions` — limited observation.** + `packages/stack-encrypt/tests/heap_hygiene.rs` watches marked frees on the + executing thread. Its existing observer excludes a named upstream aws-lc + sealing frame by backtrace and reports excluded frees. It scrubs stack residue + before observing heap frees; it does not prove stack erasure. Its JSON query + paths use borrowed/static path text because owned path copies have separate + cleanup limitations. The lifecycle runner preserves the observer's output. +- **`caller-owned-host-values` — outside the guarantee.** Caller-owned strings, + bytes, objects, returned plaintext, aliases and copies remain the caller's + responsibility. Ownership transfer or protected borrowing must be stated by + each public API. Cancellation cannot retract work already accepted by a remote + service, and dropping a provider future does not prove remote cancellation. +- **`live-provider-http-chunking` — not observed.** Hermetic tests count logical + provider calls and chunk callbacks. They do not attest to actual ZeroKMS + request counts, retry behavior or remote cleanup. +- **`python-typescript-lifecycle` — not enrolled.** Their backends, cancellation + models, host mappings and extension-owned buffers need their own evidence. + +The current required lane supports Linux and macOS. Its native heap observer +requires Unix; unsupported hosts fail explicitly rather than silently skipping +that evidence. This is not Windows lifecycle certification. Memory locking under +OS resource limits, whole-process heap scans, other threads, compiler/register +residue and arbitrary caller-supplied sinks are outside the enrolled guarantees. +Existing broader platform, Miri, fuzz, live and locking tests remain in place. + +## Running and extending the profile + +From the repository root: + +```sh +mise run wasm:guest:build +mise run wasm:guest:build:deterministic +mise run test:conformance:lifecycle +``` + +No CipherStash credentials or database are required. Some Go cancellation tests +start local HTTP stubs. The runner shares inventory/execution checks with EQL +conformance: every enrolled native test must be compiled, selected and not +ignored, and every Go test must pass with no skipped subtests. Fresh Go tests +bypass the test cache. Required fixture-update variables are rejected. + +The required `wasi-check` CI job runs the task and uploads its report, failing +if the report is absent. Reports identify the engine, platform/architecture, +lockfile, guest hashes, enrollment, executed evidence and known limitations. +Set `STACK_CONFORMANCE_REPORT_DIR` locally to retain the same JSON report. + +Add evidence to the existing requirement where possible. Keep language-specific +lifetime probes and implementation regressions: a shared semantic round-trip +cannot replace them. Do not claim a stronger cleanup guarantee by removing a +limitation or accepting a skipped test; supply executable evidence on the +backend being claimed. Secure-buffer adoption and key-copy removal remain +implementation work, distinct from this inventory. diff --git a/conformance/stack-encrypt/v1/lifecycle-v1.enrollment.json b/conformance/stack-encrypt/v1/lifecycle-v1.enrollment.json new file mode 100644 index 000000000..12b35f883 --- /dev/null +++ b/conformance/stack-encrypt/v1/lifecycle-v1.enrollment.json @@ -0,0 +1,254 @@ +{ + "version": 1, + "profile": "lifecycle-v1", + "suites": { + "pending": { + "language": "rust", + "lane": "fast", + "workspace": ".", + "package": "stack-encrypt", + "features": ["--all-features"], + "lib": true + }, + "lifecycle": { + "language": "rust", + "lane": "fast", + "workspace": ".", + "package": "stack-encrypt", + "features": ["--all-features"], + "binary": "lifecycle_conformance" + }, + "heap": { + "language": "rust", + "lane": "fast", + "workspace": ".", + "package": "stack-encrypt", + "features": ["--all-features"], + "binary": "heap_hygiene", + "success_output": "immediate" + }, + "runtime": { + "language": "rust", + "lane": "fast", + "workspace": ".", + "package": "stack-encrypt", + "features": ["--all-features"], + "binary": "runtime_conformance" + }, + "reads": { + "language": "rust", + "lane": "fast", + "workspace": ".", + "package": "stack-encrypt", + "features": ["--all-features"], + "binary": "read_conformance" + }, + "contexts": { + "language": "rust", + "lane": "fast", + "workspace": ".", + "package": "stack-encrypt", + "features": ["--all-features"], + "binary": "context_conformance" + }, + "guest-abi": { + "language": "rust", + "lane": "fast", + "workspace": ".", + "package": "stack-guest-abi", + "features": ["--all-features"], + "lib": true + }, + "kms": { + "language": "rust", + "lane": "fast", + "workspace": ".", + "package": "stack-kms", + "features": ["--all-features"], + "lib": true + }, + "go": { + "language": "go", + "lane": "fast", + "package": "./encrypt" + }, + "go-memory": { + "language": "go", + "lane": "fast", + "package": "./internal/guest" + }, + "go-sdk": { + "language": "go", + "lane": "fast", + "package": "./encrypt/gensupport" + } + }, + "requirements": [ + { + "id": "lifecycle-ownership-v1", + "evidence": { + "lifecycle": [ + "dropping_unpolled_work_releases_captures_without_key_io", + "dropping_unpolled_decrypt_releases_captures_without_key_io", + "success_and_provider_failure_release_captures_once" + ], + "pending": [ + "target::pending::tests::zipped_fulfilments_never_share_key_material" + ], + "go": [ + "TestClientKeyIsConsumedAndNeverPrinted", + "TestCloseStopsTheCleanup", + "TestUnreachableClientIsReleased", + "TestClosedClientIsState" + ] + } + }, + { + "id": "lifecycle-cancellation-v1", + "evidence": { + "lifecycle": [ + "cancelling_in_flight_work_drops_provider_and_captures", + "cancelling_in_flight_decrypt_drops_provider_and_captures" + ], + "guest-abi": [ + "step::tests::an_aborted_operation_is_dropped_and_its_handle_spent", + "step::tests::the_wrong_number_of_responses_is_refused_and_drops_the_operation" + ], + "go": [ + "TestACallerThatGivesUpBetweenStepsAbortsTheOperation", + "TestACallWhoseContextHasEndedNeverStarts", + "TestACallCancelledDuringATokenFetchFailsOnlyItself", + "TestAStepThatRunsOnAfterItsContextEndsClosesTheClient", + "TestACallThatSucceedsAfterItsContextEndsFailsWithIt", + "TestAnAbortedInitLeavesNoKeyInGuestMemory" + ] + } + }, + { + "id": "lifecycle-partial-output-v1", + "evidence": { + "heap": ["a_late_read_type_failure_leaves_no_plaintext_on_the_heap"], + "contexts": ["shared_context_boundaries"], + "go": [ + "TestContextBoundaries", + "TestInterruptedResponseBodyIsTransport", + "TestACallThatSucceedsAfterItsContextEndsFailsWithIt" + ], + "go-sdk": ["TestDecryptErrorsHoldNoPlaintext"] + } + }, + { + "id": "lifecycle-buffer-cleanup-v1", + "evidence": { + "heap": [ + "a_record_encrypted_from_a_value_frame_leaves_no_plaintext_on_the_heap", + "a_record_decrypted_with_a_nested_field_leaves_no_plaintext_on_the_heap", + "a_json_document_read_from_a_frame_leaves_no_plaintext_on_the_heap", + "the_spy_sees_an_unwiped_free" + ], + "guest-abi": [ + "buffers::tests::wipe_all_leaves_no_live_buffer", + "last_error::tests::clearing_wipes_and_frees_the_registered_buffer", + "step::tests::pending_requests_are_handed_over_once_and_replaced_wiped", + "value::tests::text_is_wiped_on_drop" + ], + "go": [ + "TestRequestBodyWipesOnClose", + "TestRequestBodyOutlivesTheRoundTrip", + "TestARefusedRequestIsWiped", + "TestPlaintextDoesNotRemainInGuestMemory" + ], + "go-memory": [ + "TestHeapMemoryWipesWhatItAbandons", + "TestGuestMemoryIsFreedOnRuntimeClose", + "TestClientKeyIsOpaqueAndWipes", + "TestMemoryOutlivesACallClosedDuringAHostImport" + ] + } + }, + { + "id": "lifecycle-diagnostics-v1", + "evidence": { + "guest-abi": [ + "last_error::tests::a_foreign_cause_nobody_vouches_for_gives_no_message", + "call::tests::a_failure_never_reports_an_earlier_calls_error", + "call::tests::a_failure_then_a_success_leaves_nothing_to_hand_over" + ], + "go-sdk": [ + "TestDecryptErrorsHoldNoPlaintext", + "TestRedactedHidesSealedFieldsAndSortsShown", + "TestRedactedLogHidesSealedFields" + ], + "go-memory": [ + "TestAFailureCarriesItsDiagnostic", + "TestATrapFetchingTheDetailIsATrap" + ] + } + }, + { + "id": "lifecycle-validation-before-io-v1", + "evidence": { + "runtime": ["shared_record_runtime_profile"], + "reads": ["shared_record_read_profile"], + "pending": [ + "target::pending::tests::an_over_long_descriptor_is_refused_before_any_call", + "target::pending::tests::a_lazy_column_stops_building_after_a_local_failure" + ], + "go": [ + "TestRuntimeConformance", + "TestReadConformance", + "TestMismatchedPlanIsRefusedBeforeTheGuest" + ] + } + }, + { + "id": "lifecycle-logical-batching-v1", + "evidence": { + "runtime": ["shared_record_runtime_profile"], + "pending": [ + "target::pending::tests::one_pending_asking_for_many_keys_is_one_call", + "target::pending::tests::all_merges_a_column_into_one_call_preserving_order", + "target::pending::tests::generate_and_retrieve_are_one_call_each", + "target::pending::tests::retrieves_group_by_keyset_and_return_in_request_order", + "target::pending::tests::all_of_nothing_resolves_empty_without_io", + "target::pending::tests::dispatch_forwards_each_requests_descriptor_in_order" + ] + } + }, + { + "id": "lifecycle-provider-chunks-v1", + "evidence": { + "kms": [ + "futures::tests::test_keeps_the_same_order", + "futures::tests::test_a_failing_chunk_fails_the_whole_call", + "futures::tests::test_works_when_chunks_dont_divide_nicely", + "futures::tests::concurrency_limit_bounds_chunks_and_drop_releases_them" + ] + } + } + ], + "limitations": [ + { + "id": "go-post-decrypt-shadow-stack", + "status": "known-gap", + "evidence": "languages/golang/encrypt/roundtrip_test.go::TestPlaintextDoesNotRemainInGuestMemoryAfterDecrypt" + }, + { + "id": "native-heap-observer-exclusions", + "status": "limited-observation", + "evidence": "packages/stack-encrypt/tests/heap_hygiene.rs" + }, + { + "id": "caller-owned-host-values", + "status": "outside-guarantee" + }, + { + "id": "live-provider-http-chunking", + "status": "not-observed" + }, + { + "id": "python-typescript-lifecycle", + "status": "not-enrolled" + } + ] +} diff --git a/docs/plans/2026-10-31-stack-encrypt-conformance.md b/docs/plans/2026-10-31-stack-encrypt-conformance.md index d5428bb22..16e526cc2 100644 --- a/docs/plans/2026-10-31-stack-encrypt-conformance.md +++ b/docs/plans/2026-10-31-stack-encrypt-conformance.md @@ -1,6 +1,6 @@ # Stack Encrypt cross-language conformance -Date: 2026-10-31. Status: scalar, declaration, initial runtime, read and context-composition slices implemented; existing value-kind, query-operator and context-field fixtures enrolled. The complete contract is not frozen. +Date: 2026-10-31. Status: scalar, declaration, initial runtime, read, context, EQL/JSON and lifecycle slices implemented; existing value-kind, query-operator and context-field fixtures enrolled. Wider boundaries, artifact exchange, Python/TypeScript enrollment and release gates remain. The complete contract is not frozen. This work is stacked on [PR #1404](https://github.com/cipherstash/stack/pull/1404), at `b684a217e2da629c762efa70fac958ec7e21646a`. It follows the accepted @@ -240,3 +240,18 @@ and batching requirements. Python/TypeScript enrollment and the broader fresh artifact/version exchange remain separate delivery slices. The EQL/JSON profile does not complete the refactoring track or introduce the future binding/search scope split. + +## Lifecycle and key-I/O checkpoint + +Implemented `lifecycle-v1` with eight named requirements and required native/Go +CI evidence. Reuses the pending, KMS chunking, heap, ABI, cancellation, memory, +diagnostic and shared runtime/read suites. New observations cover unpolled and +in-flight operation drops, provider failure, cleanup of decoded prefixes on +late read errors, and bounded chunk concurrency/cancellation. The runner shares +inventory and no-skip checks with EQL conformance. + +Reports distinguish passing requirements from known residue, observer exclusions, +caller-owned values, unobserved live-provider behavior and unenrolled languages. +This does not complete secure-buffer/key-copy refactoring. Next conformance +slices remain value/refusal/raw-frame gaps, broader fresh artifact exchange, +Python/TypeScript enrollment and explicit release gates/version combinations. diff --git a/mise.toml b/mise.toml index 07a2603aa..265975723 100644 --- a/mise.toml +++ b/mise.toml @@ -444,3 +444,7 @@ mise -C packages/eql run clean mise -C packages/eql run build node scripts/stack-eql-conformance.mjs postgres """ + +[tasks."test:conformance:lifecycle"] +description = "Required native/Go lifecycle, cleanup and key-I/O evidence (Unix; build the normal and deterministic guests first)" +run = "node scripts/stack-lifecycle-conformance.mjs" diff --git a/packages/stack-encrypt/tests/heap_hygiene.rs b/packages/stack-encrypt/tests/heap_hygiene.rs index 001b42c73..e229d412d 100644 --- a/packages/stack-encrypt/tests/heap_hygiene.rs +++ b/packages/stack-encrypt/tests/heap_hygiene.rs @@ -338,6 +338,37 @@ fn a_record_decrypted_with_a_nested_field_leaves_no_plaintext_on_the_heap() { }); } +/// A valid first row is followed by an authenticated value of the wrong type. +/// The failed batch returns no output and releases its decoded prefix cleanly. +#[test] +fn a_late_read_type_failure_leaves_no_plaintext_on_the_heap() { + let rt = runtime(); + let cipher = rt.block_on(common::stack_cipher()); + let keyset = cipher.default_keyset(); + let writer = record::plan(object(vec![spec("value", &["c"], None)]).risky_ref()).unwrap(); + let reader = + record::plan(object(vec![spec("value", &["c"], Some("string"))]).risky_ref()).unwrap(); + let source = frame(vec![ + object(vec![("value", frame(marked("first-row")))]), + object(vec![("value", frame(true))]), + ]); + let sealed = rt.block_on(async { record::encrypt(&keyset, source, &writer).unwrap().await }); + let sealed = sealed.unwrap(); + watched("late authenticated read type failure", || { + let result = rt.block_on(async { + record::decrypt(Scope::Client(&cipher), sealed, &reader, None) + .unwrap() + .await + }); + assert!(matches!( + result, + Err(stack_encrypt::Error::Plan( + stack_encrypt::plan::PlanError::FieldType { .. } + )) + )); + }); +} + #[test] fn a_query_term_leaves_no_plaintext_on_the_heap() { let rt = runtime(); diff --git a/packages/stack-encrypt/tests/lifecycle_conformance.rs b/packages/stack-encrypt/tests/lifecycle_conformance.rs new file mode 100644 index 000000000..85ec8a79a --- /dev/null +++ b/packages/stack-encrypt/tests/lifecycle_conformance.rs @@ -0,0 +1,252 @@ +//! Observe the lifetime of pending work independently of the provider's result. +//! Drop witnesses prove ownership release, not erasure of arbitrary host values. +#![cfg(feature = "dynamic")] + +mod common; + +use std::borrow::Cow; +use std::collections::BTreeMap; +use std::future::IntoFuture; +use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering}; +use std::sync::{Arc, Mutex}; +use std::task::{Context, Waker}; + +use common::frames::frame; +use stack_encrypt::dynamic::{record, Scope, ValueFrame}; +use stack_encrypt::{Pending, StackCipher, StackCipherText}; +use stack_kms::{ + DataKey, DataKeySource, DataKeyWithTag, FakeDataKeySource, GenerateKeyError, + GenerateKeyPayload, IdentifiedBy, IndexKey, IndexKeySource, RetrieveKeyPayload, + UnverifiedContext, +}; +use uuid::Uuid; + +#[derive(Default)] +struct Observations { + block: AtomicBool, + fail: AtomicBool, + batches: Mutex>, + provider_drops: Arc, + retrievals: Mutex>, + retrieve_drops: Arc, +} + +struct DropWitness(Arc); + +impl Drop for DropWitness { + fn drop(&mut self) { + self.0.fetch_add(1, Ordering::SeqCst); + } +} + +struct Source { + inner: FakeDataKeySource, + observed: Arc, +} + +impl DataKeySource for Source { + async fn generate_keys( + &self, + payloads: Vec>, + keyset: Option, + context: Option>, + ) -> Result, stack_kms::Error> { + self.observed.batches.lock().unwrap().push(payloads.len()); + let _owned_request = DropWitness(self.observed.provider_drops.clone()); + if self.observed.block.load(Ordering::SeqCst) { + std::future::pending::<()>().await; + } + if self.observed.fail.load(Ordering::SeqCst) { + return Err(GenerateKeyError::Unauthorized.into()); + } + self.inner.generate_keys(payloads, keyset, context).await + } + + async fn retrieve_keys( + &self, + payloads: Vec>, + keyset: Option, + context: Option<&UnverifiedContext>, + ) -> Result, stack_kms::Error> { + self.observed + .retrievals + .lock() + .unwrap() + .push(payloads.len()); + let _owned_request = DropWitness(self.observed.retrieve_drops.clone()); + if self.observed.block.load(Ordering::SeqCst) { + std::future::pending::<()>().await; + } + self.inner.retrieve_keys(payloads, keyset, context).await + } +} + +impl IndexKeySource for Source { + async fn load_index_key( + &self, + keyset: Option, + ) -> Result<(Uuid, IndexKey), stack_kms::Error> { + self.inner.load_index_key(keyset).await + } +} + +async fn cipher() -> (StackCipher, Arc) { + let observed = Arc::new(Observations::default()); + let cipher = StackCipher::builder() + .kms(Source { + inner: FakeDataKeySource::new(), + observed: observed.clone(), + }) + .init() + .await + .unwrap(); + (cipher, observed) +} + +fn plan() -> record::Plan { + record::plan( + &vitaminc_aead_frame::serde::to_vec(&serde_json::json!({ + "value": {"context": ["lifecycle", "value"], "outputs": ["c"], "type": "string"} + })) + .unwrap(), + ) + .unwrap() +} + +fn rows(count: usize) -> ValueFrame { + frame(vec![BTreeMap::from([("value", "lifecycle-input")]); count]) +} + +// The witness is owned by the real operation's fulfilment, not by the test's +// future. Dropping it therefore observes release of that fulfilment's captures. +fn observed<'a, T: 'a>( + pending: Pending<'a, T, Source>, + drops: Arc, +) -> Pending<'a, T, Source> { + let witness = DropWitness(drops); + pending.map(move |output| { + drop(witness); + output + }) +} + +#[tokio::test] +async fn dropping_unpolled_work_releases_captures_without_key_io() { + let (cipher, calls) = cipher().await; + let keyset = cipher.default_keyset(); + let drops = Arc::new(AtomicUsize::new(0)); + let pending = observed( + record::encrypt(&keyset, rows(2), &plan()).unwrap(), + drops.clone(), + ); + assert_eq!(drops.load(Ordering::SeqCst), 0); + assert!(calls.batches.lock().unwrap().is_empty()); + drop(pending); + assert_eq!(drops.load(Ordering::SeqCst), 1); + assert!(calls.batches.lock().unwrap().is_empty()); +} + +#[tokio::test] +async fn cancelling_in_flight_work_drops_provider_and_captures() { + let (cipher, calls) = cipher().await; + calls.block.store(true, Ordering::SeqCst); + let keyset = cipher.default_keyset(); + let drops = Arc::new(AtomicUsize::new(0)); + let mut future = observed( + record::encrypt(&keyset, rows(3), &plan()).unwrap(), + drops.clone(), + ) + .into_future(); + assert!(future + .as_mut() + .poll(&mut Context::from_waker(Waker::noop())) + .is_pending()); + assert_eq!(*calls.batches.lock().unwrap(), vec![3]); + assert_eq!(drops.load(Ordering::SeqCst), 0); + assert_eq!(calls.provider_drops.load(Ordering::SeqCst), 0); + drop(future); + assert_eq!(drops.load(Ordering::SeqCst), 1); + assert_eq!(calls.provider_drops.load(Ordering::SeqCst), 1); + // Cancellation does not poison the cipher or execute the abandoned work. + calls.block.store(false, Ordering::SeqCst); + record::encrypt(&keyset, rows(1), &plan()) + .unwrap() + .await + .unwrap(); + assert_eq!(*calls.batches.lock().unwrap(), vec![3, 1]); +} + +#[tokio::test] +async fn success_and_provider_failure_release_captures_once() { + for fail in [false, true] { + let (cipher, calls) = cipher().await; + calls.fail.store(fail, Ordering::SeqCst); + let keyset = cipher.default_keyset(); + let drops = Arc::new(AtomicUsize::new(0)); + let result = observed( + record::encrypt(&keyset, rows(4), &plan()).unwrap(), + drops.clone(), + ) + .await; + assert_eq!(result.is_err(), fail); + assert_eq!(drops.load(Ordering::SeqCst), 1); + assert_eq!(calls.provider_drops.load(Ordering::SeqCst), 1); + assert_eq!(*calls.batches.lock().unwrap(), vec![4]); + } +} + +async fn sealed(cipher: &StackCipher, count: usize) -> StackCipherText { + record::encrypt(&cipher.default_keyset(), rows(count), &plan()) + .unwrap() + .await + .unwrap() +} + +#[tokio::test] +async fn dropping_unpolled_decrypt_releases_captures_without_key_io() { + let (cipher, calls) = cipher().await; + let stored = sealed(&cipher, 2).await; + let drops = Arc::new(AtomicUsize::new(0)); + let pending = observed( + record::decrypt(Scope::Client(&cipher), stored, &plan(), None).unwrap(), + drops.clone(), + ); + assert_eq!(drops.load(Ordering::SeqCst), 0); + assert!(calls.retrievals.lock().unwrap().is_empty()); + drop(pending); + assert_eq!(drops.load(Ordering::SeqCst), 1); + assert!(calls.retrievals.lock().unwrap().is_empty()); + assert_eq!(calls.retrieve_drops.load(Ordering::SeqCst), 0); +} + +#[tokio::test] +async fn cancelling_in_flight_decrypt_drops_provider_and_captures() { + let (cipher, calls) = cipher().await; + let (stored, retry) = (sealed(&cipher, 3).await, sealed(&cipher, 3).await); + calls.block.store(true, Ordering::SeqCst); + let drops = Arc::new(AtomicUsize::new(0)); + let mut future = observed( + record::decrypt(Scope::Client(&cipher), stored, &plan(), None).unwrap(), + drops.clone(), + ) + .into_future(); + assert!(future + .as_mut() + .poll(&mut Context::from_waker(Waker::noop())) + .is_pending()); + assert_eq!(*calls.retrievals.lock().unwrap(), vec![3]); + assert_eq!(drops.load(Ordering::SeqCst), 0); + assert_eq!(calls.retrieve_drops.load(Ordering::SeqCst), 0); + drop(future); + assert_eq!(drops.load(Ordering::SeqCst), 1); + assert_eq!(calls.retrieve_drops.load(Ordering::SeqCst), 1); + // Cancellation does not poison the cipher or execute the abandoned work. + calls.block.store(false, Ordering::SeqCst); + let opened = record::decrypt(Scope::Client(&cipher), retry, &plan(), None) + .unwrap() + .await + .unwrap(); + assert_eq!(opened.risky_ref(), rows(3).risky_ref()); + assert_eq!(*calls.retrievals.lock().unwrap(), vec![3, 3]); + assert_eq!(calls.retrieve_drops.load(Ordering::SeqCst), 2); +} diff --git a/packages/stack-kms/src/futures.rs b/packages/stack-kms/src/futures.rs index 7ecc111ac..bb24224f8 100644 --- a/packages/stack-kms/src/futures.rs +++ b/packages/stack-kms/src/futures.rs @@ -90,4 +90,43 @@ mod tests { assert_eq!(input, output); } + #[test] + fn concurrency_limit_bounds_chunks_and_drop_releases_them() { + use std::sync::atomic::{AtomicUsize, Ordering}; + use std::sync::{Arc, Mutex}; + use std::task::{Context, Waker}; + + struct InFlight(Arc); + impl Drop for InFlight { + fn drop(&mut self) { + self.0.fetch_sub(1, Ordering::SeqCst); + } + } + let active = Arc::new(AtomicUsize::new(0)); + let chunks = Arc::new(Mutex::new(Vec::new())); + let input: Vec<_> = (0..10).collect(); + let mut work = Box::pin(map_async_chunked( + &input, + |chunk| { + let active = active.clone(); + let chunks = chunks.clone(); + async move { + active.fetch_add(1, Ordering::SeqCst); + let _in_flight = InFlight(active); + chunks.lock().unwrap().push(chunk.to_vec()); + std::future::pending::, ()>>().await + } + }, + 3, + 2, + )); + assert!(work + .as_mut() + .poll(&mut Context::from_waker(Waker::noop())) + .is_pending()); + assert_eq!(active.load(Ordering::SeqCst), 2); + assert_eq!(*chunks.lock().unwrap(), vec![vec![0, 1, 2], vec![3, 4, 5]]); + drop(work); + assert_eq!(active.load(Ordering::SeqCst), 0); + } } diff --git a/scripts/__tests__/stack-conformance-ci.test.mjs b/scripts/__tests__/stack-conformance-ci.test.mjs index bae94cf5a..d9bd1f10c 100644 --- a/scripts/__tests__/stack-conformance-ci.test.mjs +++ b/scripts/__tests__/stack-conformance-ci.test.mjs @@ -251,14 +251,14 @@ describe('conformance files are named after their profile and role', () => { const header = JSON.parse(readRepo(`${dir}/${file}`)) expect(header.version).toBe(1) expect(header.profile).toBe(profile) - if (role !== 'cases') { - expect(files, `${file} has no cases file`).toContain( - `${profile}.cases.json`, - ) - } if (role === 'enrollment') { + // An index of existing tests; a profile may have no cases of its own. expect(header.requirements?.length).toBeGreaterThan(0) } else if (role !== 'cases') { + // Outputs and runner inputs are keyed by the case IDs beside them. + expect(files, `${file} has no cases file`).toContain( + `${profile}.cases.json`, + ) expect(Object.keys(header.cases ?? {}).length).toBeGreaterThan(0) } if (producer) expect(header.producer).toMatch(new RegExp(`^${producer}`)) diff --git a/scripts/__tests__/stack-lifecycle-conformance.test.mjs b/scripts/__tests__/stack-lifecycle-conformance.test.mjs new file mode 100644 index 000000000..cbb53b0b0 --- /dev/null +++ b/scripts/__tests__/stack-lifecycle-conformance.test.mjs @@ -0,0 +1,189 @@ +import { readFileSync } from 'node:fs' +import { join } from 'node:path' +import { describe, expect, it } from 'vitest' +import { + checkRustInventory, + nextestArgs, + selections, +} from '../stack-conformance-evidence.mjs' +import { buildReport } from '../stack-lifecycle-conformance.mjs' +import { REPO_ROOT } from './lib/repo-root.mjs' +import { readWorkflow } from './lib/workflows.mjs' + +const read = (path) => readFileSync(join(REPO_ROOT, path), 'utf8') +const index = JSON.parse( + read('conformance/stack-encrypt/v1/lifecycle-v1.enrollment.json'), +) + +describe('required lifecycle evidence', () => { + it('enrolls each guarantee and keeps limitations outside passing evidence', () => { + expect(index.requirements.map((r) => r.id)).toEqual([ + 'lifecycle-ownership-v1', + 'lifecycle-cancellation-v1', + 'lifecycle-partial-output-v1', + 'lifecycle-buffer-cleanup-v1', + 'lifecycle-diagnostics-v1', + 'lifecycle-validation-before-io-v1', + 'lifecycle-logical-batching-v1', + 'lifecycle-provider-chunks-v1', + ]) + const contract = read('conformance/stack-encrypt/lifecycle-contract.md') + for (const r of index.requirements) expect(contract).toContain(r.id) + const selected = selections(index, 'fast', 'all') + expect([...selected.keys()].sort()).toEqual( + Object.keys(index.suites).sort(), + ) + expect( + selected + .get('go') + .tests.has('TestPlaintextDoesNotRemainInGuestMemoryAfterDecrypt'), + ).toBe(false) + expect(index.limitations).toContainEqual( + expect.objectContaining({ + id: 'go-post-decrypt-shadow-stack', + status: 'known-gap', + }), + ) + expect(index.limitations).toContainEqual( + expect.objectContaining({ + id: 'python-typescript-lifecycle', + status: 'not-enrolled', + }), + ) + for (const limitation of index.limitations) + expect(contract).toContain(limitation.id) + }) + + it('checks compiled library names as strictly as integration binaries', () => { + const suite = { + package: 'engine', + lib: true, + lane: 'fast', + tests: new Set(['nested::required']), + } + const inventory = { + 'rust-suites': { + engine: { + status: 'listed', + testcases: { + 'nested::required': { + ignored: false, + 'filter-match': { status: 'matches' }, + }, + }, + }, + }, + } + expect(() => checkRustInventory(inventory, suite)).not.toThrow() + delete inventory['rust-suites'].engine.testcases['nested::required'] + expect(() => checkRustInventory(inventory, suite)).toThrow( + 'Missing or incorrectly ignored', + ) + }) + + it('preserves package boundaries and fails on a missing requirement', () => { + const selected = selections(index, 'fast', 'go') + expect([...selected.values()].map((s) => s.package).sort()).toEqual([ + './encrypt', + './encrypt/gensupport', + './internal/guest', + ]) + const changed = structuredClone(index) + changed.requirements[0].evidence.go = [] + expect(() => selections(changed, 'fast', 'go')).toThrow('Invalid tests') + }) + + it('runs the lifecycle task in required CI and retains its report', () => { + const workflow = readWorkflow('.github/workflows/tests-golang.yml') + const job = workflow.jobs['wasi-check'] + expect(job.env.MISE_ENV).toBe('test') + const step = job.steps.find( + (s) => s.run === 'mise run test:conformance:lifecycle', + ) + expect(step).toBeDefined() + expect(step.if).toBeUndefined() + expect(step['continue-on-error']).toBeUndefined() + expect(step.env.STACK_CONFORMANCE_REPORT_DIR).toBeTruthy() + expect( + job.steps.some( + (s) => + s.uses?.startsWith('actions/upload-artifact@') && + s.with?.name === 'lifecycle-conformance' && + s.with['if-no-files-found'] === 'error', + ), + ).toBe(true) + for (const event of ['push', 'pull_request']) { + for (const path of [ + 'scripts/stack-conformance-evidence.mjs', + 'scripts/stack-lifecycle-conformance.mjs', + 'conformance/stack-encrypt/**', + ]) + expect(workflow.on[event].paths).toContain(path) + } + }) + + it('selects only enrolled tests unless a suite runs its whole binary', () => { + const suite = { + package: 'stack-encrypt', + binary: 'lifecycle_conformance', + features: ['--all-features'], + lane: 'fast', + tests: new Set(['a', 'b']), + } + const selected = nextestArgs(suite) + expect(selected).toEqual([ + '--locked', + '--manifest-path', + 'Cargo.toml', + '-p', + 'stack-encrypt', + '--test', + 'lifecycle_conformance', + '--all-features', + '--run-ignored', + 'default', + '-E', + 'test(=a) | test(=b)', + ]) + expect(nextestArgs(suite, { wholeBinary: true })).toEqual( + selected.slice(0, -2), + ) + const lib = nextestArgs({ ...suite, lib: true, lane: 'postgres' }) + expect(lib).toContain('--lib') + expect(lib).not.toContain('--test') + expect(lib.slice(lib.indexOf('--run-ignored'), -2)).toEqual([ + '--run-ignored', + 'only', + ]) + }) + + it('carries every limitation in the report and hashes guests only for Go', () => { + const meta = { + platform: 'linux', + architecture: 'x64', + engine: '0.0.0', + hash: (path) => `sha256:${path}`, + } + const go = buildReport(index, selections(index, 'fast', 'go'), 'go', meta) + expect(go.profile).toBe('lifecycle-v1') + expect(go.limitations.map((l) => l.id)).toEqual( + index.limitations.map((l) => l.id), + ) + expect(go.enrollment_sha256).toBe( + 'sha256:conformance/stack-encrypt/v1/lifecycle-v1.enrollment.json', + ) + expect(go.lock_sha256).toBe('sha256:Cargo.lock') + expect(Object.keys(go.guests).sort()).toEqual([ + 'encrypt/testdata/stack_encrypt_guest_deterministic.wasm', + 'encrypt/wasm/stack_encrypt_guest.wasm', + ]) + const rust = selections(index, 'fast', 'rust') + expect(buildReport(index, rust, 'rust', meta).guests).toBeUndefined() + expect( + buildReport(index, selections(index, 'fast', 'all'), 'all', meta).guests, + ).toBeDefined() + expect(() => + buildReport({ ...index, profile: 'eql-json-v1' }, rust, 'rust', meta), + ).toThrow('Unsupported lifecycle conformance profile: eql-json-v1') + }) +}) diff --git a/scripts/stack-conformance-evidence.mjs b/scripts/stack-conformance-evidence.mjs new file mode 100644 index 000000000..5689f4643 --- /dev/null +++ b/scripts/stack-conformance-evidence.mjs @@ -0,0 +1,199 @@ +// Execution and non-vacuity checks shared by conformance profiles. +import { spawnSync } from 'node:child_process' +import { createHash } from 'node:crypto' +import { readFileSync } from 'node:fs' +import { dirname, join, resolve } from 'node:path' +import { fileURLToPath } from 'node:url' + +export const root = resolve(dirname(fileURLToPath(import.meta.url)), '..') +export const read = (path) => readFileSync(join(root, path), 'utf8') +export const hash = (path) => + createHash('sha256') + .update(readFileSync(join(root, path))) + .digest('hex') + +export function selections(index, lane, language) { + if ( + index.version !== 1 || + !['fast', 'postgres'].includes(lane) || + !['rust', 'go', 'all'].includes(language) + ) { + throw new Error('Unsupported conformance version, lane or language') + } + const selected = new Map() + const ids = new Set() + for (const requirement of index.requirements) { + if (!/^[a-z][a-z0-9-]+-v1$/.test(requirement.id) || ids.has(requirement.id)) + throw new Error('Invalid or repeated requirement ID') + ids.add(requirement.id) + if (!Object.keys(requirement.evidence).length) + throw new Error(`Empty evidence: ${requirement.id}`) + for (const [id, tests] of Object.entries(requirement.evidence)) { + const suite = index.suites[id] + if ( + !suite || + !['rust', 'go'].includes(suite.language) || + !['fast', 'postgres'].includes(suite.lane) + ) + throw new Error(`Invalid suite: ${id}`) + if ( + !tests.length || + new Set(tests).size !== tests.length || + tests.some( + (name) => + !(suite.language === 'go' ? /^Test\w+$/ : /^[a-zA-Z0-9_:]+$/).test( + name, + ), + ) + ) + throw new Error(`Invalid tests: ${id}`) + if ( + suite.lane !== lane || + (language !== 'all' && language !== suite.language) + ) + continue + if (!selected.has(id)) selected.set(id, { ...suite, tests: new Set() }) + for (const name of tests) selected.get(id).tests.add(name) + } + } + if (!selected.size) throw new Error('No EQL conformance evidence selected') + return selected +} + +export function checkRustInventory(inventory, suite) { + const binary = + inventory['rust-suites'][ + suite.lib ? suite.package : `${suite.package}::${suite.binary}` + ] + if (binary?.status !== 'listed') + throw new Error(`Missing compiled binary: ${suite.binary}`) + for (const name of suite.tests) { + const test = binary.testcases[name] + if ( + !test || + test.ignored !== (suite.lane === 'postgres') || + test['filter-match'].status !== 'matches' + ) { + throw new Error( + `Missing or incorrectly ignored compiled test: ${suite.binary}::${name}`, + ) + } + } +} + +export function checkGoResults(output, tests) { + const passed = new Set() + for (const line of output.trim().split('\n')) { + if (!line) continue + const event = JSON.parse(line) + if (event.Action === 'skip' || event.Action === 'fail') + throw new Error( + `Go conformance ${event.Action}: ${event.Test ?? event.Package}`, + ) + if (event.Action === 'pass' && event.Test) passed.add(event.Test) + } + for (const name of tests) + if (!passed.has(name)) throw new Error(`Go test did not pass: ${name}`) +} + +export function checkEnvironment(env) { + for (const name of [ + 'STACK_CONFORMANCE_UPDATE', + 'STACK_ENCRYPT_UPDATE_FIXTURES', + 'STACK_ENCRYPT_UPDATE_JSON_TREE_VECTORS', + ]) { + if (Object.hasOwn(env, name)) + throw new Error(`Conformance cannot update fixtures: ${name}`) + } +} + +export function command(program, args, { capture = false, ...options } = {}) { + const result = spawnSync(program, args, { + cwd: root, + encoding: 'utf8', + maxBuffer: 64 * 1024 * 1024, + stdio: capture ? ['ignore', 'pipe', 'inherit'] : 'inherit', + ...options, + }) + if (result.error) throw result.error + if (result.status !== 0) { + if (capture && result.stdout) process.stderr.write(result.stdout) + throw new Error(`${program} failed (${result.status})`) + } + return result.stdout +} + +export function cargo(args, capture = false, workspace = '.') { + return command('mise', ['x', '--env', 'test', '--', 'cargo', ...args], { + capture, + cwd: join(root, workspace), + }) +} + +// The nextest arguments shared by `list` and `run`. A whole-binary suite (EQL) +// runs every test in its binary; otherwise only the enrolled tests run. +export function nextestArgs(suite, { wholeBinary = false } = {}) { + const args = [ + '--locked', + '--manifest-path', + 'Cargo.toml', + '-p', + suite.package, + ...(suite.lib ? ['--lib'] : ['--test', suite.binary]), + ...suite.features, + '--run-ignored', + suite.lane === 'postgres' ? 'only' : 'default', + ] + if (!wholeBinary) + args.push( + '-E', + [...suite.tests].map((name) => `test(=${name})`).join(' | '), + ) + return args +} + +export function runSuite(suite, { wholeBinary = false } = {}) { + const lane = suite.lane + if (suite.language === 'rust') { + const common = nextestArgs(suite, { wholeBinary }) + const inventory = JSON.parse( + cargo( + ['nextest', 'list', ...common, '--message-format', 'json'], + true, + suite.workspace, + ), + ) + checkRustInventory(inventory, suite) + // EQL retains whole binaries. Lifecycle selects enrolled tests from large + // unit suites. PostgreSQL tests are intentionally ignored in the fast lane. + cargo( + [ + 'nextest', + 'run', + ...common, + '--success-output', + suite.success_output ?? (lane === 'postgres' ? 'immediate' : 'never'), + ], + false, + suite.workspace, + ) + } else { + const pattern = `^(${[...suite.tests].sort().join('|')})$` + const output = command( + 'go', + [ + 'test', + '-C', + 'languages/golang', + '-count=1', + '-json', + '-run', + pattern, + suite.package ?? './encrypt', + ], + { capture: true, env: { ...process.env, CGO_ENABLED: '0' } }, + ) + checkGoResults(output, suite.tests) + console.log(`Passed ${suite.tests.size} required Go tests; no skips`) + } +} diff --git a/scripts/stack-eql-conformance.mjs b/scripts/stack-eql-conformance.mjs index 10012e74a..8aa9ef2f3 100644 --- a/scripts/stack-eql-conformance.mjs +++ b/scripts/stack-eql-conformance.mjs @@ -1,136 +1,25 @@ #!/usr/bin/env node // Named evidence over the existing suites. Missing, ignored and skipped tests // are failures; a successful empty selection is not conformance. -import { spawnSync } from 'node:child_process' -import { createHash } from 'node:crypto' -import { mkdirSync, readFileSync, writeFileSync } from 'node:fs' -import { dirname, join, resolve } from 'node:path' +import { mkdirSync, writeFileSync } from 'node:fs' +import { join } from 'node:path' import { fileURLToPath } from 'node:url' +import { + cargo, + checkEnvironment, + command, + hash, + read, + runSuite, + selections, +} from './stack-conformance-evidence.mjs' -const root = resolve(dirname(fileURLToPath(import.meta.url)), '..') -const read = (path) => readFileSync(join(root, path), 'utf8') -const hash = (path) => - createHash('sha256') - .update(readFileSync(join(root, path))) - .digest('hex') - -export function selections(index, lane, language) { - if ( - index.version !== 1 || - index.profile !== 'eql-json-v1' || - !['fast', 'postgres'].includes(lane) || - !['rust', 'go', 'all'].includes(language) - ) { - throw new Error( - 'Unsupported EQL conformance version, profile, lane or language', - ) - } - const selected = new Map() - const ids = new Set() - for (const requirement of index.requirements) { - if (!/^[a-z][a-z0-9-]+-v1$/.test(requirement.id) || ids.has(requirement.id)) - throw new Error('Invalid or repeated requirement ID') - ids.add(requirement.id) - if (!Object.keys(requirement.evidence).length) - throw new Error(`Empty evidence: ${requirement.id}`) - for (const [id, tests] of Object.entries(requirement.evidence)) { - const suite = index.suites[id] - if ( - !suite || - !['rust', 'go'].includes(suite.language) || - !['fast', 'postgres'].includes(suite.lane) - ) - throw new Error(`Invalid suite: ${id}`) - if ( - !tests.length || - new Set(tests).size !== tests.length || - tests.some( - (name) => - !(suite.language === 'go' ? /^Test\w+$/ : /^[a-zA-Z0-9_:]+$/).test( - name, - ), - ) - ) - throw new Error(`Invalid tests: ${id}`) - if ( - suite.lane !== lane || - (language !== 'all' && language !== suite.language) - ) - continue - if (!selected.has(id)) selected.set(id, { ...suite, tests: new Set() }) - for (const name of tests) selected.get(id).tests.add(name) - } - } - if (!selected.size) throw new Error('No EQL conformance evidence selected') - return selected -} - -export function checkRustInventory(inventory, suite) { - const binary = inventory['rust-suites'][`${suite.package}::${suite.binary}`] - if (binary?.status !== 'listed') - throw new Error(`Missing compiled binary: ${suite.binary}`) - for (const name of suite.tests) { - const test = binary.testcases[name] - if ( - !test || - test.ignored !== (suite.lane === 'postgres') || - test['filter-match'].status !== 'matches' - ) { - throw new Error( - `Missing or incorrectly ignored compiled test: ${suite.binary}::${name}`, - ) - } - } -} - -export function checkGoResults(output, tests) { - const passed = new Set() - for (const line of output.trim().split('\n')) { - if (!line) continue - const event = JSON.parse(line) - if (event.Action === 'skip' || event.Action === 'fail') - throw new Error( - `Go conformance ${event.Action}: ${event.Test ?? event.Package}`, - ) - if (event.Action === 'pass' && event.Test) passed.add(event.Test) - } - for (const name of tests) - if (!passed.has(name)) throw new Error(`Go test did not pass: ${name}`) -} - -export function checkEnvironment(env) { - for (const name of [ - 'STACK_CONFORMANCE_UPDATE', - 'STACK_ENCRYPT_UPDATE_FIXTURES', - 'STACK_ENCRYPT_UPDATE_JSON_TREE_VECTORS', - ]) { - if (Object.hasOwn(env, name)) - throw new Error(`Conformance cannot update fixtures: ${name}`) - } -} - -function command(program, args, { capture = false, ...options } = {}) { - const result = spawnSync(program, args, { - cwd: root, - encoding: 'utf8', - maxBuffer: 64 * 1024 * 1024, - stdio: capture ? ['ignore', 'pipe', 'inherit'] : 'inherit', - ...options, - }) - if (result.error) throw result.error - if (result.status !== 0) { - if (capture && result.stdout) process.stderr.write(result.stdout) - throw new Error(`${program} failed (${result.status})`) - } - return result.stdout -} - -function cargo(args, capture = false, workspace = '.') { - return command('mise', ['x', '--env', 'test', '--', 'cargo', ...args], { - capture, - cwd: join(root, workspace), - }) -} +export { + checkEnvironment, + checkGoResults, + checkRustInventory, + selections, +} from './stack-conformance-evidence.mjs' function metadata(workspace) { return JSON.parse( @@ -161,6 +50,8 @@ export function main(args = process.argv.slice(2)) { const index = JSON.parse( read('conformance/stack-encrypt/v1/eql-json-v1.enrollment.json'), ) + if (index.profile !== 'eql-json-v1') + throw new Error(`Unsupported EQL conformance profile: ${index.profile}`) const selected = selections(index, lane, language) process.env.STACK_CONFORMANCE_REQUIRED = '1' delete process.env.STACK_CONFORMANCE_INPUT @@ -225,59 +116,7 @@ export function main(args = process.argv.slice(2)) { } for (const [id, suite] of selected) { console.log(`EQL conformance: ${id} (${suite.tests.size} required tests)`) - if (suite.language === 'rust') { - const common = [ - '--locked', - '--manifest-path', - 'Cargo.toml', - '-p', - suite.package, - '--test', - suite.binary, - ...suite.features, - '--run-ignored', - lane === 'postgres' ? 'only' : 'default', - ] - const inventory = JSON.parse( - cargo( - ['nextest', 'list', ...common, '--message-format', 'json'], - true, - suite.workspace, - ), - ) - checkRustInventory(inventory, suite) - // Run the whole original suite, retaining unindexed regression and property - // tests. PostgreSQL tests are intentionally #[ignore] outside this lane. - cargo( - [ - 'nextest', - 'run', - ...common, - '--success-output', - lane === 'postgres' ? 'immediate' : 'never', - ], - false, - suite.workspace, - ) - } else { - const pattern = `^(${[...suite.tests].sort().join('|')})$` - const output = command( - 'go', - [ - 'test', - '-C', - 'languages/golang', - '-count=1', - '-json', - '-run', - pattern, - './encrypt', - ], - { capture: true, env: { ...process.env, CGO_ENABLED: '0' } }, - ) - checkGoResults(output, suite.tests) - console.log(`Passed ${suite.tests.size} required Go tests; no skips`) - } + runSuite(suite, { wholeBinary: true }) report.evidence[id] = [...suite.tests].sort() } console.log(JSON.stringify(report, null, 2)) diff --git a/scripts/stack-lifecycle-conformance.mjs b/scripts/stack-lifecycle-conformance.mjs new file mode 100644 index 000000000..472f6f91e --- /dev/null +++ b/scripts/stack-lifecycle-conformance.mjs @@ -0,0 +1,99 @@ +#!/usr/bin/env node +import { mkdirSync, writeFileSync } from 'node:fs' +import { join } from 'node:path' +import { fileURLToPath } from 'node:url' +import { + cargo, + checkEnvironment, + hash, + read, + runSuite, + selections, +} from './stack-conformance-evidence.mjs' + +const ENROLLMENT = 'conformance/stack-encrypt/v1/lifecycle-v1.enrollment.json' +const GUESTS = [ + 'encrypt/wasm/stack_encrypt_guest.wasm', + 'encrypt/testdata/stack_encrypt_guest_deterministic.wasm', +] + +export function lifecycleIndex(index) { + if (index.profile !== 'lifecycle-v1') + throw new Error( + `Unsupported lifecycle conformance profile: ${index.profile}`, + ) + return index +} + +// Every report carries the profile's limitations, so passing evidence is +// never read without the gaps it does not cover. The Go guests are hashed +// whenever a Go suite is selected. +export function buildReport( + index, + selected, + language, + { platform, architecture, engine, hash }, +) { + lifecycleIndex(index) + const report = { + profile: index.profile, + language, + platform, + architecture, + engine, + enrollment_sha256: hash(ENROLLMENT), + lock_sha256: hash('Cargo.lock'), + limitations: index.limitations, + evidence: {}, + } + if ([...selected.values()].some((suite) => suite.language === 'go')) + report.guests = Object.fromEntries( + GUESTS.map((name) => [name, hash(`languages/golang/${name}`)]), + ) + return report +} + +export function main(args = process.argv.slice(2)) { + const [language = 'all'] = args + if (args.length > 1) + throw new Error('Usage: stack-lifecycle-conformance.mjs [rust|go|all]') + checkEnvironment(process.env) + if (!['linux', 'darwin'].includes(process.platform)) + throw new Error( + 'The current lifecycle profile requires a Unix heap observer; this platform is not enrolled', + ) + const index = JSON.parse(read(ENROLLMENT)) + const selected = selections(lifecycleIndex(index), 'fast', language) + process.env.STACK_CONFORMANCE_REQUIRED = '1' + delete process.env.STACK_CONFORMANCE_INPUT + delete process.env.STACK_CONFORMANCE_OUTPUT + const metadata = JSON.parse( + cargo(['metadata', '--locked', '--no-deps', '--format-version', '1'], true), + ) + const report = buildReport(index, selected, language, { + platform: process.platform, + architecture: process.arch, + engine: metadata.packages.find((p) => p.name === 'stack-encrypt').version, + hash, + }) + for (const [id, suite] of selected) { + console.log( + `Lifecycle conformance: ${id} (${suite.tests.size} required tests)`, + ) + runSuite(suite) + report.evidence[id] = [...suite.tests].sort() + } + console.log(JSON.stringify(report, null, 2)) + if (process.env.STACK_CONFORMANCE_REPORT_DIR) { + mkdirSync(process.env.STACK_CONFORMANCE_REPORT_DIR, { recursive: true }) + writeFileSync( + join( + process.env.STACK_CONFORMANCE_REPORT_DIR, + `lifecycle-${language}.json`, + ), + `${JSON.stringify(report, null, 2)}\n`, + ) + } +} + +if (process.argv[1] === fileURLToPath(import.meta.url)) main()