[tobyhede]

Summary

Proves that data written by Stack 1.x can be moved to the new EQL v4 format and keeps answering the same database queries. EQL is the format CipherStash uses to store encrypted values in Postgres. v3 is what Stack 1.x writes today; v4 is what the new Stack Encrypt engine writes. The two use different ciphertext and search-term formats, so moving data means decrypting and re-encrypting every value. Until now no test did that.

First of two stacked PRs. The second covers EQL v2 (Protect.js) data, and is stacked on this one.

Changes

New live test: integration-tests/tests/migrate-v3-to-v4.test.ts (11 tests). It runs the migration a customer would run, in one table, against real ZeroKMS (the key service) and Postgres with EQL v3 and v4 installed together:

1. Write v3 rows with protect-ffi in v3 mode, the engine @cipherstash/stack 1.x encrypts through.
2. Add a v4 column beside each v3 column.
3. Backfill: decrypt each v3 value and re-encrypt it as v4.
4. Check that equality, ORE range, ORE and OPE ordering, and text-match queries return the same rows from the v3 and v4 columns, and match a plaintext oracle.
5. Check that every v4 value decrypts to the original plaintext.
6. Drop the v3 columns, rename the v4 ones into their place, and check that queries still work. The v4 values are encrypted under the logical column name (customers/email, not customers/email_v4), so the rename doesn't break them.

It also pins three mistakes:

• casting a v3 payload into a v4 column fails Postgres's domain check;
• changing v to 4 on a v3 payload gives a value Stack Encrypt refuses to open;
• querying a v4 column with a v3 query term silently returns no rows.

v4 encoder: packages/eql/tests/encryption/examples/v4_codec.rs. No TypeScript SDK writes v4 yet, so the test sends plaintexts to a small Rust program that reads JSON on stdin and writes JSON on stdout. It encrypts, builds query terms and decrypts through eql_bindings::encryption::targets, the same by-name entry point the Go SDK uses, against live ZeroKMS. To build a cipher from environment credentials, the test crate turns on stack-encrypt's http feature, which adds lines to packages/eql/Cargo.lock. The crate's existing tests still use fake key sources. The TypeScript side is tests/v4-codec.ts.

Setup and CI

• integration-tests/tasks.toml: eql:v4:install builds the v4 SQL bundle from the in-tree EQL source and installs it. eql:v4:codec builds the encoder. Both are part of mise run setup.
• .github/workflows/integration-protect-ffi.yml: adds a Rust cache for the EQL workspace and path filters for the inputs those two tasks read. packages/eql/crates/** is deliberately absent: wasm-build-inputs-paths-filter.test.mjs forbids it here, and test-eql.yml already covers it.
• scripts/__tests__/tests-workflow-gate.test.mjs: records why v4-codec.ts may read packages/eql/target. Only the ungated live workflow runs it.
• AGENTS.md: the integration suite section now covers v4 and the encoder.

No changeset: only test code changed. The integration-tests package is private and the EQL test crate is publish = false.

Verification

Run locally against live ZeroKMS and the suite's Docker Postgres with EQL v2, v3 and v4 installed:

• vitest run tests/migrate-v3-to-v4.test.ts tests/postgres-v3.test.ts tests/postgres.test.ts: 25/25 passed, including all 11 new tests.
• pnpm run test:scripts: 85 files passed.
• protect-ffi src/lintWiring.test.ts and src/integrationSuiteCi.test.ts: passed.
• cargo test --locked -p eql-encryption-tests: passed. cargo clippy -D warnings on the example: clean. cargo fmt --check: clean.
• mise run eql:v4:install run twice against the same database: both runs succeed.
• pnpm run typecheck in integration-tests: clean. Biome: no new findings.

The CI run of integration-protect-ffi.yml on this PR is the first time the setup runs on a fresh runner.

Related

Refs #1436 (the Protect.js → Stack 1.x DynamoDB compatibility suite this extends).

Review notes

• Start with the comment at the top of the test file, then v4_codec.rs.
• Only scalar types are covered. v4 JSON uses a new tree format that the encoder doesn't handle, and SteVec documents can't be converted.
• The relabel test shows that the v4 query term differs from the relabelled v3 one. It doesn't insert the relabelled payload into a v4 column.

[changeset-bot]

⚠️ No Changeset found

Latest commit: 4968a46

Merging this PR will not cause a version bump for any packages. If these changes should not result in a new version, you're good to go. If these changes should result in a version bump, you need to add a changeset.

This PR includes no changesets

When changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types

Click here to learn what changesets are, and how to add one.

Click here if you're a maintainer who wants to add a changeset to this PR

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true