Skip to content

chore(creds): both platform keys rotated — tripwire goes green on the fix, not the escape - #321

Merged
Polichinel merged 3 commits into
developmentfrom
chore/key-rotation-2026-10-29
Oct 29, 2026
Merged

Polichinel merged 3 commits into
developmentfrom
chore/key-rotation-2026-10-29

Conversation

@Polichinel

Copy link
Copy Markdown
Collaborator

What this does

Both platform Appwrite keys were rotated on 2026-10-29. This updates KEY_EXPIRY
to the successors' dates, rewrites the tripwire's warning text to describe the
world that now exists, and closes C-84 and C-110.

ACKNOWLEDGED_UNTIL was never set. The check goes green on the underlying
fact, not on an acknowledgement.

The rotation

was now scopes verified before the old one was stood down
VIEWS Pipeline Core pipeline-core-writer 6 load-bearing appwrite_store STORE_ACTIVE (1007 files); unfao_delivery DELIVERING; 3 collections, 1006 documents
UN FAO / faoapi-legacy-shared-RETIRING faoapi-server 4, all reads faoapi smoke test ALL PASS, appwrite=True, both legs serving 1,030 cells

C-84 fired on 2026-10-18 exactly as predicted and held main closed for eleven
days with nothing queued behind it. The date did not depend on anyone remembering
it, which was the entire point of building it.

Why the warning text changed

It described a property the keys no longer have. "Neither can carry traffic while
the other is replaced"
was true because both were 36-scope keys shared across
every identity on the seam
. The successors serve different consumers with
different scopes, so the next rotation is two independent, routine operations.

The message now says to rotate them separately, carries the create-before-delete
procedure, and test_the_tripwire_actually_fires_inside_the_lead_window asserts
that it does. Leaving the old all-at-once warning in place would have taught the
next operator something false.

The finding this PR exists to preserve

Scope sets derived from source were wrong twice — once by the pipeline-core
seat tracing every SDK call site, once independently from this seat. Appwrite has
split the scope vocabulary into documentsdb.*, tables.*/rows.* and
vectorsdb.*, and what actually authorises appwrite==19.2.0 is the three scopes
the console tags Deprecated. The server named them itself:

401 general_unauthorized_scope  missing scopes (["collections.read"])
401 general_unauthorized_scope  missing scopes (["documents.read"])

views-faoapi's own guard had already written the rule — "That is Appwrite's
contract, not ours, and no local test can establish it — only a live round trip
can"
— and their C-282 was the first instance of this exact error.

Create-before-delete made the rediscovery free. The old key carried production
throughout; the cost of being wrong was one console edit.

ADRs requested at views-pipeline-core#572 and views-faoapi#543, because a
scope table recorded without why source analysis cannot produce it will be wrong
again at the next SDK bump.

Also recorded

Shortening LEAD_DAYS from 30 to 7 was attempted during the incident to quiet the
alarm. Measured, it does not clear the check — it reddens
test_the_lead_window_is_not_quietly_shrunk and the firing-path probe instead, so
buying a shorter window costs deleting the two guards that keep the tripwire honest
for the next rotation. The guard did exactly the job its own message describes. The
obvious shortcut is closed, deliberately, and C-110 now says so.

Decisions taken here rather than asked about

  • Dates are declared, not read back. Each key was created with a one-year
    expiry, so the date is known; the times are approximate (creation order, an
    hour apart) and the comment says so explicitly. At a 30-day window an hour is
    immaterial, and claiming a console reading we did not take is the failure that
    comment exists to prevent.
  • Entry names follow the console, not the old registry labels, so the next
    reader can find them.
  • C-110 closed as "resolved by a different remedy than proposed." It asked for
    a runbook in docs/operations/; what exists instead is the rewritten failure
    message. That is arguably the better home — you meet it as a red check, not by
    opening a document — but it is not what the entry proposed, and the entry says so
    rather than claiming the plan was followed.

Residual, deliberately not chased

  • files.write / documents.write on pipeline-core-writer are unproven —
    verifying them means writing to the production store. The old key remains
    available until a real producer run exercises them.
  • Whether the FAO have adopted faoapi-caller-fao, or still hold only the retired
    shared key, is unconfirmed and tracked at views-faoapi#543.

Tests

526 passed, 3 skipped, 41 xfailed locally. The one local failure is
test_the_installed_runtime_dependencies_match_the_lockfile, which correctly
reports that this run used a borrowed 3.11 conda env rather than the repo's locked
environment — CI uses the locked one.

The register's own integrity guards caught three real mistakes in the first draft
of this change (entries renamed but left under Open Concerns, header counts
stale, and a foreign C-282 reference not namespaced to views-faoapi). All fixed;
test_register_integrity.py is green.

🤖 Generated with Claude Code

https://claude.ai/code/session_01ANY1CCy9Xo7zjMY4XJ69v9

Polichinel and others added 3 commits October 30, 2026 00:17
… the fix, not the escape

C-84 predicted a total seam outage on 2026-11-17 and C-110 predicted that the
tripwire guarding it would close `main` from 2026-10-18. Both happened, on the
day, and `main` was gated for eleven days with nothing queued behind it.

Cleared by rotating, not by acknowledging. `ACKNOWLEDGED_UNTIL` was never set.

  VIEWS Pipeline Core (36 scopes) -> pipeline-core-writer (6 load-bearing)
  UN FAO / faoapi-legacy-shared-RETIRING (36) -> faoapi-server (4, all reads)

Each successor was verified live BEFORE its predecessor was stood down:
appwrite_store STORE_ACTIVE (1007 files), unfao_delivery DELIVERING, 3
collections / 1006 documents, and faoapi's own smoke test ALL PASS with
appwrite=True and both legs serving 1,030 cells.

KEY_EXPIRY now carries 2027-10-29 for both. The dates are DECLARED (one-year
expiry set at creation), not read back from the console, and the comment says
so — the times are approximate and immaterial at a 30-day window.

The warning text is rewritten because it described a world that no longer
exists. "Neither can carry traffic while the other is replaced" was a property
of the keys being SHARED across every identity on the seam. The successors
serve different consumers with different scopes, so the next rotation is two
independent operations; the message now says to rotate them separately, and a
guard asserts that it does.

The finding worth keeping: scope sets derived from source were wrong TWICE —
once by the pipeline-core seat tracing every call site, once independently
here. Appwrite split the vocabulary into documentsdb.* / tables.*+rows.* /
vectorsdb.*, and what authorises appwrite==19.2.0 is the three scopes the
console tags Deprecated. The server named them itself:

    401 general_unauthorized_scope  missing scopes (["collections.read"])
    401 general_unauthorized_scope  missing scopes (["documents.read"])

views-faoapi's own guard had already written the rule, and their C-282 was the
first instance of this exact error. Create-before-delete made the rediscovery
free: the old key carried production throughout. ADRs requested at
views-pipeline-core#572 and views-faoapi#543.

Also recorded in C-110: shortening LEAD_DAYS from 30 to 7 was attempted during
the incident and does not clear the check — measured, it reddens two other
guards instead. The obvious shortcut is closed, deliberately.

Residual, not chased: files.write/documents.write on pipeline-core-writer are
unproven (verifying means writing to production); whether the FAO have adopted
faoapi-caller-fao is unconfirmed and tracked at views-faoapi#543.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ANY1CCy9Xo7zjMY4XJ69v9
Moving C-84 and C-110 into Resolved Concerns appended a separator each
where one already followed, leaving `---` / blank / `---`. Adjacent
separator pairs went 11 -> 13; now back to 11, matching development.

Cosmetic, and caught by the review's own summary pass rather than by a
test — `test_register_integrity.py` does not assert separator shape.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ANY1CCy9Xo7zjMY4XJ69v9
…ented values

Three findings from the review of 7c4ca88, all real.

1. The diff contradicted itself about the old keys. The docstring said they
   "were stood down"; the register said one "remains available" as the
   fallback for the unproven write scopes. Both were in the same change and
   they cannot both be true. Corrected to what actually happened: the
   predecessors still EXIST, nothing authenticates with them, and they were
   kept rather than deleted.

   The sharper half of that finding: if the old key is the standing fallback,
   it expires 2026-11-17 and this commit had just removed that date from
   KEY_EXPIRY — creating exactly the unguarded date this file exists to
   prevent. Now stated outright in both places: the retained fallback has a
   deadline, the tripwire deliberately does not guard it (nothing
   authenticates with it, so it is not this guard's subject), and that is
   acceptable only because the fallback is a convenience rather than a
   dependency — a missing write scope is fixed by a console edit, which is
   what fixed the two read scopes on the day.

2. `assert name == "pipeline-core-writer"` pinned an ordering the same diff
   admits it invented. The old `name == "VIEWS Pipeline Core"` asserted a
   console fact: 12:35 vs 16:10, and the earlier key was the one both delivery
   paths used. After the rotation the ordering is an artefact of two
   approximate creation times, so the assertion would fail the day someone
   reads the real values and finds the other key came first — a guard refusing
   its own documented fix, which is the shape of the literal pin this file
   already removed once. Dropped, with the reasoning in the docstring.

   `name in KEY_EXPIRY` stood in its place briefly and was removed too:
   `days_until_earliest_expiry` returns `min(KEY_EXPIRY, ...)`, so it cannot
   fail. C-102's family. An assertion that cannot fire reads as cover.

3. `"1h00m apart"` had the same defect. Under the old constants 3h35m was
   C-84's actual finding and worth pinning; it is now an artefact of the same
   invented times. Replaced with a format assertion, `\d+h\d{2}m apart`, which
   proves the gap still renders without pinning a number this file says it
   invented. Mutation-proven: deleting {gap} from the message fails it.

Suite: 526 passed, 3 skipped, 41 xfailed. ruff clean.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01ANY1CCy9Xo7zjMY4XJ69v9
@Polichinel
Polichinel merged commit 0c0adff into development Oct 29, 2026
4 checks passed
@Polichinel
Polichinel deleted the chore/key-rotation-2026-10-29 branch October 29, 2026 23:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant