[knittingcode]

Owner decision needed before merge (scope): the SHA-256 checksum procedure (236a110: .sha256 sidecars in steps 2a/2b, shasum -a 256 -c before every restore in steps 3 and 4) goes beyond the original task scope of this runbook. It needs the owner's explicit acceptance before merge. This PR does not decide it. Since then, b567b99 refines one sentence of the same checksum text. If the expansion is not accepted, revert b567b99 and then 236a110 (checked: applies cleanly on the current head); the rest of the runbook does not depend on them.

Summary

Adds a tested backup and restore runbook for the development database to docs/dev-environment.md (linked from the README), and removes a hard-coded project name (flowcrm-1d1ca1) from the recovery commands in docs/architecture/sessions-and-cache.md. Documentation only; no new make targets.

Motivation and technical context

The only backup path was the one inside make db-upgrade-roles (ADR-018); there was no documented, tested way to back up the development database and restore it, and no procedure that proves a backup restores before the working database is touched. The sessions-and-cache recovery steps used docker compose -p flowcrm-1d1ca1 …, the project of one particular checkout: copied into another checkout they would have targeted that other checkout's database.

Implementation

docs/dev-environment.md, new section "Backup and restore of the development database":

1. Name the project from .runtime/dev.env and define fingerprint <project> (applied migrations and the row count of every table in schema public, read as the superuser inside the db container).
2. a. Logical dump (pg_dump -Fc, with a fingerprint before and after; schema, data, ownership, grants, no role passwords). b. Or a volume archive of the stopped db-data volume (pinned busybox tar) plus a copy of .runtime/secrets, which the archive needs.
3. Rehearsal in a separate checkout (git worktree add): own project (taken from bin/devenv doctor), volume, secrets and port. Logical dump: make up, pg_restore --clean --if-exists --single-transaction --exit-on-error, make up again (checks roles, migrations, runtime privileges), fingerprint diff. Volume archive: archived secrets, a labelled volume filled with the archive before the first start, make up, fingerprint diff. Then removal of the rehearsal project.
4. Switch, only after a rehearsal of the same backup passed: a dump of the current state under its own name (flowcrm-before-restore-$pre, the way back), then the restore of the rehearsed backup ($ts) and the same verification in the working checkout.

5316e07 fixes review finding M-29-1: step 4 used to say "take a new dump (step 2a)", which reassigns ts, so the restore and its fingerprint check used the dump just taken, reported identical, and never restored the rehearsed backup. The runbook also says that its steps share variables (src, backups, ts, fingerprint) and how to define them again in a new shell (L-29-1).

236a110 adds SHA-256 checksums: steps 2a and 2b write <backup>.sha256 next to the dump or archive ((cd "$backups" && shasum -a 256 "<file>" >"<file>.sha256")), and step 3 (both kinds) and step 4 check it before the restore ((cd "$backups" && shasum -a 256 -c "<file>.sha256")). The checksum file names the backup without a directory, so the backup directory can be moved as a whole; shasum -c resolves that name in the current directory, so every check runs in $backups in a subshell. That is why it works from the rehearsal checkout (../flowcrm-restore-check): running shasum -c "$backups/<file>.sha256" from there fails with "No such file or directory" (measured). The pass condition of the rehearsal now includes the checksum.

Backups are kept outside the checkout and any volume, mode 0700. sessions-and-cache.md: project=$(sed -n 's/^COMPOSE_PROJECT_NAME=//p' .runtime/dev.env) (as the README and authentication.md do) and -p "$project" in all five commands.

Testing performed

Re-run on 236a110 (2026-10-25, auto004/d-restore-test.sh, results auto004/d-restore/steps.txt), heavy lock held 14:11-14:13. Every runbook command run as written against two disposable git worktrees in a temp directory (source flowcrm-f92ca1, rehearsal flowcrm-4a818a, ports 18640-18659), synthetic data from seed_e2e, backups in a temp directory (the only deviation from the text, besides cd - >/dev/null). The owner's project flowcrm-1d1ca1 and real backups were not read or written (containers' StartedAt unchanged afterwards).

• 2a: pg_dump 0, second fingerprint diff 0, sidecar 941a9ba3… flowcrm-20261025T131208Z.dump.

• 3 (dump), run from the rehearsal checkout: checksum flowcrm-20261025T131208Z.dump: OK (exit 0); fresh project differs; pg_restore 0; make up 0; fingerprint diff 0; volume and worktree removed.

• Controls: shasum -a 256 -c "$backups/….sha256" without the cd from the rehearsal checkout: exit 1 No such file or directory … FAILED open or read; the backup directory copied elsewhere verifies (exit 0); one changed byte in the copy: exit 1 FAILED … computed checksum did NOT match.

• 4: current state users 8 vs backup users 4; ts still the rehearsed dump; safety dump 0; checksum OK; pg_restore 0; make up 0; fingerprint equals the rehearsed backup (exit 0) and differs from the state before (exit 1); users 4. This confirms the M-29-1 fix.

• 2b: tar 0 (50.2 MB), sidecar written, secrets copied, make up 0. 3v: checksum OK, secrets, labelled volume, tar -xpf 0, make up 0, fingerprint diff 0, cleanup.

• Cleanup: 0 containers, volumes and networks left for both projects; no worktree left registered.

• scripts/ci/guard-tracked-files.sh: OK. git merge-tree with origin/main e03d151, feat(organizations): membership administration API (FND-010C) #19 (3534fe8), feat(frontend): member administration page (FND-010C, stacked on #19) #20 (8051d68), test(security): pin tenant route permissions and audit every operator user change #22, fix(devenv): fail closed when the edge does not serve again after a rotation #15, docs: describe the implemented backend modules, authentication and the Api rule #25, docs(api): declare the 500 of session routes when the session store fails #31: clean.

• CI on 236a110: run 40196656960, success.

Earlier run on 5316e07 (Agent C)

Re-run on 5316e07 (2026-10-25, c-restore-test2.sh, c-restore2/steps.txt), step 4 run literally right after the logical-dump rehearsal, with the current state changed first (more synthetic users) so a wrong restore is visible:

• 2a dump (users 4); rehearsal in flowcrm-37976b: pg_restore 0, make up 0, fingerprint diff empty;
• 4: current state differs from the backup (users 8 vs 4); ts still names the rehearsed dump; dump of the current state 0; pg_restore of the rehearsed backup 0; make up 0; fingerprint equals the rehearsed backup (diff exit 0) and differs from the state before the restore (exit 1); users 4;
• negative control, the previous step-4 text (re-running 2a): reports "identical" while users is 8, i.e. the rehearsed backup was not restored — the bug the fix removes;
• 2b/3 volume archive: tar 0, make up 0, rehearsal make up 0, diff empty; cleanup 0 resources left.

First run (85ce69e):

Every command of the runbook was run as written (bash) by a script against disposable git worktrees of this branch with synthetic data (users, organizations, memberships via the operator console, seed_e2e of tests/app/lib.sh), backups in a temp directory instead of ~/flowcrm-backups; ports 18600-18619. Results (2026-10-24, macOS, Docker Desktop):

• source make up 0; fingerprint migrations|4, audit_events|11, memberships|5, organizations|2, users|4, …; the sessions-and-cache command docker compose -p "$project" exec -T db psql … 'SELECT current_database(), current_schema()' exit 0 from the checkout;
• 2a: pg_dump 0, fingerprint unchanged;
• 3 (dump): new project flowcrm-0b4ca0 ≠ source flowcrm-4968d4; fresh fingerprint differs; pg_restore 0; make up 0; fingerprint diff empty; make url serves; volume and worktree removed;
• 2b: tar 0 (49.9 MB), secrets copied (drwx------), make up 0;
• 3 (archive): secrets, docker volume create with Compose labels, tar -xpf 0, make up 0 (no warning), fingerprint diff empty, cleanup 0;
• 4: after new data the source differs; safety dump 0; pg_restore 0; make up 0; fingerprint diff empty (this run reset ts by hand before step 4, so it did not follow the text literally and missed M-29-1);
• all disposable projects removed (0 containers, volumes, networks left), no rehearsal worktree left registered.
  scripts/ci/guard-tracked-files.sh OK. The script is not part of the repository (documentation-only change); CI runs on the push.

Updated on 2026-10-28: brought up to date with main 202ea9e (#39), plus one doc fix

• cc27e1d chore(merge): bring main (#39) into the branch: normal merge. docs/dev-environment.md auto-merged with no conflicts.
• cd5dc36 docs(dev-environment): describe the blocked stack after a failed make up in the restore switch: since fix(devenv): fail closed when the edge does not serve again after a rotation #15, a failed check of make up stops the edge and records .runtime/gate/security-gate. While the gate exists, an edge started by bin/devenv answers 503 except /health/live and /health/ready, and make status reports blocked until a make up passes. Step 4 now says so, and says that the way back still works, because the database keeps running. Checked against gate_failed in bin/devenv, which stops only the edge ("the database and its volume are kept"), and against phase 7 of the same document.

Validation of the backup and checksum steps, 2026-10-28, disposable resources only. The heavy-Docker lock was held from 10:06:01Z to 10:07:24Z. The resource prefix was int7e-7276d0. Every container and volume was named int7e-7276d0-* and removed by name afterwards. The owner's flowcrm-1d1ca1 stack was not used: its four containers kept the same StartedAt values. The existing backup archives and .runtime/secrets were not read or written. Container, volume and network inventories were identical before and after. The step 1 fingerprint function and the commands of 2a, 2b, 3 (both kinds) and 4 were copied verbatim from the runbook. Adaptations:

• Instead of a stack from make up, each project was one bare db container on the pinned base image of docker/db/Dockerfile (postgres:18.6-alpine3.24@sha256:77f58511…), --network none, with the Compose labels com.docker.compose.project=<name> and com.docker.compose.service=db. The runbook's label filters therefore found it unchanged. The volume was mounted at /var/lib/postgresql, as in compose.yaml. The data was synthetic: 4 migrations, 4 users, 2 organizations, 5 memberships.
• src/chk were set by hand (int7e-7276d0-src, -chk, -chkv) instead of from .runtime/dev.env and bin/devenv doctor. No git worktree was used.
• backups was a directory in the agent's scratch space (mode drwx------) instead of ~/flowcrm-backups.
• docker stop/docker start replaced make down/make up. .runtime/secrets was not copied, because the bare container has no project secrets.
• Not covered by this validation: the make up checks after a restore (roles, migrations, runtime privileges, exit 6) and the blocked state described above. They need a full disposable stack. The full-stack runs on 236a110 and earlier are described further down.

Results (exit codes):

• 2a: pg_dump 0; second fingerprint unchanged; sidecar 5a7cc66c… flowcrm-20261028T100705Z.dump (bare file name).

• Checksum: in $backups flowcrm-…dump: OK (exit 0). Without cd, from another directory: exit 1, No such file or directory … FAILED open or read. The backup directory copied elsewhere verifies (exit 0).

• Negative controls on copies: one byte changed in the middle of a copied dump (cmp confirms the difference): shasum -a 256 -c exit 1 FAILED … computed checksum did NOT match, and sha256sum -c also reports FAILED. A truncated copy: exit 1. The original still verifies afterwards (exit 0).

• 3 (dump): the fresh rehearsal differs before the restore (diff exit 1); checksum OK; pg_restore --clean --if-exists --single-transaction --exit-on-error 0; fingerprint identical.

• 2b: source stopped; busybox tar 0 (49,476,608 bytes); sidecar written. 3v: checksum OK; docker volume create with labels 0; tar -xpf 0; db started on the restored volume; fingerprint identical.

• 4: the current state differs from the backup (users 8 vs 4); ts still names the rehearsed dump; safety dump flowcrm-before-restore-… 0; checksum OK; pg_restore 0; fingerprint equals the rehearsed backup (exit 0) and differs from the state before (exit 1); users 4.

• Observation: the files inside $backups are created with the default umask (0644). They are protected only by the directory's 0700 mode, which is what the runbook's mkdir -m 700 gives.

• scripts/ci/guard-tracked-files.sh OK; git diff --check clean. git merge-tree against main 999d4d1 (test(stack): report why an interrupted restore check of ST-50 failed #38) and against docs(security): record the undici findings of npm's bundled undici (CVE-2026-109869, CVE-2026-109383, CVE-2026-87018) #24 and docs: describe the implemented backend modules, authentication and the Api rule #25: clean.

• CI on head cd5dc36 (merge commit 5781701 into 202ea9e): run 40669666687, failure. Repository guard succeeded (TOTAL 36/36, 73/73, 46/46). Allocator unit tests: TOTAL 288 passed=288 failed=0. Allocator integration tests: TOTAL 140 passed=140 failed=0. Backend checks: TOTAL 18 passed=17 failed=1. Only BE-03 composer audit failed, on advisory GHSA-9vw2-f4vp-x7gf for phpunit/phpunit 13.4.3; BE-18 reported 53 of 53. The frontend, application stack, production-like and Trivy steps were skipped. This failure is not caused by the PR: main fails the same way, and the lockfile fix is a separate PR. The run was not re-run. The PR needs a fresh run after the fix.

Updated on 2026-10-28 (second time): brought up to date with main 1badf76 (#38, #42), plus one checksum wording fix

• c5c518b chore(merge): bring main (#38, #42) into the branch: normal merge; docs/dev-environment.md auto-merged with fix(security): remediate dependency advisories and guard Caddy exposure (ADR-021) #42's scan paragraph, no conflicts.
• b567b99 docs(dev-environment): say which shasum commands sha256sum replaces in the backup checks (part of the checksum expansion, see the top): "sha256sum and sha256sum -c work the same way" now says that, where shasum is missing, they replace shasum -a 256 and shasum -a 256 -c and use the same file format. Checked locally: a file written by shasum -a 256 is byte-identical to one written by sha256sum, and each tool's -c verifies the other's file (exit 0).
• No Docker rehearsal was re-run: since the 2026-10-28 validation (base 202ea9e), main changed nothing in bin/, Makefile, compose*.yaml or docker/db, and docs/dev-environment.md only in the container-scan paragraph; no runbook procedure changed. gate_failed in bin/devenv is unchanged, so the step 4 note of cd5dc36 still holds.
• Revertability of the checksum expansion: reverting b567b99 and then 236a110 on head b567b99 applies cleanly (checked on a throwaway local branch, not pushed); 236a110 alone no longer reverts cleanly because b567b99 edits its text.
• scripts/ci/guard-tracked-files.sh OK; git diff --check clean.
• CI on head b567b99 (workflow CI, pull_request, merged into main 1badf76): run 40752541320, success; every step of Repository guard and Dev environment succeeded, none skipped. TOTAL lines: guard regression 36/36, Dockerfile frontend guard 73/73, scan exception guard 87/87, edge configuration guard 221/221; allocator unit 288/288, allocator integration 140/140 (24 scenarios); backend 18/18 (BE-18 53 of 53); frontend 5/5; application stack 124/124; production-like stack 39/39; container scan 20/20. The BE-03 failure of the previous run (40669666687) is gone because fix(security): remediate dependency advisories and guard Caddy exposure (ADR-021) #42 updated phpunit/phpunit to 13.4.5.

Security considerations

Backups contain all application data and the volume archive also the role password hashes and, with the secrets copy, the passwords: the runbook keeps them outside the checkout with mode 0700. The rehearsal never touches the working project; the switch requires a passed rehearsal and a new dump of the current state first. Commands derive the project from the checkout, so they cannot target another checkout's database by copy-paste.

Known limitations

• Development database only; a production backup concept is out of scope.
• The volume archive needs the same PostgreSQL major version and the archived secrets.
• Requests during a switch may fail; sessions in the restored data are those of the backup.
• The runbook is verified manually by the script described above, not by a CI test.
• The runbook has not been re-run end to end with make up against post-fix(devenv): fail closed when the edge does not serve again after a rotation #15 bin/devenv. The 2026-10-28 validation covers the backup, checksum and restore commands on bare db containers only (see above).
• The SHA-256 checksum steps (236a110, b567b99) need the owner's scope acceptance (see the top of this description).
• Backup files inside $backups are created with the default umask (0644); only the directory's 0700 mode protects them (observed in the 2026-10-28 validation; the runbook text is unchanged).