[knittingcode]

Summary

When one of the three ST-50 checks built on interrupted_restore fails, it now reports which step failed and why. These are the SIGKILL and SIGINT checks right after the edge started again, and the SIGTERM check before it.

This is a test-only change to tests/app/stack.sh. No condition is removed, loosened, reordered or retried, and no wait or timeout changes.

Motivation and technical context

CI run 40215242347 (on #35's synced head 325471d) failed a check that #35 does not touch:

FAIL  ST-50 SIGTERM before the edge started again: the gate stays recorded, nothing listens, make up serves
      exit 6, edge [], gate recorded, probe: /=000 ... /health/ready/x=000, make up 0

• The rotation runs with fault restore-before-edge and should be killed by SIGTERM just before restore_edge, exiting 143. It exited 6 (EX_SAFETY) instead.

• The stack stayed fail-closed: no edge, gate recorded, every probe 000. The following make up recovered (exit 0).

• In cmd_rotate_db_password the exit-6 paths before restore_edge are:

  • rotation_prepare;
  • protect_for;
  • "setting the new password failed";
  • "cannot log in with the new password";
  • the file promotion;
  • verify_role_client.

  refresh_secret_mounts fails with exit 2, not 6. The report did not include the rotation's stderr, so the path cannot be identified.

The same pattern now appears in three checks, each passing on rerun:

Check	CI run	Observed
ST-64, make up with up-after-migrations:kill	40176460595 a1	up=6/000/yes/0: exit 6 before the migration, edge down, gate recorded
ST-50 owner rotation resumed	40186923380 a1	condition unknown. The run was shorter than passing runs, which fits an early exit such as a rotation exit of 6 (not proven)
ST-50 SIGTERM before the edge started again	40215242347	exit 6 instead of 143, fail-closed state, make up recovered

• All three are a fail-closed refusal (exit 6) by bin/devenv shortly after earlier steps recreated the db or api containers (refresh_secret_mounts, make up).
• This fits the db-recreation-timing hypothesis from the earlier investigation, but it is not confirmed.
• test(stack): report which condition failed in ST-50 owner and ST-64 #35 adds the same kind of reporting to ST-50 owner and ST-64.

Implementation

interrupted_restore keeps its condition chain unchanged and records its status.

When that status is a failure, it builds IR_DIAG after the checks. IR_DIAG never contains a secret value and holds:

• the exit code of make status and its first line (with the real devenv this is the project: … line; the blocked: … verdict appears in the full status> lines printed below the check);
• whether the gate is still recorded after make up;
• a fresh edge probe;
• the lifecycle line (db and api container ID, start time, restarts, health and secret-mount comparison).

Each of the three checks:

• keeps its condition;
• captures its own status (ir=$?) right after it;
• appends IR_DIAG to the report;
• on failure, calls ir_details, which shows the stderr tail of the interrupted command, the output and stderr of make status, and the stderr of make up.

On a pass, nothing extra is computed or printed.

Testing performed

The branch is synced with main 202ea9e (#39) at bfbde64. Against main it changes only interrupted_restore, adds ir_details and changes the three ST-50 checks; #35's ST-50 owner and ST-64 reporting is unchanged.

• bash -n tests/app/stack.sh: OK.
• ShellCheck v0.11.0 (the version CI uses), on every script CI checks: no findings.
• tests/devenv/unit.sh on 843a00a and on bfbde64: TOTAL 287 passed=287 failed=0 (macOS). U-16 needs passwordless sudo and runs only in CI.
• I ran the three checks' code from stack.sh with stubbed Docker, curl and devenv, for main 5e4a150 and this branch:
  • all rotations exit as expected (137, 130, 143): every check passes on both, and neither prints anything extra;
  • the SIGTERM rotation exits 6: both fail the SIGTERM check with the same verdict. Main prints only exit 6, edge [], gate recorded, probe: …, make up 0. This branch adds status 1 [<first status line>], after make up: gate removed, probe: …, db=… api=…, then the rotation's stderr (| devenv: ERROR: …), the status> lines and make up's stderr;
  • the SIGKILL rotation exits 6: the same, for the SIGKILL check only.
  • The stub secret files hold a marker value, and it does not appear in any output.
• The full stack suite was not run locally.
• CI run 40412225938 on bfbde64: success. Repository guard and Dev environment pass every step.
  • The three ST-50 interrupted-restore checks (SIGKILL, SIGINT, SIGTERM) pass and print nothing extra (no status> lines).
  • Application stack TOTAL 124 passed=124 failed=0, production-like stack 39/39, allocator unit 288/288, integration 140/140.
  • Trivy: SC api-prod, SC edge-prod and SC db (runtime) report no HIGH or CRITICAL vulnerabilities, TOTAL 14 passed=14 failed=0.
• The earlier run 40383573386 on 843a00a failed only at Trivy (curl/libcurl CVE-2026-92392 in the runtime images), which fix(security): upgrade curl and libcurl in runtime images #39 fixed on main.

Security considerations

The added output is exit codes, make status text, HTTP status codes, container IDs and timestamps, secret-mount comparisons (same, different or missing) and devenv's stderr. devenv never prints password values. The fail-closed behaviour of bin/devenv is unchanged.

Known limitations

• This does not fix the exit-6 flake. It makes the next occurrence name the devenv error. A fix follows once that evidence exists, keeping the refusal fail-closed.
• The edge probe in the diagnostics is a single snapshot with no wait. If a check stops before make up finished starting the edge, the probe can show 000 or 502 while the edge is still starting.
• If interrupted_restore passes but the check's own edge or probe condition fails, there is no IR_DIAG; only the stderr and make status details are printed.

Note (review, FCRM-INT-007): the stubbed make status in the harness printed blocked: … first; the real make status prints project: … first. The status excerpt is therefore less specific than intended, but no information is lost, because ir_details prints the last 15 lines of the status output, which include the blocked: line.