[knittingcode]

Summary

Makes two intermittent failures of tests/app/stack.sh report their cause. The root cause has not been identified yet: the CI output does not contain the information needed, and the failures did not reproduce locally. Only test code changes: no check is removed or loosened.

• ST-64 "SIGKILL after a committed change but before its verification" failed once on test(security): pin tenant route permissions and audit every operator user change #22, CI run 40176460595 attempt 1, with rot=137/000/yes up=6/000/yes/0. The make up that should be killed after the migrations (FLOWCRM_DEVENV_FAULT=up-after-migrations:kill) exited 6 instead of 137, and the migration was not applied. It passed on rerun.
• ST-50 "owner rotation interrupted after the database change and resumed" failed once on feat(organizations): membership administration API (FND-010C) #19, CI run 40186923380 attempt 1. The report showed only the last two lines of the resumed rotation, which had succeeded. It passed on rerun.

Motivation and technical context

Neither report could say which condition failed:

• ST-50 joins six conditions with && and prints tail -n 2 of the resume.
• ST-64 prints exit codes but not the stderr of the commands.

What the evidence shows:

• Exit 6 in ST-64: make up stops with exit 6 (EX_SAFETY) before the migrations only through these paths:

  • check_roles → gate_failed, for role-state exits 10, 11 or 14;
  • protect_for (gate write, or an edge that could not be stopped);
  • ensure_secrets;
  • require_owned_project;
  • the lock-owner record.

  The observed 000 and blocked (edge down, gate present, no migration) fit check_roles → gate_failed and a protect_for refusal. Which one it was is not in the log.

• Timing of the ST-50 failure: it took 8.2 s, against 9.1–9.6 s in passing runs. That suggests a condition before the migrate-job check failed, perhaps the exit code of the interrupted rotation, but it is not proof.

• Common pattern: both failures came a few seconds after a db recreation by refresh_secret_mounts (make up after ST-50's rotate-after-promote, and the rotation finished before ST-64's make up).

Investigation (local, Docker Desktop; evidence in the task notes)

1. Focused loop of the exact ST-50 owner and ST-64 sequences on e03d151: 8 + 8 iterations, 16/16 passed. Instrumented ST-64 on 266c729: 2/2 passed. Container IDs show that the make up in ST-64 does not recreate the db: it had been up for about 7 s when check ran.
2. Hypothesis: docker compose ps --quiet edge still lists the edge right after docker compose stop edge returns (which would trigger protect_for's "could not be stopped"). 150 stop/ps cycles: 0 occurrences.
3. Hypothesis: flowcrm-db-roles state fails on a healthy db right after a recreation (a role-state failure). A 0.2 s poller ran during 10 full rotations, each recreating db and api. All 8 anomalies were while the old container was shutting down or being removed, or the new one was still starting, which devenv waits out. Against a healthy db that was not being replaced: 0. A psql error inside state also makes check exit 2 (EX_DOCKER), not 6.

Neither hypothesis reproduced, so no change to bin/devenv is proposed: changing fail-closed code without a confirmed cause would risk weakening it.

Implementation

• ST-50 owner: every condition is evaluated and reported:

  • the exit codes of rotate-after-db and of the resume;
  • the pending file;
  • the new and the old owner login, with psql's last message (never a password) through a new helper, login_result;
  • the migrate job's doctrine:migrations:up-to-date result with its last lines;
  • leftover temporary files;
  • the existing lifecycle line (container identity and secret-mount comparison).

  On failure it shows the stderr of both rotation commands. The old login must still be refused, and a missing file no longer counts as a refusal.

• login_result and login_ok share one criterion: a login succeeded when psql printed the row (a line 1 on stdout), whatever it wrote to stderr. stderr is kept for the diagnostic only, and login_ok is defined through login_result. (The first version compared psql's combined stdout and stderr with 1: a warning on stderr next to the returned row would have made a working old password count as refused, which is looser than login_ok.) New unit checks U-25 in tests/devenv/unit.sh cover a returned row, a row with a stderr warning, another row, a refusal, a docker error, no answer, and a missing or empty password file.

• ST-64: the report adds the exit codes of the finishing rotation and of the recovering make up. On failure it shows the stderr of the finishing rotation, the killed make up and the recovery.

Testing performed

• bash -n; ShellCheck v0.11.0 (the CI image) on every script CI checks: no findings.

• tests/devenv/unit.sh: 287 passed locally. Against a copy of stack.sh with the first login_result criterion, U-25 fails 2 cases (the stderr warning and no answer).

• Diagnostics check: the full stack suite ran locally (Docker Desktop) on a scratch copy of b33ad129 in which one condition of each check was forced false (old = ok in ST-50 owner, migrated = 2 in ST-64). Both failed and named the violated condition:

  • ST-50 owner: rc=143/0 pending=none new=ok old=psql: error: ... FATAL: password authentication failed for user "flowcrm_owner" migrate=ok leftovers=none db=..., followed by the stderr of both rotations;
  • ST-64: rot=137/000/yes finish=0 up=137/000/yes/1 recovery=0, followed by the stderr of the finish, the killed make up and the recovery.

  Every other application check passed. ST-34 is not part of a Docker Desktop run. Two CLEAN checks failed because other disposable stacks running at the same time were removed during the run.

• CI run 40208960751 on b33ad129: success (unit 288/288, application stack 121/121 including ST-50 owner and ST-64, production-like 38/38).

Security considerations

• Test-only change. psql's error line and the lifecycle line contain no secret values: passwords are passed through the environment and never printed.
• The fail-closed behaviour of bin/devenv is unchanged, and no assertion was relaxed.

Known limitations

• This does not fix the flakes. It makes the next occurrence in CI identify the failing condition and the devenv error. A fix follows once that evidence exists.