[knittingcode]

Summary

protect_for in bin/devenv stops the edge before any database change (role corrections, migrations, upgrade, password rotation). After the stop it now confirms that the edge stopped from the containers themselves (docker container inspect), instead of from a new docker compose ps listing. Its error now names the exit status and last error line of docker compose stop. A failing docker compose ps is no longer taken to mean "no edge".

Motivation and context

CI run 40836555626 (PR #37, head cc2e11d, attempt 1) failed ST-50 "a verified rotation whose edge cannot get ready" with:

rotation exit 6, edge container [], GET / 000, make up exit 0
devenv: ERROR: the edge of flowcrm-9f1eb1 could not be stopped, so the database was not changed; stop the stack with make down

The failure stayed safe: the database was not changed. But it was the wrong outcome. The only source of that message is protect_for. It ran compose_op stop edge >/dev/null 2>&1 without checking its exit status, then failed if compose ps --quiet edge still listed the edge. Yet the test's docker ps, taken right afterwards, found no running edge. Nothing in bin/devenv stops the edge after die, and the edge has no restart policy. So the stop had taken effect. The whole rotation, make status, the api wait and a full make up took 9.35 s, so the stop did not run into its 10 s grace period.

The most likely cause is in the Docker daemon. When a container exits, the daemon wakes the waiting stop call first, so docker compose stop returns. Only afterwards does it update the container list that docker ps and docker compose ps read. A listing taken in that window can still show the container as running. docker container inspect reads the container itself.

This was inferred from the daemon's code, by an independent read-only review, and not reproduced. In 200 recent CI runs the message appears only in this run. The PR's own change (go-chi/chi 5.3.2 → 5.4.0 in the edge's Caddy build) was assessed separately as not plausibly involved:

• chi is linked into the edge only through the ACME server module, which is not configured;
• chi has no init, signal or shutdown code;
• the bump changes no other module.

Implementation

• protect_for: it lists the edge containers once and fails (exit 6) if that listing fails. This happens before the gate is recorded and before the database changes. After docker compose stop it checks those IDs with edges_stopped. stop's stderr goes to a temporary file (new_tmp); its last line, stripped of control characters and capped at 200 characters, is added to the error together with the exit status.
• edges_stopped (new):
  • "not running" (false) or "No such container/object" passes.
  • Any other result fails closed: still running, paused or restarting, or any other inspect error.

The gate is still recorded before the stop, the call still runs in the parent shell (signal handling unchanged), and the success message is unchanged.

Testing performed

• shellcheck -x bin/devenv tests/devenv/unit.sh: clean, the same as on main.
• tests/devenv/unit.sh, run locally inside the Docker isolation from test(ci): isolate the unit and guard suites from Docker #43 (scripts/ci/no-docker.sh, on a local, unpushed combination of this branch with test(ci): isolate the unit and guard suites from Docker #43):
  • 341/341 passed.
  • The same suite against main's unchanged bin/devenv: 5 of the new U-23 checks fail, including the stale-listing case, which fails exactly as in CI (exit 6, "could not be stopped"). The removed-container case passes on both and serves as a control.
  • docker events during the runs: no events from any FlowCRM container or project.
• New U-23 cases:
  • a stale listing after a successful stop;
  • a container removed after the stop;
  • a failed stop, with its exit status and error in the message;
  • a stop that returns 0 while the edge still runs;
  • an inspect error;
  • a failing listing (nothing is recorded and stop is not called).
• The application stack suites (tests/app/stack.sh): run only in CI.

Security considerations

The property is unchanged: no database change may start while an edge listed before the stop could still serve.

• An unknown or failed inspect counts as running.
• A failed listing now fails closed. Before, it skipped the stop.

The error message carries only Docker's error text, sanitised and truncated. It contains no secrets.

Known limitations and follow-ups

• Only the IDs listed before the stop are confirmed. An edge started in between by something outside the devenv lock, such as a manual docker compose up, is not detected. The old second listing would have caught one; this is outside the lock's threat model.
• Different daemons: the IDs come from Docker Compose and are checked with plain docker, as edge_state already does. If the two ever reached different daemons, "No such container" would pass.
• Other functions: gate_failed and cancel_db_op still treat a failed compose ps as "no edge". That is a separate follow-up.
• The race explanation: it was inferred from the daemon's code and has not been reproduced.