Skip to content

feat: run several flows in one command, with JSON selection and a CI summary - #18

Merged
piekstra merged 3 commits into
mainfrom
piekstra/multi-flow-run
Oct 26, 2026
Merged

piekstra merged 3 commits into
mainfrom
piekstra/multi-flow-run

Conversation

@piekstra

Copy link
Copy Markdown
Member

Summary

lest run ran one flow, so CI had to loop in shell, merge exit codes by hand and write one JUnit file per flow. This lets one command select and run a set of flows and report them in forms CI reads.

  • Selection: lest run a b c, --tag <t> (repeatable) and --affected <ref>, in any mix, deduplicated in that order. Every selected flow and every flow it calls is validated before any runs. A missing flow or a tag no flow has exits 4, a validation error exits 2, and an --affected selection that matches nothing exits 0.
  • Concurrency: one flow at a time by default. --max-parallel <n> runs up to n at once; then each run prints only its result line and sign-ins never wait for input. Ctrl-C cancels running flows and queued ones never start.
  • Exit code: the worst result wins: cancelled (130) over could not run (3) over failed (1) over passed (0).
  • Inputs: each flow gets the --input values it declares; an input no selected flow declares is a usage error.
  • Outputs: with a selection, -o is a directory of <flow>.json reports and --junit one file with a <testsuite> per flow in <testsuites>. Naming exactly one flow keeps the existing single-file behavior. A summary table (flow, result, duration, run, failed step, why) closes a multi-flow run.
  • GitHub step summary: when GITHUB_STEP_SUMMARY is set, the run appends a Markdown table built from the (already redacted) reports, with cells flattened to one line, pipes escaped and capped at 300 characters.
  • JSON: lest list --json and lest affected --json print one line, an array of objects with id (plus name, path, tags or reasons), for fromJSON in a matrix.
  • JUnit fix: a run that errored before its steps now counts its preflight test case in the suite's tests and errors attributes.

docs/ci.md now shows a single job using --affected and a select-then-matrix pair; README, flow-format.md, development.md and the agent guide mention the new forms.

Tests

crates/lest/tests/cli.rs drives the built binary against a temporary project: several flows with per-flow reports, multi-suite JUnit and the step summary (including an escaped pipe); exit precedence; --tag with --max-parallel 2; selection mistakes failing before anything runs; per-flow inputs; single-flow outputs unchanged; list --json; and affected --json plus run --affected in a git repository. A unit test covers exit precedence.

make check passes. Two timing-based runner tests in lest-core (parallel_group_runs_concurrently and one other, different each time) failed while the machine's load average was above 9; they pass with --test-threads=1 and this change does not touch lest-core's runner.

…summary

lest run takes flow ids, --tag and --affected together, runs the selection one at a time or up to --max-parallel at once, and exits with the worst result (cancelled over could not run over failed). With a selection, -o is a directory of per-flow reports and --junit one file with a test suite per flow. lest list and lest affected print --json for a CI matrix, and a run appends a Markdown table to GITHUB_STEP_SUMMARY when it is set.
@piekstra

Copy link
Copy Markdown
Member Author

Review of the diff, with dispositions:

  • Matrix example passed ${{ matrix.flow.id }} straight into run:. Fixed in b9c651d: the id goes through env and the script reads $FLOW, the usual guard against expression injection in workflow scripts.
  • Preflight test case not counted in the JUnit suite attributes. Fixed in this PR: tests and errors now include it.
  • lest run a a dedups to one flow but uses the directory form of -o. No change: the rule is "exactly one named flow keeps single-file outputs", which is predictable from the command line rather than from what a selection happens to match. Documented in docs/ci.md.
  • --max-parallel above 1 turns off interactive sign-in even when only one flow is selected. No change: concurrent runs cannot share the terminal, and the switch depends on the flag, not the selection, for the same predictability reason.
  • Exit precedence puts could not run (3) above failed (1). Kept: a run that could not start says nothing about the system under test, so it should not be masked by a failure elsewhere in the batch. Documented in ci.md, flow-format.md and development.md.
  • Usage errors (missing flow, unknown tag) do not write the GitHub step summary. No change: they fail before any run with a message on stderr, which the job log shows.

Test note, correcting the body: make check stopped at lest-core's timing-based runner tests (parallel_group_runs_concurrently, services_start_before_steps_and_stop_after, cancel_stops_steps_but_runs_finally_and_cleanup, a different pair each run) while the load average was above 9. All 22 pass with --test-threads=1; lint, the CLI tests, the UI tests and the release build pass. This change does not touch lest-core's runner.

@piekstra
piekstra marked this pull request as ready for review October 26, 2026 17:39
@piekstra
piekstra merged commit 4f87da2 into main Oct 26, 2026
5 checks passed
@piekstra
piekstra deleted the piekstra/multi-flow-run branch October 26, 2026 18:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant