Skip to content

CI: a change to one tool or test project runs the test projects it reaches, not the whole solution #268

Closed

Description

@CaffeinatedCoder

Why

CI picks its lanes by path class (tools/CiLanes, decision 113), and each lane builds and tests the whole solution. A change to one tool or one test project therefore runs every test project on Linux and Windows.

#266 changed tools/BashGuard/Program.cs and its tests, plus docs and one line of .gitignore. ci-lanes classify put the files in these classes:

Files Class Lanes
AGENTS.md, CONTRIBUTING.md, design/** prose Linux
tools/BashGuard/Program.cs, test/CodoMetis.TypeKit.BashGuard.Tests/GuardTests.cs code no package reads Linux, Windows
.gitignore unlisted Linux, Windows, Pack

Its first run (41215127177) took 12m55s on Linux and 13m32s on Windows, and it packed and smoke-tested too. The tests that could notice the change were CodoMetis.TypeKit.BashGuard.Tests (112 tests, 5 s locally) and Conventions. A docs-only PR like #267 runs test/Prose.slnf alone, in 1m49s.

Proposal

  • The test suite follows the projects a change reaches, not only its lane. When every changed path that isn't prose is under tools/<Name>/** or test/<Project>/**, CiLanes outputs a suite of the changed test projects, every test project that references a changed project (transitively, through ProjectReference), Conventions, and the test/Prose.slnf projects when prose changed too. The legs build and test that suite the way they build test/Prose.slnf today. Any other path (src/**, tools/Shared/**, tools/Directory.Build.*, the build files, an unlisted path) keeps the whole solution.
  • A test that reads a project by path, not by reference, counts as reaching it. Conventions reads the repository by globbing, and some tests may run a tool's DLL or read its files. Measure which tests do that first. Either declare those dependencies so CiLanes reads them, or keep that tool on the whole solution. A dependency left undeclared means a test that silently stops running.
  • .gitignore gets a class. Find which tests read it (git listings, the packability rules) and send it to the lanes those tests run in. Pack is not one of them, since dotnet pack doesn't read .gitignore. Check the other root dotfiles at the same time: .gitattributes sets line endings, so it stays on Windows.
  • The Windows leg keeps its reason. "code no package reads" goes to Windows because of paths, line endings and child processes. A narrower suite on Windows still includes Conventions and the changed tool's tests.
  • A decision record amending 0113's selection, numbered with the next free number when it is merged (0125 and 0126 are taken, CI: split the Linux test step across two jobs, since it is bound by the runner's four cores #253's text assumes 0127 and 0128 for Release: publish the packages CI packed instead of building them again, and check the tag before CI #247 and CI: cancel a superseded release-branch push run, and select the next push's lanes from the last green one #248, so check against release/2.4.0 before merging).

Done when

  • CiLanes.Tests have rows for a tool-only change, a test-project-only change, a change to tools/Shared, a mix with src/, and .gitignore. Each is shown failing first against the current selection (verify-the-guard).
  • A conventions test fails when a test project reads a tool by path without declaring it, shown by seeding one.
  • A PR shaped like build: run parallel work in worktrunk worktrees beside the repository #266 (one tool, its tests, docs) runs a narrowed suite on both legs, with no Pack. Its run time is measured in this issue against 41215127177.
  • The gate still fails a run whose selected suite was skipped or cancelled.

Activity

  1. added this to the 2.4.0 milestone on Oct 31, 2026
  2. added
    choreBuild, CI, tooling, packaging, refactoring with no behaviour change
    area/toolingThe repository's tools: prober, verdict, guard, docs-query, smoke test
    on Oct 31, 2026
  3. CaffeinatedCoder commented on Oct 31, 2026

    @CaffeinatedCoder
    OwnerAuthor

    Implemented in #274 (decision 0127).

    This PR's own run, 41236579959, cannot show the narrowing: it changes tools/CiLanes and .github/workflows/dotnet.yml, so the changes job selected the whole solution on Linux and Windows with Pack and Consumer, exactly as before (Changes 10 s). Its timings go here when it completes, against 41215127177: Changes 27 s, Test (ubuntu-latest) 12m55s, Test (windows-latest) 13m32s, Pack 7m26s, Consumer 6m10s, wall clock 14m15s.

    What a #266-shaped PR selects with #274, measured locally with ci-lanes suite over #266's eleven paths (af3f95e^1..af3f95e):

    Before (0113) After (0127)
    Linux, Windows, Pack and Consumer (.gitignore was unlisted) Linux and Windows, no Pack
    whole solution, 33 projects test/Reach.slnf: the 5 projects of test/Prose.slnf, tools/BashGuard, CodoMetis.TypeKit.BashGuard.Tests

    test/Reach.slnf written as the leg writes it builds locally (17 projects with the shipping projects they reference). The first CI measurement of a narrowed run needs the first tool-or-test PR into release/2.4.0 after #274 merges; that run's leg times belong here too.

    Measured while implementing: the only test projects that read another project by path rather than by reference are the conventions (the whole repository) and the oracle tests (every test project's sources, for each claim's test); both now declare it with <ReadsByPath>. No test reads .gitignore: the conventions list the index (git ls-files), and dotnet pack does not read it.

  4. CaffeinatedCoder commented on Oct 31, 2026

    @CaffeinatedCoder
    OwnerAuthor

    Landed on release/2.4.0 by #274 (merge commit 500231d), with decision 127. The narrowed suite is measured on the first PR shaped like #266 after this merge.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area/toolingThe repository's tools: prober, verdict, guard, docs-query, smoke testchoreBuild, CI, tooling, packaging, refactoring with no behaviour change

    Projects

    No projects

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions