Skip to content

Speed up CI tests and the docs build; unify test conventions - #266

Merged
PatrickRitchie merged 4 commits into
masterfrom
ci/test-and-docs-speedup
Nov 14, 2026
Merged

PatrickRitchie merged 4 commits into
masterfrom
ci/test-and-docs-speedup

Conversation

@PatrickRitchie

Copy link
Copy Markdown
Contributor

CI on a PR took 40–70 minutes, and the docs build plus its route check about 30. Almost all of that was overhead rather than tests: the unit tests themselves run in under a minute.

What changed

CI test time (ci: cut test wall-clock…)

  • Coverlet's branch analysis on two generated files in JSON-cppagent (JsonEvents.g.cs / JsonSamples.g.cs, ~5,000-line constructors) took ~9.5 minutes per run. They're excluded from coverage in both runsettings files; the full unit suite with coverage now runs in under a minute locally.
  • Coverage is collected on the Ubuntu leg only; Windows runs the tests without it.
  • Each test project writes its own TRX file (LogFilePrefix) instead of overwriting one shared file, and a shared tools/ci/check-trx.sh guard fails the job on a missing, unreadable or empty result.
  • Workflow concurrency cancels superseded PR runs; NuGet packages are cached.
  • The E2E route-check shards only build the Docs-Tests project instead of the whole solution.
  • docs.yml: PR runs get their own concurrency group; the Pages group now applies only to the deploy job.

Test conventions (test: unify test project conventions)

  • New tests/Directory.Build.props: one version of Test.Sdk, NUnit, the NUnit adapter and coverlet for every test project (there were three Test.Sdk versions, two NUnit and three adapter versions), and a default RootNamespace of MTConnect.Tests.<Area>.
  • Test namespaces normalized to MTConnect.Tests.<Area> (SHDR-Tests no longer uses the product namespace MTConnect.Shdr); the Compliance embedded-schema resource prefix follows.
  • SHDR's 27 if … Assert.Pass(); else Assert.Fail(); blocks are now Assert.That(actual, Is.EqualTo(expected)), so failures show both values.
  • A "Writing tests" section in CONTRIBUTING.md documents the conventions.

Compliance (test(compliance): pin the cppagent parity image…)

  • The cppagent parity tests ran mtconnect/agent:latest, which now resolves to 2.8.0.4 and fails the fixture's 2.7.x version check. Pinned to the 2.7.0.7 digest.

Docs site (docs: cut the site build and route check…)

  • Every API page rendered the whole ~2,500-entry API sidebar and inlined the site config, so the build wrote 3.2 GB of HTML. The API sidebar is now one per namespace and the site data is a shared chunk (metaChunk). Local VitePress build: 267 s → 117 s; HTML 3.28 GB → 0.18 GB.
  • The route check walks every narrative page and a sample of the API pages (overview, each namespace page, one type per namespace): 390 of 2,445 routes, ~33 s. ROUTE_CHECK_ALL_API=true walks all of them.
  • The route check no longer leaks a vitepress preview server on Windows, and its rebuild check no longer fails when dist/ already exists.
  • Adds the XML doc comments missing from two durable-buffer tests. The docs build compiles every project with warnings as errors, so these broke npm run build.

How to verify

  • CI: the Ubuntu leg should finish in minutes and still publish coverage; the Windows leg runs without coverage.
  • Locally: ./tools/test.sh, or dotnet test tests/MTConnect.NET-Docs-Tests --filter Category=E2E after npm run build in docs/ (needs docfx).
  • Verified locally: the solution builds with 0 errors; 5,929 unit tests (14 projects, CI filter) and 13 integration tests pass; Compliance passes 247/247 including the Docker parity tests; Docs-Tests passes including the E2E route check.

Known failures not addressed here

These also fail on master and are left for a separate fix:

  • ShdrNetworkTests (Adapter_AnswersPing, QueueAdapter_SendsEveryQueuedValueInOrder_AndAssetsToo) are flaky: they can connect before the adapter's listener is bound.
  • DeviceFilesTests.NewFile_IsCreatedFromTheTemplate fails on Linux because DeviceFiles.cs mixes a case-insensitive lookup with a case-sensitive File.Exists.
  • The format job flags AgentClientConnectionListener.cs and DeviceDocumentTests.cs.

🤖 Generated with Claude Code

PatrickRitchie and others added 4 commits November 14, 2026 10:22
The tests themselves run in under a minute; CI time was coverlet
instrumentation. Two generated members in MTConnect.NET-JSON-cppagent
(Streams/JsonEvents.g.cs and Streams/JsonSamples.g.cs, ~5,000-line
constructors plus getters with hundreds of branches) took ~9.5 minutes
to instrument, repeated in every test project that loads the assembly
(JSON-cppagent, Common, Compliance, Integration).

- Exclude those two files from coverage (both runsettings). The
  solution-wide unit run with coverage drops to ~44 s locally; the
  JSON-cppagent project alone from 9m34s to 7s.
- Collect coverage on ubuntu only; the Windows leg runs tests plain.
- Give every test project its own TRX (LogFilePrefix) instead of a
  shared LogFileName that each project overwrote, and replace the three
  copied zero-test guards with tools/ci/check-trx.sh, which checks
  every TRX.
- Add a per-PR concurrency group that cancels superseded PR runs, a
  NuGet package cache, and a 45 min job timeout.
- E2E route-check shards build only Docs-Tests (not the whole
  solution) and skip coverage that was never merged into the report.
- docs.yml: scope the `pages` concurrency group to the deploy job so PR
  docs builds no longer queue behind each other.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- Add tests/Directory.Build.props. It imports the root props and adds
  Microsoft.NET.Test.Sdk 17.14.1, NUnit 3.14.0, NUnit3TestAdapter 4.6.0
  and coverlet.collector 6.0.4 to every *-Tests project (NUnit packages
  skipped for the one xunit project), replacing three different
  Test.Sdk versions, two NUnit versions and three adapter versions
  pinned per project. It also defaults RootNamespace to
  MTConnect.Tests.<Area>.
- Normalize namespaces to MTConnect.Tests.<Area>[.<Folder>]:
  MTConnect.NET_{Common,Docs,JSON,JSON_cppagent}_Tests,
  MTConnect.AgentModule.<X>.Tests, MTConnect.Compliance.Tests and
  MTConnect.Tests.Xml are renamed; SHDR-Tests no longer uses the
  product namespace MTConnect.Shdr as its root. The Compliance XSD
  embedded-resource prefix follows the new root namespace.
- SHDR-Tests/DataItems.cs: replace 27 `if (a == b) Assert.Pass(); else
  Assert.Fail()` blocks with Assert.That(actual, Is.EqualTo(expected)),
  fixing the swapped input/output naming so failure messages label the
  expected and actual values correctly.
- CONTRIBUTING.md: document the conventions for projects, namespaces,
  files, test names, assertions, async waits and network ports.

All 5,953 unit tests and 13 integration tests pass with the same counts
as before.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CppAgentParityWorkflowTests ran mtconnect/agent:latest, which now
resolves to cppagent 2.8.0.4; the fixture's own version probe expects
2.7.x, so all three parity tests failed in OneTimeSetUp. Pin the image
to the 2.7.0.7 digest the fixture comment already recorded, and update
the two other places that named :latest.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every API page rendered the whole ~2,500-entry API sidebar, and every
page inlined the full site config, so the build wrote 3.2 GB of HTML
(~1.4 MB per API page) and needed an 8 GB Node heap.

- sidebar.ts: one sidebar per namespace (parents, child namespaces, own
  types) keyed `/api/<Namespace>.`, plus an `/api/` index of all
  namespaces, instead of one sidebar holding every type.
- config.ts: metaChunk so the site data is one shared chunk instead of
  ~0.4 MB inlined into every page.
  VitePress build locally: 267 s -> 117 s; HTML 3.28 GB -> 0.18 GB.

Route check (MTConnect.NET-Docs-Tests, Category=E2E):
- Walk every narrative page but only a sample of the docfx API pages
  (overview, each namespace page, one type per namespace): 390 of 2,445
  routes, ~33 s. ROUTE_CHECK_ALL_API=true walks all of them.
- Start `vitepress preview` with node directly. Through npx.cmd the
  server outlived its parents on Windows, Kill(entireProcessTree) missed
  it, and every run leaked a preview server.
- The dist-rebuild check is inconclusive when the fixture reused an
  existing dist/ instead of failing every run that had one.
- Note docfx as a local prerequisite.

Also add the XML doc comments the durable-buffer tests were missing:
the docs build compiles every project with GenerateDocumentationFile
and warnings as errors, so CS1591 there broke `npm run build`.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@PatrickRitchie
PatrickRitchie merged commit 9ddd71d into master Nov 14, 2026
7 of 13 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant