Skip to content

Speed up CI - #32

Merged
JieXiong9119 merged 1 commit into
developfrom
speed-up-ci
Nov 18, 2026
Merged

JieXiong9119 merged 1 commit into
developfrom
speed-up-ci

Conversation

@JieXiong9119

@JieXiong9119 JieXiong9119 commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Speed up CI: split unit and integration tests, run simulations in parallel shards, add caching

Closes #12

Why

The single Tests job ran every spec in one job, including about 20 full-year EnergyPlus simulations one after another. Each run took roughly 1h15m–1h30m, even for PRs that don't touch simulation code.

What changed

Workflow (.github/workflows/ci.yml)

  • Replaced the single Tests job with three jobs:

    • unit runs spec/tests/unit on every PR and push. It doesn't install external measures.
    • changes checks whether a PR changes anything besides documentation (*.md, docs/, LICENSE*).
    • integration (shard N) runs the simulation specs as 4 parallel shards with fail-fast: false.
  • When integration runs:

    Event Integration
    PR with code changes Runs (default)
    PR with documentation-only changes Skipped
    PR labeled skip-integration Skipped
    Push to develop or a tag Runs
    Nightly schedule (07:00 UTC) Runs
    Manual workflow_dispatch Runs
  • PRs also trigger on labeled / unlabeled, so adding or removing skip-integration takes effect right away.

  • Added concurrency with cancel-in-progress so a new push cancels the outdated run.

  • Caching:

    • .bundle/install, keyed on Gemfile.lock and the gemspec.
    • vendor/external_measures, keyed on config/external_measure_repos.yml.
  • Added --profile 10 so the slowest tests show up in CI logs.

  • When an integration shard fails, its spec/output is uploaded as an artifact.

Integration spec (spec/tests/integration/write_and_run_osws_spec.rb)

  • BOSS_SHARDS / BOSS_SHARD select the fixtures where index % BOSS_SHARDS == BOSS_SHARD. Out-of-range values raise an error.
  • BOSS_SMOKE=1 runs only building_151.xml and example-smalloffice-level1.xml, for quick local checks.
  • With neither variable set, every fixture runs as before. No fixtures or assertions were changed.

README

  • The Tests section now covers installing external measures and running the unit tests, the smoke subset, a single shard, and the full suite. It also explains when CI runs integration.

Verification

  • An rspec --dry-run for each shard selected 5, 5, 5 and 5 fixtures: 20 in total, no duplicates, matching the full list.
  • BOSS_SMOKE=1 --dry-run selected 2 fixtures.
  • Not yet verified on GitHub Actions: real shard run times, cache hits, cancelling outdated runs, and skipping doc-only PRs.

Follow-ups for maintainers

  • Create the skip-integration label.
  • Update branch protection so the required checks are unit and the integration (shard N) jobs instead of Tests. Skipped jobs count as passing.
  • Decide how nightly failures should notify the team (default GitHub email, an automatically opened issue, or Slack/Teams).
  • After the first full run, check shard durations with --profile and rebalance or move to 5 shards if one shard is much slower than the others.

Notes

  • comstock stays on main, as before. The cache only provides a starting checkout, and rake measures:install_external always fetches the latest main. Upstream changes can still affect CI results.
  • With integration running on PRs by default, PR feedback takes as long as the slowest shard, not the issue's 10-minute target. Use skip-integration for PRs that clearly don't touch simulations.
  • Out of scope: the known BuildingEQ-1.0.0_gemtest.xml sizing failure, and any change to what the tests assert or how simulations behave.

doc-only PRs skip integration
skip-integration label skips integration
@JieXiong9119
JieXiong9119 requested a review from kflemin November 17, 2026 21:05
@JieXiong9119 JieXiong9119 self-assigned this Nov 17, 2026
@JieXiong9119
JieXiong9119 merged commit e0bb39b into develop Nov 18, 2026
6 checks passed
JieXiong9119 added a commit that referenced this pull request Nov 18, 2026
commit b45ec37
Merge: 0496378 e0bb39b
Author: JieXiong9119 <69597597+JieXiong9119@users.noreply.github.com>
Date:   Tue Nov 17 17:39:18 2026 -0700

    Merge branch 'develop' into scenario-workflow

commit e0bb39b
Merge: 1929a26 65bb0a9
Author: Jie Xiong <jiexiong9119@gmail.com>
Date:   Tue Nov 17 17:30:42 2026 -0700

    Merge pull request #32 from BuildingSync/speed-up-ci

    Speed up CI

commit 65bb0a9
Author: JieXiong9119 <69597597+JieXiong9119@users.noreply.github.com>
Date:   Tue Nov 17 14:01:59 2026 -0700

    split jobs into unit and integration, split integration into 4 shards

    doc-only PRs skip integration
    skip-integration label skips integration

commit 1929a26
Merge: 871317d 711e45f
Author: Katherine Fleming <2205659+kflemin@users.noreply.github.com>
Date:   Thu Nov 12 10:43:31 2026 -0700

    Merge pull request #22 from BuildingSync/fix/floor-area-building-first

    Fix/floor area building first

commit 711e45f
Author: Katherine Fleming <2205659+kflemin@users.noreply.github.com>
Date:   Wed Nov 11 12:29:18 2026 -0700

    fix test

commit bcf2317
Author: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Date:   Tue Nov 10 22:21:50 2026 +0000

    Handle missing Building in floor area fallback

    Co-authored-by: kflemin <2205659+kflemin@users.noreply.github.com>

commit 35abcbf
Author: Katherine Fleming <2205659+kflemin@users.noreply.github.com>
Date:   Tue Nov 10 12:57:19 2026 -0700

    Require logger and constants in BuildingSyncReader

    The reader calls BOSS.BOSS_logger and uses ASHRAE90_1 and other
    constants, but relied on other files to load them. Running
    buildingsync_reader_spec.rb on its own failed with "undefined method
    `BOSS_logger' for BOSS:Module". Require both directly.

    Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

commit 43fc2da
Author: Katherine Fleming <2205659+kflemin@users.noreply.github.com>
Date:   Tue Nov 10 12:53:24 2026 -0700

    Prefer Building floor area over Site

    get_total_floor_area checked Site FloorAreas before Building, so a
    Site value overrode the Building's own value. Check the more specific
    Building first and fall back to Site, matching the method comment.

    Add specs for Building only, Site only, and both present.

    Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Speed up CI: split unit and integration tests, shard simulations, add caching

2 participants