Skip to content

Design note: Cloud Pathways patterns for the run controller - #896

Open
anchapin wants to merge 1 commit into
developfrom
docs/cloud-pathways-patterns
Open

anchapin wants to merge 1 commit into
developfrom
docs/cloud-pathways-patterns

Conversation

@anchapin

@anchapin anchapin commented Nov 10, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Draft design note only — no implementation, no code changes.

Google Cloud Pathways went GA on 9 Nov 2026. This note evaluates it for OpenStudio Server and concludes:

  • Do not adopt Cloud Pathways itself — it is a JAX runtime for Google Cloud TPUs on GKE, and OpenStudio / EnergyPlus workloads are CPU-based, embarrassingly parallel simulations.
  • Do adopt three of its patterns, with this repository owning the first and defining the contracts for the other two:
    1. Single-controller orchestration — one run controller owns an analysis end-to-end and is the sole writer of run state, with a durable manifest so a restarted controller resumes.
    2. Elastic N-1 fault tolerance — a datapoint completion contract (done only when outputs + completion marker are committed to object storage); the controller re-queues only missing/incomplete datapoints.
    3. Colocated sidecars — the staging contract the sidecar implements: fetch + hash-verify inputs, upload results/logs and write the completion marker even if the simulation crashed.

Adds doc/cloud_pathways_patterns_design.md.

Companion PR

The deployment-side half of this design (Helm chart Job template, sidecar containers, disruption/backoff settings) is in NatLabRockies/openstudio-server-helm#163: NatLabRockies/openstudio-server-helm#163

The two documents cross-reference each other by repo and path.

Test plan

None — documentation only. The note itself proposes a three-check gate (kill a worker node mid-run, restart the controller mid-run, wall-clock/cost comparison) for any future implementation PR.

Please do not merge on my behalf — opening for review/discussion only.

Also in this series: the OSimFlow mapping/gap note — anchapin/OSimFlow#2133: anchapin/OSimFlow#2133

Draft design note (no implementation): do not adopt Cloud Pathways
(TPU/JAX/GKE runtime), adopt three of its patterns - single-controller
orchestration with a durable run manifest, a datapoint completion
contract for N-1 fault tolerance, and staging contracts for colocated
sidecars. Companion deployment-side doc: NatLabRockies/
openstudio-server-helm PR #163.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant