Skip to content

Automated Sync from main to stable - #40

Open
rhods-devops-app[bot] wants to merge 622 commits into
stablefrom
main
Open

rhods-devops-app[bot] wants to merge 622 commits into
stablefrom
main

Conversation

@rhods-devops-app

@rhods-devops-app rhods-devops-app Bot commented Oct 24, 2026 •

Copy link
Copy Markdown

Automated Sync from main to stable

This PR automatically syncs the main branch to the stable branch by opening a pull request from main into stable.

Sync Summary

  • Latest commit: 80648b2b6 Merge remote-tracking branch 'upstream/main'

  • Total commits to sync: 610

  • Source: https://github.com/red-hat-data-services/OpenShell.git @ main

  • Target: https://github.com/red-hat-data-services/OpenShell.git @ stable

  • PR head: main

Commits to be synced

Merging

GitHub automerge is enabled for this pull request once required checks pass.

Comment thread .github/workflows/codex-security.yml Dismissed
Comment thread .github/workflows/release.yml Fixed
Comment thread .github/workflows/snap-publish.yml Dismissed
@rhods-devops-app
rhods-devops-app Bot enabled auto-merge October 24, 2026 01:13
@coderabbitai

coderabbitai Bot commented Oct 24, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

Run configuration

  • Source mode: Auto
  • Repository: Unavailable
  • Branch: Unavailable
  • File: No configuration file from this repository applied
  • Commit: Unavailable
  • Layers: Organization UI
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 5fb1fdbf-30e9-46bc-bbb4-baaad6efdc0c

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review

Comment thread .github/workflows/release-auto-tag.yml Dismissed
Comment thread .github/workflows/release-prerelease.yml Dismissed
matthewgrossman and others added 11 commits November 2, 2026 17:42
* fix(ci): stop skipped release notes from skipping publication

Stable releases that reused the prerelease notes preview skipped the
Write Release Notes job. Every job that depends on Publish Release Assets
used an implicit success() check, which treats a skipped job anywhere
upstream as a failure. As a result, v0.1.6 never published its snap,
TypeScript SDK, Go SDK tag, wheels, or Fern docs. Prereleases hit the same
problem: Publish Prerelease Snap was always skipped.

Move notes production out of release-publish.yml. release-notes.yml now
chooses between reusing the preview and generating notes inside its steps,
so the job always succeeds when notes exist. Release runs it as a sibling
of Build, and Publish requires it. Publish Release Assets depends only on
assembly and the Helm chart again.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* fix(ci): make stable release publication rerunnable

Re-dispatching Release for an existing stable tag rebuilt the artifacts,
overwrote the published GHCR version tags, and then failed at the Helm
duplicate check before reaching the package publishers. Each publisher now
skips what is already published:

- GHCR keeps existing version tags and still verifies their SBOMs.
- Helm skips packaging and pushing when both charts exist, and still fails
  when only one exists. The Helm action is loaded from the workflow
  revision, so retrying an older tag uses the current duplicate handling.
- The GitHub Release uploads only missing assets.
- npm and the PyPI wheel trigger skip versions that already exist, whether
  called by Release or dispatched by hand.

test:release-workflows checks that publish logic is loaded from the
workflow revision and that no needed publish job can be skipped on its own.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* fix(ci): fail release runs when an expected publisher does not succeed

GitHub treats skipped jobs as passing, so v0.1.6 showed a green Release
run although its publishers never ran. A final Check Publication job in
release-publish.yml runs always and fails when any publisher expected for
the release kind did not succeed, which also fires the existing Pre-release
Slack alert. Release now sends the same Slack failure notice.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

* fix(ci): fail instead of overwriting when a registry lookup errors

The GHCR and Helm existence checks treated any lookup error as not
published, so a transient registry error could fall through to a push that
overwrites a published tag. Only a confirmed missing tag now counts as
unpublished; any other lookup error fails the job.

Also state in CI.md that published artifacts are never overwritten, and
remove the static release workflow test; check-publication catches skipped
publishers at runtime.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

---------

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
…#4918)

* feat(sdk): route Python SDK exec to the sandbox owner replica

Ask GetSandbox for the owner replica, cache it for 30 seconds, and send
it as x-openshell-replica on ExecSandbox. Retry once without the hint
when the per-replica route has no healthy endpoint, forgetting the
cached owner.

Part of NVIDIA#4223

Signed-off-by: divesh <dgude@nvidia.com>

* refactor(sdk): share the Python GetSandbox request for owner lookup

Reuse get's request for the owner lookup, point the routing headers at
their documented contract, and accept only string owner metadata.

Part of NVIDIA#4223

Signed-off-by: divesh <dgude@nvidia.com>

---------

Signed-off-by: divesh <dgude@nvidia.com>
…IA#5131)

Signed-off-by: Artem Lytvyn <letv1n2007@icloud.com>
…VIDIA#5082)

Every OTLP export (gateway traces, OCSF logs, supervisor traces, and the
agent trace relay) spoke gRPC only, so trace backends that accept only
OTLP/HTTP needed a translating collector in between.

Each [openshell.gateway.otlp] table accepts protocol = "grpc" or
"http/protobuf". A table without its own endpoint uses the whole shared
collector; a table with its own endpoint inherits nothing from it. Compute
drivers take the collectors through one shared set of flags, read from
the same environment variables that supervisors receive, and pass them on
through one SupervisorOtlp value that replaces the per-lane driver fields.
OTLP/HTTP goes through a single client in openshell-otel, shared by the
SDK exporters and the relay, that bounds each request by the export
timeout and fails exports whose response is incomplete or larger than
64 KiB. Exporters send no query parameters, so an endpoint with a query
string is a configuration error.

Refs NVIDIA#5067

Signed-off-by: Kris Hicks <khicks@nvidia.com>
… replica (NVIDIA#4877)

* feat(sdk): route TypeScript SDK exec and forward to the sandbox owner replica

Ask GetSandbox for the owner replica and send it as x-openshell-replica
on ExecSandbox, ExecSandboxInteractive and ForwardTcp. Retry once
without the hint when the per-replica route has no healthy endpoint,
holding stream input until response headers arrive so the retry only
resends the first frame.

Part of NVIDIA#4223

Signed-off-by: divesh <dgude@nvidia.com>

* refactor(sdk): share the TypeScript GetSandbox call for owner lookup

Reuse get's request and error mapping for the owner lookup, and point the
routing headers at their documented contract.

Part of NVIDIA#4223

Signed-off-by: divesh <dgude@nvidia.com>

---------

Signed-off-by: divesh <dgude@nvidia.com>
The agent launcher applied manifest settings with `settings set --global`,
so every gator launch switched proposal_approval_mode to auto for the whole
gateway. Apply them to the launched sandbox after creation instead. If a
gateway-global value already manages a key, warn and leave it in effect.

Related to NVIDIA#3469.

Signed-off-by: Jim Meyer <jimeyer@nvidia.com>
…VIDIA#4936)

* feat(supervisor): add independent application listener and endpoints

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>

* test(supervisor): automate Kubernetes application listener e2e

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>

* fix(supervisor): enforce app route limits and shutdown fencing

Reject excess service declarations before persistence, fence registration against shutdown, and require the expected application Service count in listener E2E checks. Add focused regression coverage.

Signed-off-by: divesh <dgude@nvidia.com>
Signed-off-by: Taylor Mutch <taylormutch@gmail.com>

* test(supervisor): cover application streams on config rejection

Verify rejected application configuration closes existing WebSockets without affecting another sandbox, and accepted recovery cannot revive old streams. Document raw SDK message migration and shared-owner selection before discovery.

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>

* fix(app-listener): fix standalone lint and capture CNI setup failures

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>

* fix(app-listener): preserve polling freshness and HTTP keepalive

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>

---------

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
Signed-off-by: divesh <dgude@nvidia.com>
Co-authored-by: divesh <dgude@nvidia.com>
…VIDIA#5083)

Collectors that require authentication or a routing header could only get
one through OTEL_EXPORTER_OTLP_*HEADERS in the gateway's environment, which
reached the gateway's own exporters and supervisors' OCSF records but not
supervisor spans or the agent relay, so a backend such as MLflow needed a
sidecar collector to add its experiment header.

Each [openshell.gateway.otlp] table now takes a headers table, or a
headers_file of name=value lines that keeps credentials out of the config
file, inherited with the rest of the collector by tables without their own
endpoint. Exporters send them as gRPC metadata or HTTP headers, with
environment headers replacing configured ones of the same name. Supervisors
receive each lane's headers in their launch bundle and the VM driver in its
environment; the agent relay uses only its table's headers. Config parse
errors give a line and column instead of quoting the file, so a malformed
header table cannot echo a token.

Closes NVIDIA#5067

Signed-off-by: Kris Hicks <khicks@nvidia.com>
…s recovery deadline (NVIDIA#5076)

Signed-off-by: divesh <dgude@nvidia.com>
rhods-devops-app Bot and others added 5 commits November 14, 2026 06:09
Update the root Cargo.lock resolution for GHSA-9264-jvgf-6w2f and
GHSA-736f-c93q-9fwx. The patched release verifies the server host key before
calling kex_done and rechecks SSH agent lock state after confirmation.

Drop the generic-array compatibility edge no longer enabled by russh.

Signed-off-by: Adrien Langou <alangou@nvidia.com>
* test(tmachine): run a compute driver as an external service

Add the openshell_external_driver role and a deb-external-docker-driver
installer. The role installs the driver binary and a system service, generates
the gateway certificates the driver trusts, waits for its socket, and points
the packaged gateway at it through the environment gateway overrides.

The packaged gateway role starts it after the runtime images load and before
the gateway, and selects it by name; a built-in driver's name cannot be
selected over a socket.

Signed-off-by: Evan Lezar <elezar@nvidia.com>

* ci: run the Docker external-driver conformance on tmachine

Replace the docker-external-driver-e2e lane, which ran the conformance
tests against a driver-free gateway and a driver started from a shell
wrapper, with a tmachine row that installs the packaged gateway and runs
openshell-driver-docker as a service (deb-external-docker-driver).

The release workflows share e2e-suites.yml and
prepare-integration-inputs.yml but do not build external drivers, so the
row and the driver download are opt-in (external-drivers).

Signed-off-by: Evan Lezar <elezar@nvidia.com>

---------

Signed-off-by: Evan Lezar <elezar@nvidia.com>
…VIDIA#5415)

Add operational health, operational logs, and OCSF dashboards with configurable monitoring mixin exports, mise generation and validation tasks, and path-filtered CI checks.

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
moulalis and others added 20 commits November 16, 2026 16:06
No Kubernetes lane exported traces to a collector it could inspect, so
regressions in gateway or compute-driver span export, trace propagation,
or resource attributes went unnoticed.

Add an e2e:kubernetes:tracing lane. OPENSHELL_E2E_KUBE_JAEGER=1 deploys a
Jaeger collector, points gateway OTLP export at it, and port-forwards its
query API. The kubernetes_tracing test creates a sandbox and fails unless
Jaeger reports the driver's provision span, the gateway's driver client
span in the same trace, and the gateway's CreateSandbox request span, each
without error status and with a service version and Kubernetes resource
attributes. The lane sets the new
OPENSHELL_E2E_KUBE_CONFORMANCE=0 so it does not repeat the conformance
suites other Kubernetes lanes already run.

Closes NVIDIA#3767

Signed-off-by: Kris Hicks <khicks@nvidia.com>
…VIDIA#5506)

* feat(server): make durable mutation admission capacity configurable

Request-ID mutations are capped at 1,000 durable admissions per caller,
and successful admissions hold their slot for the 24-hour replay window.
That turns the storage bound into a throughput ceiling for automation
that creates and deletes many sandboxes with request IDs.

Add two [openshell.gateway] settings that keep today's defaults:
mutation_admission_limit_per_caller (1 through 20000) and
mutation_replay_window_seconds (60 through 604800). Admission, expiry
checks, and expired-success cleanup now read the configured values.

Closes NVIDIA#5341

Signed-off-by: divesh <dgude@nvidia.com>

* refactor(server): trim mutation admission setting comments

Signed-off-by: divesh <dgude@nvidia.com>

* feat(server): keep mutation admission cost flat at large per-caller limits

Admission counted the caller's whole bucket on every request, expired
receipts were only removed once a bucket was full, and that cleanup
loaded the full bucket in one query.

- Track each bucket's admissions in a counter row. Admission reads the
  counter and recounts only when it reaches the limit, so deletions
  never need to decrement it.
- Add a partial index on completed admissions by completion time.
- Delete expired successful admissions in batches of 1,000 every
  minute, and through the same indexed path when a bucket is full.
- Raise the per-caller limit ceiling to 1,000,000.

Refs NVIDIA#5341

Signed-off-by: divesh <dgude@nvidia.com>

* refactor(server): keep mutation admission SQL in the backend modules

Address review feedback on the admission counter and expiry sweeper.

- Move the admission SQL into the Postgres and SQLite stores. Store
  only dispatches, and one shared function decides admission through a
  small per-backend transaction trait.
- Deleting admissions marks the bucket counter for recount. Admission
  recounts only a marked counter at the limit, so a rejection at
  capacity reads one row. Rejections no longer sweep inline; the
  background sweeper owns expiry.
- Log sweeper failures, clamp configured limits to their ranges, share
  the startup range check, and document the literal that the partial
  index predicate depends on.
- Test rejections at capacity and a sweep past a full 1,000-row batch.

Refs NVIDIA#5341

Signed-off-by: divesh <dgude@nvidia.com>

* fix(server): reclaim expired admissions at capacity and recount stale counters

A caller at the limit deletes its own expired admissions through a bucket index before rejecting. A full counter not written for a minute is recounted, so counts left stale by older gateways during a rolling upgrade self-correct.

Signed-off-by: divesh <dgude@nvidia.com>

* fix(server): reject at capacity when the inline admission reclaim fails

A failed reclaim, such as a Postgres deadlock with a concurrent sweep, now returns RESOURCE_EXHAUSTED instead of a storage error. Document the counter's reuse of hit_count and status.

Signed-off-by: divesh <dgude@nvidia.com>

---------

Signed-off-by: divesh <dgude@nvidia.com>
* docs(labels): select classification by reviewer relevance

Signed-off-by: Jim Meyer <jimeyer@nvidia.com>

* docs(labels): favor clear substantive topic relevance

Signed-off-by: Jim Meyer <jimeyer@nvidia.com>

* docs(labels): exclude routine supporting test coverage

Signed-off-by: Jim Meyer <jimeyer@nvidia.com>

---------

Signed-off-by: Jim Meyer <jimeyer@nvidia.com>
* refactor(triage): mark stale closures with stable metadata

Signed-off-by: Jim Meyer <jimeyer@nvidia.com>

* ci(triage): run labeling contract tests in branch checks

Signed-off-by: Jim Meyer <jimeyer@nvidia.com>

* test(triage): type stale closure fixtures for negative cases

Signed-off-by: Jim Meyer <jimeyer@nvidia.com>

---------

Signed-off-by: Jim Meyer <jimeyer@nvidia.com>
Signed-off-by: Jim Meyer <jimeyer@nvidia.com>
…ns (NVIDIA#5575)

* feat(review): annotate Review Readiness failures with their cause

Signed-off-by: Jim Meyer <jimeyer@nvidia.com>

* fix(review): explain how to resolve Review Readiness label failures

Signed-off-by: Jim Meyer <jimeyer@nvidia.com>

* fix(review): correct label failure guidance for missing areas and roster edits

Signed-off-by: Jim Meyer <jimeyer@nvidia.com>

* fix(ci): name the failing step in Review Readiness errors

Label failures now say what is wrong with this PR's labels, and GitHub
API failures name the step that failed, such as requesting reviewers.

Signed-off-by: Jim Meyer <jimeyer@nvidia.com>

* fix(review): report unknown area and topic labels as invalid

A label missing from REVIEWERS.md is treated as a wrong label on the PR,
so the message asks the author to replace it rather than edit the roster.

Signed-off-by: Jim Meyer <jimeyer@nvidia.com>

* fix(review): tell authors to remove invalid area and topic labels

Signed-off-by: Jim Meyer <jimeyer@nvidia.com>

---------

Signed-off-by: Jim Meyer <jimeyer@nvidia.com>
NVIDIA#5606)

The gateway recorded authentication and authorization outcomes but not the
state changes users and operators made, so there was no structured record of
who created, changed, or deleted a resource or whether the call succeeded.

Every call by a user or mTLS operator to a write-scoped RPC now emits one API
Activity event naming the actor, the method and its create, update, delete,
or other activity, the request ID, and the outcome derived from the gRPC
status. A streamed call is recorded when its response stream ends, with
the status from its trailers. A call or stream dropped before it
finished, such as when its client disconnected, records an unknown
outcome, since the change it started may still apply. An authorization denial raised for the call stands as its only
record. A catalog in the gateway lists the audited RPCs, and a test fails
when a write-scoped RPC is missing from it.

Refs NVIDIA#2911

Signed-off-by: Kris Hicks <khicks@nvidia.com>
tracing-subscriber's formatter enables ANSI unless NO_COLOR is set and
does no terminal detection, so the gateway, compute drivers, credential
drivers, and certgen wrote escape codes into container logs collected by
GKE and other log pipelines.

Move the CLI's automatic color rules (NO_COLOR, FORCE_COLOR, terminal
plus TERM capability) into openshell_core::color and pass the result to
with_ansi in every long-running binary's console layer. The CLI keeps
--color always|never and delegates its auto branch to the shared helper.

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
…VIDIA#4941)

* feat(gateway): expose workspace service route discovery

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>

* fix(gateway): gate service discovery on the app listener

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>

* test(discovery): remove redundant request defaults before reader extension

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>

* fix(discovery): separate and bound ingress discovery service

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>

* fix(discovery): wait for collection capacity

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>

* fix(discovery): share collection inputs and refresh accepted leases

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>

---------

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>
* test(conformance): port sandbox create --upload e2e tests to testkit

Move the three create_with_upload tests from e2e/rust into the conformance
suite. They read each uploaded file back and compare exact content, check the
nested file and that the directory is not flattened, and use the default
upload path and the target's default image. Remove the legacy test and its
now-unused SandboxGuard helpers.

Signed-off-by: Evan Lezar <elezar@nvidia.com>

* test(conformance): use expect_stdout for create-upload file checks

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

---------

Signed-off-by: Evan Lezar <elezar@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Co-authored-by: Matthew Grossman <mgrossman@nvidia.com>
…ow (NVIDIA#5526)

* test(supervisor): cover a wait that loses its boundary transport twice

A supervisor wait recovers from one boundary transport loss, but after
that recovery a later loss on the replacement connection is reported as
"boundary lost during wait" without any reattach attempt, once the first
outage's 30s recovery window has passed.

Add a regression test that holds a wait open across two transport
aborts more than CONNECT_RETRY_TIMEOUT apart and expects the wait to
reattach both times and return the exit status. It fails on this
commit; the following commit fixes it.

Signed-off-by: mr-miles <miles.waller@gmail.com>

* fix(supervisor): give each boundary outage its own wait recovery window

call_wait_stream opened its 30s recovery window at the first transport
failure and never reset it. A wait lasts as long as the workload, so
after one successful recovery any later transport loss found the window
long expired and returned Unavailable without attempting a reattach.
BoundaryProcess::wait() then reported "boundary lost during wait" and
the supervisor exited, even though the boundary was still reachable.
accept_tcp shares the same path and stopped the proxy the same way.

Key the window by connection generation. A failure on a newer
generation means a recovery replaced the connection, so it starts a
fresh window. Repeated failures on one generation still share a single
bounded window, so a live connection whose streams keep failing does
not retry forever.

This makes wait_survives_a_second_transport_loss_after_recovery pass,
and adds unit tests for the per-generation window itself.

Signed-off-by: mr-miles <miles.waller@gmail.com>

---------

Signed-off-by: mr-miles <miles.waller@gmail.com>
…5609)

A gateway mutation record named the method and outcome of a delete but not
what it deleted, and a delete of a resource that was already gone looked the
same as one that removed it.

Delete records now list the selected workspace and the requested resource
by name, with the sandbox ID once a sandbox delete is accepted. A delete
that found nothing to remove records status_detail no_op:already_absent, and
an asynchronous sandbox delete records accepted. Rejected, cancelled, and
handler-denied deletes still name the requested resource.

Refs NVIDIA#2911

Signed-off-by: Kris Hicks <khicks@nvidia.com>
…nts (NVIDIA#5450)

* test(runner): pass an explicit per-test timeout and grace to program rows

Program rows relied on the program's 900 s default, so one hung test could use much of the 45-minute run timeout. Derive the per-test timeout from the same constants as the nextest profile (120 s) and pass it with a 10 s grace to the program.

Part of NVIDIA#5416.

Signed-off-by: Evan Lezar <elezar@nvidia.com>

* test(runner): keep program rows on the program's default timeout

Passing the runner's 120 s limit to the program would stop a test before
testkit's own step timeouts (up to 10 minutes) can report which step
stalled, contrary to the program's documented contract. Keep program rows
on the program's default until testkit learns the per-test deadline
(NVIDIA#5333).

Write nextest's grace period into the generated profile from the same
constants, rather than relying on nextest's default.

Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>

---------

Signed-off-by: Evan Lezar <elezar@nvidia.com>
Signed-off-by: Matthew Grossman <mgrossman@nvidia.com>
Co-authored-by: Matthew Grossman <mgrossman@nvidia.com>
Signed-off-by: Eric Curtin <eric.curtin@docker.com>
* fix(ci): let active dev releases finish

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

* fix(ci): preserve E2E cancellation outside dev releases

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

* docs(ci): shorten release concurrency guidance

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

* refactor(ci): make E2E cancellation a caller option

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>

---------

Signed-off-by: Piotr Mlocek <pmlocek@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.