Repository navigation
[5.2] Backport HWORKS-3457, HWORKS-3473, HWORKS-3581, HWORKS-3579, HWORKS-3562 - #729
Merged
robzor92 merged 5 commits intoOct 27, 2026
Merged
Conversation
…ough Run:ai (logicalclocks#702) * [HWORKS-3457] Run:ai: manage GPU resources for Hopsworks projects through Run:ai https://hopsworks.atlassian.net/browse/HWORKS-3457 On a cluster where NVIDIA Run:ai manages the GPUs, Hopsworks read GPU availability from the Kubernetes API and created GPU pods that Run:ai's quotas, fair share and preemption did not see. With the new run_ai_enable variable each Hopsworks project is bound to a Run:ai project on its own namespace, and the project's GPU view comes from Run:ai. logicalclocks.github.io adds the administrator page "GPU Scheduling with Run:ai": what Hopsworks creates in Run:ai, the service account and role it needs, the Helm values, the GPU availability fields, and what turning the integration off leaves behind. The generated configuration variables reference is regenerated from hopsworks-ee, which adds the run_ai_* keys and catches up with keys master already had. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [HWORKS-3457] Run:ai: manage GPU resources for Hopsworks projects through Run:ai https://hopsworks.atlassian.net/browse/HWORKS-3457 On jim-run-ai an access rule created through the Run:ai API with the service account's id as subjectId was Ready but granted nothing: every call answered 403 until the rule named the service account instead, which is what Run:ai's own UI sends. logicalclocks.github.io says so in the prerequisites of the Run:ai admin page. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [HWORKS-3457] Run:ai: manage GPU resources for Hopsworks projects through Run:ai https://hopsworks.atlassian.net/browse/HWORKS-3457 Review found two gaps: GPU notebooks, terminals, Python apps and model deployments wait forever at the default quota of 0, because Run:ai admits non-preemptible workloads only within quota; and the page did not say how Hopsworks behaves when Run:ai is down. logicalclocks.github.io documents the quota check Hopsworks now runs before starting those workloads, what keeps working during a Run:ai control-plane outage, and that Run:ai's fail-closed admission webhooks block every pod in bound project namespaces while they are down. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [HWORKS-3457] Run:ai: manage GPU resources for Hopsworks projects through Run:ai https://hopsworks.atlassian.net/browse/HWORKS-3457 Kueue and Run:ai are now mutually exclusive in Hopsworks: the Helm install and Cluster settings both refuse a configuration with both on. logicalclocks.github.io states the rule on the Run:ai admin page. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [HWORKS-3457] Run:ai: manage GPU resources for Hopsworks projects through Run:ai https://hopsworks.atlassian.net/browse/HWORKS-3457 Hopsworks now moves a project's Run:ai project when an administrator moves the project to another department. logicalclocks.github.io describes the move on the Run:ai admin page: when it is refused, that the Run:ai project is recreated without GPU quota, and that the quota must be re-added in Run:ai. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [HWORKS-3457] Run:ai: manage GPU resources for Hopsworks projects through Run:ai https://hopsworks.atlassian.net/browse/HWORKS-3457 A workload in a project not yet bound to Run:ai no longer starts without a queue during a Run:ai outage. logicalclocks.github.io says such a workload waits until Run:ai is reachable, and why. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [HWORKS-3457] Run:ai: manage GPU resources for Hopsworks projects through Run:ai https://hopsworks.atlassian.net/browse/HWORKS-3457 Hopsworks department names now follow the Kubernetes name rule (HWORKS-3456), so they map one to one to Run:ai department names. logicalclocks.github.io: state the department name rule in the Run:ai guide. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [HWORKS-3457] Run:ai: manage GPU resources for Hopsworks projects through Run:ai https://hopsworks.atlassian.net/browse/HWORKS-3457 With Run:ai enabled, the admin Departments and Projects pages show the GPU quota and allocated GPUs of the Run:ai department or project each one is linked to, or that it is not linked in Run:ai. logicalclocks.github.io: describe the Run:ai columns of the admin Departments and Projects pages. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [HWORKS-3457] Run:ai: manage GPU resources for Hopsworks projects through Run:ai https://hopsworks.atlassian.net/browse/HWORKS-3457 HWORKS-3456 as merged also runs the department-move check when a department is deleted, one project at a time. logicalclocks.github.io: describe what deleting a department does to the Run:ai projects of its projects. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [HWORKS-3457] Run:ai: manage GPU resources for Hopsworks projects through Run:ai https://hopsworks.atlassian.net/browse/HWORKS-3457 Review fixes for the department move, the admin Run:ai view and the Run:ai secret wiring. logicalclocks.github.io: a department without a Run:ai counterpart maps to run_ai_default_department; a move needs an empty Run:ai project, loses the project's access rules and limits, and restores the old project on failure; RUN_AI_SCHEDULER_NOT_ENFORCED listed. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [HWORKS-3457] Run:ai: manage GPU resources for Hopsworks projects through Run:ai https://hopsworks.atlassian.net/browse/HWORKS-3457 A department move is refused while the project namespace has a live pod. On jim-run-ai (Run:ai 3.0.20) a move with deletionType=IfEmpty went through with a pod running: the workload list lags pod creation and the delete did not check it. logicalclocks.github.io: the docs describe the busy check as a running or waiting pod in the project namespace. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [HWORKS-3457] Run:ai: manage GPU resources for Hopsworks projects through Run:ai https://hopsworks.atlassian.net/browse/HWORKS-3457 PR review fixes (Copilot, o-alex). logicalclocks.github.io: project creation and run_ai_default_department describe the department mapping, and the Secret example quotes the placeholder. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [HWORKS-3457] Run:ai: manage GPU resources for Hopsworks projects through Run:ai https://hopsworks.atlassian.net/browse/HWORKS-3457 PR review fixes (o-alex). logicalclocks.github.io: the Run:ai guide lists the permissions per call, says which projects Hopsworks deletes or moves, that reading GPU availability does not bind, and to remove the Run:ai bindings before enabling Kueue. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [HWORKS-3457] Run:ai: manage GPU resources for Hopsworks projects through Run:ai https://hopsworks.atlassian.net/browse/HWORKS-3457 The Hopsworks Run:ai service account keeps the System administrator role; the permissions Hopsworks uses are listed for security reviews. logicalclocks.github.io: the prerequisite asks for System administrator again and drops the untested Editor alternative. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [HWORKS-3457] Run:ai: manage GPU resources for Hopsworks projects through Run:ai https://hopsworks.atlassian.net/browse/HWORKS-3457 Run:ai is enabled through the Hopsworks Helm chart only, which also mounts its client secret; Cluster settings no longer changes run_ai_enable. logicalclocks.github.io: the Run:ai guide says the integration is enabled through Helm only. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit a0c3e5c)
robzor92
force-pushed
the
5.2-backports-2026-10-26
branch
from
October 26, 2026 22:24
737affb to
4304315
Compare
…d-on image tags https://hopsworks.atlassian.net/browse/HWORKS-3473 The 5.2.2 chart pulls Hopsworks rebuilds of the KServe controller and the KEDA HTTP add-on images, tagged 0.21.0-h1 and 0.16.0-h2 under their upstream names. Those tags exist only on docker.hops.works and its mirrors, so the KEDA page tells airgapped installs to mirror them from there. This is the 5.2 counterpart of the chart README change in hopsworks-helm #2566; on main the same paragraph reached keda.md when the chart docs moved to this site. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
agent_deployment_otel_memory_mb is 2048 now. The page is written whole from the Java sources, so the regeneration also picks up six keys merged since it was last generated: the agent pipeline base image, the four OpenCost settings and the Trino subset-share flag. Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> (cherry picked from commit 13621d1)
…rks Kueue models to v1beta2 (logicalclocks#709) * [HWORKS-3565] Upgrade Kueue from the bundled 0.12.2 to 0.18.1 installed outside the Hopsworks namespace, migrating existing queues and CRDs https://hopsworks.atlassian.net/browse/HWORKS-3565 Every release from 4.3 bundles Kueue 0.12.2 in the Hopsworks release namespace. From 5.3 Hopsworks can use a Kueue installed as its own release in kueue-system instead (hw-kueue.kueue.enabled=false), and the chart ships files/migrate-to-external.sh to move a running cluster to Kueue 0.18.1 while keeping every Kueue object. The docs add an operator page for that move: why helm upgrade alone cannot do it, the script's phases, what to prepare (snapshot, maintenance window, tools, airgapped images, admin API key), the command, what changes for queues and Hopsworks afterwards (no topologyName on flavors, parentName on Cohorts, queues managed with kubectl, Hopsworks-namespace jobs queued, the queue utilisation view failing until the backend reads Cohorts at v1beta1) and how to verify. The hw-kueue values page links to it. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [HWORKS-3565] Upgrade Kueue from the bundled 0.12.2 to 0.18.1 installed outside the Hopsworks namespace, migrating existing queues and CRDs https://hopsworks.atlassian.net/browse/HWORKS-3565 Clusters whose releases Flux manages. Flux reconciles a HelmRelease back to Git, so the helm upgrades in remove-bundled, install-kueue and enable-hopsworks would be reverted on a Flux-managed cluster. logicalclocks.github.io: the Kueue operator page gains a section on clusters managed by Flux (what to commit at each phase, what the script waits for, the queue export) and lists export-queues. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [HWORKS-3565] Upgrade Kueue from the bundled 0.12.2 to 0.18.1 installed outside the Hopsworks namespace, migrating existing queues and CRDs https://hopsworks.atlassian.net/browse/HWORKS-3565 Step-by-step instructions for clusters managed by Flux. logicalclocks.github.io: the Flux section of the Kueue operator page becomes a runbook for --gitops: what to have before starting (Flux 2.3 or later, Git access, the Helm release name and namespace, where the HelmRelease values live, a terminal that survives a dropped connection), then one step per change to commit, with the command, the HelmRelease YAML, the flux reconcile command that applies it at once and what the script waits for, then where the queue manifests go (a Kustomization depending on the Kueue one) and how to resume after a timeout. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [HWORKS-3565] Upgrade Kueue from the bundled 0.12.2 to 0.18.1 installed outside the Hopsworks namespace, migrating existing queues and CRDs https://hopsworks.atlassian.net/browse/HWORKS-3565 Review comments on logicalclocks/hopsworks-helm#2599. logicalclocks.github.io: the Kueue operator page runs the migration script with bash in every command, because helm package stores it without the execute bit and a direct call fails with permission denied. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [HWORKS-3579] Upgrade Kueue from 0.18.1 to 0.20.2 and move the Hopsworks Kueue models to v1beta2 https://hopsworks.atlassian.net/browse/HWORKS-3579 Kueue 0.20.2 is the latest release and stops serving kueue.x-k8s.io/v1beta1, which every Hopsworks Kueue model still used. Kueue 0.18.1 serves both v1beta1 and v1beta2, so the backend moves to v1beta2 first and runs on 0.18.1 and 0.20.2, then the external Kueue installed by HWORKS-3565 is upgraded to 0.20.2 and the bundled one in hw-kueue moves to 0.20.2 as well. Tested on the Proxmox cluster jim-kueue-ui (snurran, 4 GPUs), migrated to 0.18.1 by HWORKS-3565: the upgrade kept all 106 queue objects and verify passed, and the loadtests gave 31 passed, 1 skipped on 0.20.2 (one Ray MNIST run failed in its training script and passed when rerun). logicalclocks.github.io: the Kueue operator page says 5.3 bundles Kueue 0.20.2 and that a cluster on the bundled 0.12 moves to the external Kueue when it upgrades. It adds the upgrade-kueue phase, the managedFields reset, the path for a cluster already on 0.18.1, the mirrors an airgapped cluster needs, the 0.19 and 0.20 default changes, and the prerequisite check for both modes. The queue utilisation gap is gone. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [HWORKS-3579] Upgrade Kueue from 0.18.1 to 0.20.2 and move the Hopsworks Kueue models to v1beta2 https://hopsworks.atlassian.net/browse/HWORKS-3579 Spark jobs on Kueue. Kueue never managed Hopsworks Spark jobs: they run as Spark Operator SparkApplications, which Kueue admits only through its SparkApplication integration, alpha since 0.17 and off by default (SparkApplicationIntegration gate, sparkoperator.k8s.io/sparkapplication commented out of the default frameworks). The integration reserves quota for the driver plus spec.executor.instances and refuses dynamicAllocation.enabled unless the job is elastic, and even an elastic job leaves the executors its driver adds uncharged. So the integration is turned on and, while Kueue is on, Spark jobs run with static executors only. logicalclocks.github.io: the Kueue details page gains a Spark jobs section (static executors only, error 130082, defaults, jobs saved before Kueue, notebooks unaffected) and its examples move to kueue.x-k8s.io/v1beta2 (cohortName, no topologyName on the default flavor). The scheduler page, the Spark and PySpark job guides and the Kueue operator page say Spark jobs are queued with static executors. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [HWORKS-3579] Upgrade Kueue from 0.18.1 to 0.20.2 and move the Hopsworks Kueue models to v1beta2 https://hopsworks.atlassian.net/browse/HWORKS-3579 Spark jobs reach Kueue. With the SparkApplication integration on, a test run still showed no Spark Workload: the backend gave Spark jobs no queue, so their SparkApplications carried no queue-name label and Kueue ignored them. With a queue, Kueue admitted the job and then could not start it: unsuspending sets the driver and executor pod templates, and the Spark Operator 2.5.1 webhook refused that with "pod template feature requires Spark version 3.0.0 or higher", because it compares spec.sparkVersion as semver and ranks an invalid one below every valid one, and Hopsworks writes 4.1.3.0. On jim-kueue-ui, Kueue 0.20.2 with the integration on, the Kueue, Spark job, default job configuration, rerun and job failure loadtests gave 14 passed, and Kueue admitted 9 SparkApplication Workloads into ClusterQueue other. logicalclocks.github.io: the Kueue details page says Spark jobs go to the project's default queue unless they select another, and that a Spark job saved without a queue gets the default when it starts. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [HWORKS-3579] Upgrade Kueue from 0.18.1 to 0.20.2 and move the Hopsworks Kueue models to v1beta2 https://hopsworks.atlassian.net/browse/HWORKS-3579 Fixes from a Codex review of the change set (nine findings and one suspicion, each checked against the code before it was fixed). logicalclocks.github.io: the Kueue operator page says the managedFields cleanup drops only old entries, that the 0.20.2 chart comes from the upstream OCI registry (--kueue-target-repo), and which flags point an airgapped cluster at its mirrors. Reviewed-by: OpenAI Codex (codex-plugin-cc 1.0.6) <codex@openai.com> Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [HWORKS-3579] Upgrade Kueue from 0.18.1 to 0.20.2 and move the Hopsworks Kueue models to v1beta2 https://hopsworks.atlassian.net/browse/HWORKS-3579 Rebased on HWORKS-3565 with its Flux runbook. logicalclocks.github.io: the Flux runbook on the Kueue operator page gains Step 5, upgrade-kueue: the script applies the 0.20.2 CRDs and rewrites the Kueue HelmRelease for 0.20.2 (chart from the upstream OCI registry, Spark settings, a post-render patch that keeps the chart's templated CRDs out of the release), the operator replaces the file committed in Step 3, and the script waits for the controller to run 0.20.2. Verification and the queue export become Step 6. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [HWORKS-3579] Upgrade Kueue from 0.18.1 to 0.20.2 and move the Hopsworks Kueue models to v1beta2 https://hopsworks.atlassian.net/browse/HWORKS-3579 Kueue 0.20.3. The upgrade-kueue entry states that it waits for the Hopsworks release to finish deploying, keeps release kueue's own values and refuses a manager configuration that is not v1beta2. Reviewed-by: OpenAI Codex (codex-plugin-cc 1.0.6) <codex@openai.com> Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [HWORKS-3579] Upgrade Kueue from 0.18.1 to 0.20.2 and move the Hopsworks Kueue models to v1beta2 https://hopsworks.atlassian.net/browse/HWORKS-3579 The 5.3 chart no longer ships migrate-to-external.sh: a cluster on the bundled Kueue 0.12 moves to the external Kueue 0.18.1 on Hopsworks 5.1, upgrades Hopsworks to 5.3, then upgrades Kueue to 0.20.3 with the 5.1 script's upgrade-kueue phase. docs: kueue_external.md runs the move in those three steps, with the latest 5.1 chart, and the Flux steps follow the same order. all no longer includes upgrade-kueue, which refuses a Hopsworks release older than 5.3. The --kueue-image option for the upstream images is listed. The upgrade guide's Kueue entry says where the script comes from. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [HWORKS-3579] Upgrade Kueue from 0.18.1 to 0.20.2 and move the Hopsworks Kueue models to v1beta2 https://hopsworks.atlassian.net/browse/HWORKS-3579 The Kueue 0.20 changes ship in a 5.2.x patch release as well as in 5.3. docs: kueue_external.md and the upgrade guide's Kueue entry name the target as a Hopsworks release that bundles Kueue 0.20, a 5.2.x patch release or 5.3, in place of 5.3, and upgrade-kueue's refusal as a chart that bundles an older Kueue. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [HWORKS-3579] Upgrade Kueue from 0.18.1 to 0.20.2 and move the Hopsworks Kueue models to v1beta2 https://hopsworks.atlassian.net/browse/HWORKS-3579 A 5.1 cluster stays on Kueue 0.18; the step to Kueue 0.20.3 ships with the Hopsworks release that bundles it. docs: kueue_external.md names two scripts. The 5.1 chart's migrate-to-external.sh moves a cluster to Kueue 0.18.1 on 5.1, and the new chart's upgrade-kueue.sh, run after the Hopsworks upgrade, takes it to 0.20.3. The upgrade-kueue phase row gives way to a description of upgrade-kueue.sh and its refusals; the Flux Step 5 runs it with --gitops and commits the HelmRelease it writes to ./kueue-upgrade. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [HWORKS-3579] Upgrade Kueue from 0.18.1 to 0.20.2 and move the Hopsworks Kueue models to v1beta2 https://hopsworks.atlassian.net/browse/HWORKS-3579 docs: a 5.2.0 or 5.2.1 cluster on the bundled Kueue 0.12 moves with the 5.1 chart's migrate-to-external.sh run phase by phase: the phases up to migrate-storage on its own 5.2 chart, then enable-hopsworks with the new release, so Hopsworks is never downgraded. The upgrade note says the move is required for a cluster on the bundled Kueue, since the new chart refuses it, and that upgrade-kueue.sh, not the 5.1 script, takes Kueue to 0.20.3. The guide also covers what changed in the scripts and chart: Spark jobs saved with dynamic allocation run their dynamic-allocation minimum instead of being refused; upgrade-kueue.sh keeps TopologyAwareScheduling off when a release turns it off, and with --gitops writes kueue-crds.yaml and waits for the SparkApplication integration; the bundled CRDs carry an Argo CD no-prune, no-delete option; the chart tells its own CRDs by label and refuses an upgrade that turns Kueue on while they are missing. Neither script supports a release managed by Argo CD. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011tczJCVzi8QNsr26x1rDNh * [HWORKS-3579] Upgrade Kueue from 0.18.1 to 0.20.2 and move the Hopsworks Kueue models to v1beta2 https://hopsworks.atlassian.net/browse/HWORKS-3579 docs: the "Jobs in the Hopsworks namespace are queued" change is removed. Hopsworks creates its Kueue-queued system jobs in the project namespaces, so Kueue 0.18 excluding its own namespace changes nothing for Hopsworks. The Kueue entry in upgrade.md lists what changes for users and operators: static Spark executors, the 30-minute waitForPodsReady, Ray sidecars charged to quota and the default priority class, with a link to the details. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011tczJCVzi8QNsr26x1rDNh * [HWORKS-3579] Upgrade Kueue from 0.18.1 to 0.20.3 and move the Hopsworks Kueue models to v1beta2 https://hopsworks.atlassian.net/browse/HWORKS-3579 docs: the intro names the releases that bundle Kueue 0.20.3; "Those releases" pointed back at 4.3 to 5.2, which bundle 0.12.2. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011tczJCVzi8QNsr26x1rDNh * [HWORKS-3579] Upgrade Kueue from 0.18.1 to 0.20.3 and move the Hopsworks Kueue models to v1beta2 https://hopsworks.atlassian.net/browse/HWORKS-3579 docs: the Hopsworks chart ships no Kueue, as it ships no Kyverno. A new Fresh install section installs Kueue 0.20.3 first as release kueue in kueue-system with the chart's upgrade-kueue.sh install, and covers Argo CD: an Application of its own for the upstream chart with the SparkApplication settings, synced first and without the resources finalizer. The chart creates the default queues only when missing, and its prerequisite check refuses kueue.enabled=true. The bundled 0.20.3 and its crds/ are gone from the guide and the upgrade note. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011tczJCVzi8QNsr26x1rDNh --------- Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit f195a5d)
…one-way sync into a project dataset (logicalclocks#715) * [HWORKS-3562] SharePoint data source with one-off copy and scheduled one-way sync into a project dataset https://hopsworks.atlassian.net/browse/HWORKS-3562 Users cannot bring files from SharePoint Online document libraries into Hopsworks. Add SharePoint as a data source and a sync that copies a library folder into a new project dataset with rclone, once or on a schedule, run as an ordinary Python job whose script lives in the project. Backend (hopsworks-ee), client and CLI (hopsworks-api), UI (hopsworks-front), image (docker-images), variables (hopsworks-helm), an e2e test (loadtest) and docs change together. Add the SharePoint data source guide: the Entra ID application and its permissions, network access, the minimum client version, the cluster variables, creating the data source and a sync in the UI, the data source page, how a sync runs (the job and who it runs as, the editable sync script, the mirrored dataset and what data owners may change in it, .partial download files, the CSV audit log and querying it with DuckDB, skipped item types, how file names are stored, failures, deletion), and the Python client and CLI. Reviewed-by: OpenAI Codex (codex-plugin-cc 1.0.6) <codex@openai.com> Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [HWORKS-3562] SharePoint data source with one-off copy and scheduled one-way sync into a project dataset https://hopsworks.atlassian.net/browse/HWORKS-3562 A sync can index the files it copies in a Delta feature group, one row per file, partitioned by the day the file was first copied in. The row holds the SharePoint ids, path, web link, type, size, times, author and editor, quickXorHash and the path of the copy in Hopsworks, but no content. Each run diffs a full listing of the folder against the table and upserts or deletes rows with delta-rs through the SDK, then compacts earlier days to about 200 MiB files and vacuums after 7 days. Deletion vectors stay off: hops-deltalake 1.4 still rewrites files on delete. The SharePoint guide documents the index option, its columns and maintenance rules, the Python and CLI calls, and a DuckDB example. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [HWORKS-3562] SharePoint data source with one-off copy and scheduled one-way sync into a project dataset https://hopsworks.atlassian.net/browse/HWORKS-3562 The terminal mounts each feature group under the project's database directory, /hopsfs/featurestore/<project>_featurestore.db/<fg>_<version>, so the index paths in the search skill and the guide gain that level. Checked with delta_scan in a terminal on a 269-file index. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [HWORKS-3562] SharePoint data source with one-off copy and scheduled one-way sync into a project dataset https://hopsworks.atlassian.net/browse/HWORKS-3562 Fixes from a Codex review of the file index. A sync deletes only the index feature group its runs created, recognised by the description the script writes, and an index name must not be an existing feature group. The partition is now described as the day a file was first indexed, since files already in the dataset when indexing is turned on get that day. The guide states the ownership and path-reuse rules, and keeps one sentence per line. Reviewed-by: OpenAI Codex (codex-plugin-cc 1.0.6) <codex@openai.com> Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [HWORKS-3562] SharePoint data source with one-off copy and scheduled one-way sync into a project dataset https://hopsworks.atlassian.net/browse/HWORKS-3562 Review fix: the SharePoint guide says what bounds the size of a sync. The synced files count against the project's storage quota; the maximum file size limits each file, not the total. A run that fills the quota is partial with a quota error, and the next run copies the remaining files once there is room. The file index table also gets the separator row markdownlint expects. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * [HWORKS-3562] SharePoint data source with one-off copy and scheduled one-way sync into a project dataset https://hopsworks.atlassian.net/browse/HWORKS-3562 The SharePoint guide's API reference box names the SharePointConnector and SharePointSync methods as code instead of linking them. The docs build resolves those links against hopsworks-api main, which has no SharePoint classes until hopsworks-api#1353 merges, so the strict build failed on 13 unresolved references. Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Signed-off-by: Jim Dowling <jim@logicalclocks.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com> (cherry picked from commit 359af4a)
robzor92
force-pushed
the
5.2-backports-2026-10-26
branch
from
October 27, 2026 06:31
4304315 to
180968c
Compare
robzor92
changed the title
robzor92
marked this pull request as ready for review
October 27, 2026 12:10
gibchikafa
approved these changes
Oct 27, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cherry-picks to
branch-5.2, with-x, in the order they merged on master:HWORKS-3456 (Departments) and HWORKS-3457 (Run:ai) are included because the later picks build on them.
Conflicts and what differs from master
Settings.javadoes not have:docker_base_image_agent_pipeline(HWORKS-2868) andenable_opencost,opencost_api_url,opencost_currency,opencost_history_retention_days(HWORKS-3461). It also carries master's upgrade note "Agent deployments reserve 2 GiB for their OTLP sidecar" (from HWORKS-3580, [HWORKS-3580] Publish the Helm upgrade guide and Helm 4 migration notes on the docs site, then stop packaging them in the chart #711), placed under "Changes from 5.2.2" with 5.3 changed to 5.2.2, since the 5.2 chart has noUPGRADE.mdfor it.Checked
Link anchors checked against the 5.2 tree; mkdocs not built.
The loadtests have not yet run against a 5.2 cluster with these changes.
🤖 Generated with Claude Code