[jimdowling]

https://hopsworks.atlassian.net/browse/HWORKS-3579

Stacked on #708 (HWORKS-3565): until it merges this PR also shows its commit.

Summary

docs/setup_installation/admin/kueue_external.md:

• 5.3 bundles Kueue 0.20.2 and its backend reads v1beta2, so a cluster on the bundled 0.12 moves to the external Kueue when it upgrades (it was optional).
• The upgrade-kueue phase, the managedFields reset, and the two-phase path for a cluster already on 0.18.1.
• Airgapped mirrors for 0.18.1 and 0.20.2; --kueue-repo takes oci://.
• The 0.19 and 0.20 default changes, and the prerequisite check in both modes.
• The queue utilisation gap is removed: the HWORKS-3579 backend fixes it.

markdownlint-cli2 passes.

Spark jobs on Kueue

Kueue never managed Hopsworks Spark jobs: they run as Spark Operator SparkApplications, which Kueue admits only through its SparkApplication integration (alpha since 0.17, off by default). That integration reserves quota for the driver plus spec.executor.instances and refuses dynamicAllocation.enabled unless the job is elastic, and even then the executors the driver adds are not charged. This PR set therefore:

• turns the integration on (SparkApplicationIntegration gate plus the sparkoperator.k8s.io/sparkapplication framework) in migrate-to-external.sh, the bundled hw-kueue values and terraform's install_kueue.sh (logicalclocks/terraform#178);
• refuses Spark jobs with dynamic allocation while Kueue is on (error 130082, at save, project default and start) and makes default Spark job configurations static, with the dynamic-allocation maximum as the executor count; notebook Spark (client mode through Livy) is unchanged;
• puts Spark jobs in the project's default queue, including jobs saved before (at save and at start);
• writes spec.sparkVersion as semver (4.1.3, not 4.1.3.0): the Spark Operator 2.5.1 webhook ranks an invalid version below 3.0.0 and refused the pod templates Kueue sets when it resumes an admitted job, which left every Spark job suspended.

Spark jobs saved before this release have dynamic allocation on (the old default) and are refused on a Kueue cluster until they are switched to static executors; UPGRADE.md and the docs say so.

Spark testing on jim-kueue-ui (Kueue 0.20.2, integration on, backend from this branch): test_kueue_queues.py, test_spark_job_execution.py, test_default_job_configuration.py, test_execution_rerun.py and test_job_failure_handling.py gave 14 passed. A watcher on the cluster recorded 9 Workloads owned by SparkApplications, all admitted to ClusterQueue other.

Related PRs

• logicalclocks/hopsworks-ee#3548 (backend models on v1beta2)
• logicalclocks/hopsworks-helm#2602 (upgrade-kueue, bundled 0.20.2, prerequisite check)
• logicalclocks/loadtest#1181 (test_kueue_queues.py)
• logicalclocks/docker-images#1130 (Kueue image mirrors)
• [HWORKS-3579] Upgrade Kueue from 0.18.1 to 0.20.2 and move the Hopsworks Kueue models to v1beta2 #709 (operator docs)

🤖 Generated with Claude Code

[o-alex]

Move the upgrade note into upgrade.md. logicalclocks.github.io#711 has merged: docs/setup_installation/admin/upgrade.md is now the upgrade guide, with a ## 5.2.x to 5.3.x section (#upgrade-52-to-53). hopsworks-helm#2609 deletes UPGRADE.md, and maismail asked on hopsworks-helm#2602 for that PR's upgrade note to move to the docs site and merge no later than the chart PR.

Please add a ### Kueue 0.20 subsection under 5.2.x to 5.3.x and drop the 150 UPGRADE.md lines from hopsworks-helm#2602. Given the forced hop through 5.1/5.2 (see the comment on hopsworks-helm#2602), it only needs:

• the prerequisite: a cluster with Kueue must already be on external Kueue 0.18.1, moved there by the 5.1 or 5.2 migrate-to-external.sh. A cluster still on the bundled 0.12 upgrades to 5.1 or 5.2 first.
• the step: upgrade Hopsworks to 5.3, then run upgrade-kueue and verify.
• the behaviour changes: Spark jobs need static executors on Kueue, 0.19/0.20 waitForPodsReady 30m, Ray sidecars charged to quota, the default priority class.
• a link to kueue_external.md for the details.

kueue_external.md itself can then drop the bundled 0.12 migration and its Flux runbook (:19-78, :89-215), which apply to 5.1/5.2, not 5.3.

[o-alex]

Please drop this bullet. It is not a change an operator sees. Hopsworks creates its Kueue-queued system jobs in the project namespaces, not the Hopsworks namespace: zipping, git, library installs and image builds, and DLT schema fetches (hopsworks-ee KubeSystemJobManger.java:178,185, KubeDockerUtils.java:1712, NoSqlSourceSchemaController.java:317). Before 0.18, Kueue also managed batch Jobs in its own namespace, so the bundled 0.12 queued these jobs too. Kueue 0.18 excluding its own namespace (CHANGELOG-0.18.md:661-673) therefore changes nothing for Hopsworks, with either the bundled or the external Kueue.

The same claim, in its bundled form ("system jobs are not queued with the bundled Kueue"), is in hopsworks-helm#2602's UPGRADE.md and specification.md. That PR has a comment asking for it to be removed too, so please don't carry it into the 5.3 upgrade section either.