[bubriks]

https://hopsworks.atlassian.net/browse/FSTORE-1973

Summary

From 5.3, with an external Kafka (enable_bring_your_own_kafka), Hopsworks creates and deletes the topics it needs on that cluster through each project's kafka_connector, a feature group's notification topic becomes a topic of its project, and Kafka changes through a shared feature store are refused (hopsworks-ee PR below). The external Kafka guide said Hopsworks provisions no topics there and told users to pre-create them.

• setup_installation/admin/upgrade.md, 5.2.x to 5.3.x (moved here from the chart's UPGRADE.md, which hopsworks-helm#2609 removed):
  • "Bring your own Kafka: Hopsworks creates and deletes topics": broker reachability from the Hopsworks pods, existing topics used as they are, the operations that create topics and the principal's Create and Delete, what is deleted and on which cluster, rows recorded before the upgrade staying on the bundled Kafka (migration V126), partitions and replicas, the OnlineFS topicPattern, inference logging staying on the bundled Kafka.
  • "Notification topics and shared feature stores": notification topics belong to their project (190026 for another project's), and changes through a shared feature store fail with 190027.
  • The 5.2.2 OnlineFS topic section now says the external Kafka exception holds on 5.2.2 only, and links the 5.3 rule.
• setup_installation/on_prem/external_kafka_cluster.md:
  • Configuration is shown as Helm values (hopsworks.variables.enable_bring_your_own_kafka, onlinefs.dependencies.kafka, onlinefs.configmapOverrides.kafka) in place of the cluster-definition YAML.
  • OnlineFS must subscribe by topicPattern (default .*_onlinefs), since a fixed topicList misses the topics Hopsworks creates later.
  • "Topics on the external Kafka cluster": which topics are created, that existing topics are used and never deleted, which topics are deleted with their feature group or project, notification topics, and that owned topics are not shared between projects.
  • A warning that topics are deleted by name on the cluster the kafka_connector points at when they are deleted.
  • A permissions table for the kafka_connector principal (Describe, Read, Write, Create, Delete) and what fails without each.
  • Partitions and replicas: onlinefs_service_thread_number and kafka_num_replicas, the requested values for topics created through the topic API, and the min.insync.replicas interaction.
• user_guides/fs/feature_group/topic.md: a topic that does not exist is created on the external Kafka too, and the topicPattern note names the Helm value.
• user_guides/fs/feature_group/notification.md: the notification topic is a topic of the project that Hopsworks creates and deletes, instead of one to create first.
• user_guides/fs/data_source/usage.md: a confluent_options() example for producers and consumers of your own.
• reference/rest_error_codes.md: regenerated from hopsworks-ee. It adds 190026 and 190027, rewords 190003, and picks up the codes other merged changes added since the last run.

Related

• hopsworks-ee (backend): https://github.com/logicalclocks/hopsworks-ee/pull/3508
• hopsworks-api (SDK: Kafka and ADLS connector type names, SASL mechanism): [FSTORE-1973] Allow for managed BYOK hopsworks-api#1296
• hopsworks-helm (satellite CI check): https://github.com/logicalclocks/hopsworks-helm/pull/2572
• loadtest (e2e_byok tests, notification topic tests): https://github.com/logicalclocks/loadtest/pull/1161

Merge with or after the hopsworks-ee PR, since this describes its behaviour, and no later than the hopsworks-helm PR. The change ships in 5.3 only, so there is no branch-5.2 counterpart.

Test plan

• uv run hopsworks-docs markdownlint passes.
• Strict uv run hopsworks-docs check build passes.
• The new Python block passes the snakeoil ruff rule set.
• The described behaviour was checked on dev cluster bubriks with the backend branch: the hopsworks-ee PR lists the loadtests.

🤖 Generated with Claude Code

[Copilot]

🔵 Needs a closer look

The linked backend and Helm changes remain unavailable for verification of destructive topic behavior, and the permissions guidance needs correction.

1 open issue

• Add Kafka topic Write permission for Feature Store ingestion · New

What changed in this PR

Updates the documentation for Hopsworks 5.3 managed topic creation and deletion on external Kafka clusters.

Changes:

• Replaces cluster-definition examples with Helm configuration and pattern-based subscriptions.
• Documents topic ownership, deletion risks, permissions, replication, and client configuration.
• Aligns Feature Group topic guidance with external topic provisioning.

File	Description
docs/​user_guides/​fs/​feature_group/​topic.md	Updates topic creation and subscription guidance.
docs/​setup_installation/​on_prem/​external_kafka_cluster.md	Documents configuration and managed external-topic behavior.

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Destructive topic-lifecycle guarantees need human confirmation against the linked backend change, which was unavailable for verification.

1 open issue

• Move lifecycle concepts to Feature Group documentation · New

1 previously missed issue

In code that hasn't changed since last review

Move the SDK example to user guide documentation

docs/​setup_installation/​on_prem/​external_kafka_cluster.md:149

This introduces a user-facing SDK how-to inside the infrastructure/admin guide. .claude/docs/README.md:26–27 places task-focused feature usage under user_guides/. Move the confluent_options() example to the Kafka Data Source usage documentation and link to it here, keeping this page focused on cluster integration.

1 resolved since last review

• Add Kafka topic Write permission for Feature Store ingestion

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

The example needs correction, and topic-deletion behavior and Helm settings still require verification against the inaccessible backend and chart changes.

1 open issue

• Use feature_store instead of undefined fs · New

1 resolved since last review

• Move lifecycle concepts to Feature Group documentation

_{Review effort: Balanced}

[Copilot]

🔵 Needs a closer look

Destructive topic-lifecycle behavior and Helm settings still need confirmation against the linked backend and chart changes.

No open issues 🎉

2 previously missed issues

In code that hasn't changed since last review

Keep each operation description to one sentence per line

docs/​setup_installation/​on_prem/​external_kafka_cluster.md:105

The Describe, Write, Read, Create, and Delete descriptions each contain multiple sentences on one source line, contrary to the repository's one-sentence-per-line rule. Rephrase each description as a single sentence while keeping the Markdown table rows intact.

Use heading-ID syntax for the cross-reference

docs/​setup_installation/​on_prem/​external_kafka_cluster.md:147

This new cross-reference uses a relative Markdown file path rather than the heading-ID syntax required by the repository's linking guidelines. Use [Data Source usage guide][streaming-based-connector] to follow the versioned documentation's linking convention.

1 resolved since last review

• Use feature_store instead of undefined fs

_{Review effort: Balanced}