Skip to content

Allow running the parquet converter as a StatefulSet - #685

Merged
kd7lxl merged 1 commit into
masterfrom
parquet-converter-statefulset
Oct 27, 2026
Merged

kd7lxl merged 1 commit into
masterfrom
parquet-converter-statefulset

Conversation

@friedrichg

Copy link
Copy Markdown
Member

What this PR does:

Adds parquet_converter.statefulSet.enabled, the same switch as ingester.statefulSet.enabled, so that the parquet converter can run as a StatefulSet with a persistent volume per pod.

The converter downloads each block into its data_dir before it converts it, and it wipes <data_dir>/compact before each download, so replicas must not share a volume. As a Deployment it has no volume of its own: the container runs with readOnlyRootFilesystem: true, so the default data_dir (./data) cannot be written unless extraVolumes adds one, and a generic ephemeral volume there is created and deleted with every pod.

With parquet_converter.statefulSet.enabled: true:

  • The converter is a StatefulSet (templates/parquet-converter/parquet-converter-statefulset.yaml) and the Deployment is not rendered.
  • Each pod gets a Persistent Volume Claim named storage from parquet_converter.persistentVolume (size, storageClass, accessModes, annotations, subPath), or an emptyDir when persistentVolume.enabled is false, mounted at /data. The image's working directory is /, so the default data_dir ./data is on that volume with no config change.
  • The claim belongs to the pod's ordinal, so a restart, a rollout or an eviction keeps the same volume.
  • persistentVolume.retentionPolicy sets persistentVolumeClaimRetentionPolicy (Kubernetes 1.23 and later), as for the ingester, store-gateway and compactor.
  • statefulSet.podManagementPolicy defaults to Parallel, because the converter ring does not need ordered starts. statefulStrategy is the StatefulSet's updateStrategy.

The default stays a Deployment, so nothing changes for existing users.

Which issue(s) this PR fixes:
None.

Testing:

  • helm template with the default values (Deployment), with statefulSet.enabled (StatefulSet with a storage claim mounted at /data), with persistentVolume.enabled: false (emptyDir), and with retentionPolicy, storageClass: "-" and subPath set: each renders as expected.
  • kubectl apply --dry-run=server of the rendered StatefulSet on kind (Kubernetes v1.35.0): accepted.
  • README regenerated with helm-docs v1.11.0.
  • The CI install test values do not enable the parquet converter, so CI covers the new template through ct lint only.

Checklist

  • CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX], [DEPENDENCY]

@friedrichg
friedrichg force-pushed the parquet-converter-statefulset branch from 501999f to 385c97f Compare October 22, 2026 23:50
@SungJin1212

Copy link
Copy Markdown
Member

Should we mount an emptyDir to the Deployment as a default?

@kd7lxl

kd7lxl commented Oct 26, 2026

Copy link
Copy Markdown
Collaborator

Can you rebase this?

The parquet converter downloads each block into its data_dir before it
converts it, and it wipes <data_dir>/compact before each download, so
replicas must not share a volume. As a Deployment it has no volume of
its own: the container runs with readOnlyRootFilesystem, so the default
data_dir (./data) cannot be written unless extraVolumes adds one.

Add parquet_converter.statefulSet.enabled, the same switch as
ingester.statefulSet.enabled. When it is on, the converter is a
StatefulSet with one Persistent Volume Claim per pod from
parquet_converter.persistentVolume (or an emptyDir when that is off),
mounted at /data. The image's working directory is /, so the default
data_dir ./data is on that volume with no config change. The claim keeps
the same volume across restarts and rollouts, and
persistentVolume.retentionPolicy controls the claims on scale-down and
on deletion, as for the ingester, store-gateway and compactor.

The default stays a Deployment, so nothing changes for existing users.

Signed-off-by: Friedrich Gonzalez <1517449+friedrichg@users.noreply.github.com>
@friedrichg
friedrichg force-pushed the parquet-converter-statefulset branch from 385c97f to ec10406 Compare October 27, 2026 01:21
@kd7lxl
kd7lxl enabled auto-merge (squash) October 27, 2026 01:26
@kd7lxl
kd7lxl merged commit 1beabb1 into master Oct 27, 2026
3 checks passed
@kd7lxl
kd7lxl deleted the parquet-converter-statefulset branch October 27, 2026 01:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants