Repository navigation
Allow running the parquet converter as a StatefulSet - #685
Merged
Merged
Conversation
friedrichg
force-pushed
the
parquet-converter-statefulset
branch
from
October 22, 2026 23:50
501999f to
385c97f
Compare
Member
|
Should we mount an |
kd7lxl
approved these changes
Oct 23, 2026
Collaborator
|
Can you rebase this? |
The parquet converter downloads each block into its data_dir before it converts it, and it wipes <data_dir>/compact before each download, so replicas must not share a volume. As a Deployment it has no volume of its own: the container runs with readOnlyRootFilesystem, so the default data_dir (./data) cannot be written unless extraVolumes adds one. Add parquet_converter.statefulSet.enabled, the same switch as ingester.statefulSet.enabled. When it is on, the converter is a StatefulSet with one Persistent Volume Claim per pod from parquet_converter.persistentVolume (or an emptyDir when that is off), mounted at /data. The image's working directory is /, so the default data_dir ./data is on that volume with no config change. The claim keeps the same volume across restarts and rollouts, and persistentVolume.retentionPolicy controls the claims on scale-down and on deletion, as for the ingester, store-gateway and compactor. The default stays a Deployment, so nothing changes for existing users. Signed-off-by: Friedrich Gonzalez <1517449+friedrichg@users.noreply.github.com>
friedrichg
force-pushed
the
parquet-converter-statefulset
branch
from
October 27, 2026 01:21
385c97f to
ec10406
Compare
kd7lxl
enabled auto-merge (squash)
October 27, 2026 01:26
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this PR does:
Adds
parquet_converter.statefulSet.enabled, the same switch asingester.statefulSet.enabled, so that the parquet converter can run as a StatefulSet with a persistent volume per pod.The converter downloads each block into its
data_dirbefore it converts it, and it wipes<data_dir>/compactbefore each download, so replicas must not share a volume. As a Deployment it has no volume of its own: the container runs withreadOnlyRootFilesystem: true, so the defaultdata_dir(./data) cannot be written unlessextraVolumesadds one, and a generic ephemeral volume there is created and deleted with every pod.With
parquet_converter.statefulSet.enabled: true:templates/parquet-converter/parquet-converter-statefulset.yaml) and the Deployment is not rendered.storagefromparquet_converter.persistentVolume(size,storageClass,accessModes,annotations,subPath), or anemptyDirwhenpersistentVolume.enabledis false, mounted at/data. The image's working directory is/, so the defaultdata_dir./datais on that volume with no config change.persistentVolume.retentionPolicysetspersistentVolumeClaimRetentionPolicy(Kubernetes 1.23 and later), as for the ingester, store-gateway and compactor.statefulSet.podManagementPolicydefaults toParallel, because the converter ring does not need ordered starts.statefulStrategyis the StatefulSet'supdateStrategy.The default stays a Deployment, so nothing changes for existing users.
Which issue(s) this PR fixes:
None.
Testing:
helm templatewith the default values (Deployment), withstatefulSet.enabled(StatefulSet with astorageclaim mounted at/data), withpersistentVolume.enabled: false(emptyDir), and withretentionPolicy,storageClass: "-"andsubPathset: each renders as expected.kubectl apply --dry-run=serverof the rendered StatefulSet on kind (Kubernetes v1.35.0): accepted.ct lintonly.Checklist
CHANGELOG.mdupdated - the order of entries should be[CHANGE],[FEATURE],[ENHANCEMENT],[BUGFIX],[DEPENDENCY]