Repository navigation
Conversation
Add spec.enableServiceMonitor to DorisCluster. When enabled, the operator creates a Secret named <cluster>-prometheus-additional with the key prometheus-additional.yaml, containing scrape_configs for FE /metrics and BE /metrics and /brpc_metrics. The Secret is owned by the DorisCluster and is removed when the switch is disabled. - add the scrape config builder and unit tests - reconcile the Secret from the DorisCluster controller - grant the operator secrets write RBAC - regenerate CRDs and expose the field in the Doris helm chart Closes apache#334
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What problem does this PR solve?
Issue Number: close #334
Related PR: #xxx
Problem Summary:
Doris exposes Prometheus metrics on FE
http_port(/metrics) and BEwebserver_port(/metrics) /brpc_port(/brpc_metrics). prometheus-operator users normally describe this with aServiceMonitor, but creating one from the operator requires themonitoring.coreos.comCRD to be installed first, which is not available on every Doris installation.This PR avoids that dependency by letting the operator generate the raw scrape config instead:
spec.enableServiceMonitortoDorisCluster(defaultfalse);<cluster>-prometheus-additionalwith keyprometheus-additional.yaml;scrape_configslist that can be referenced byspec.additionalScrapeConfigsof a Prometheus CR;DorisClusterand is deleted when the switch is disabled.Generated jobs:
<ns>/<cluster>/fe<cluster>-fe-servicehttp-port/metrics<ns>/<cluster>/be<cluster>-be-servicewebserver-port/metrics<ns>/<cluster>/be-brpc<cluster>-be-servicebrpc-port/brpc_metricsThe ports are matched by Kubernetes Service port name (not by number), so custom
http_port/webserver_port/brpc_portvalues infe.conf/be.confkeep working.Release note
Add
spec.enableServiceMonitorto generateprometheus-additional.yamlfor Doris FE/BE metrics.Check List (For Author)
Test
Behavior changed:
false); the operator now requires secrets write permission.Does this need documentation?
Manual test
Environment: kind cluster, Kubernetes v1.32.2, namespace
doris, clusterdoriscluster-helm(3 FE, 1 BE).DorisCluster:$ kubectl -n doris get secret doriscluster-helm-prometheus-additional NAME TYPE DATA AGE doriscluster-helm-prometheus-additional Opaque 1 3m $ kubectl -n doris get secret doriscluster-helm-prometheus-additional \ -o jsonpath='{.data.prometheus-additional\.yaml}' | base64 -d - job_name: doris/doriscluster-helm/fe metrics_path: /metrics kubernetes_sd_configs: - role: endpoints namespaces: names: - doris relabel_configs: - source_labels: [__meta_kubernetes_service_name] regex: doriscluster-helm-fe-service action: keep - source_labels: [__meta_kubernetes_endpoint_port_name] regex: http-port action: keep - source_labels: [__meta_kubernetes_namespace] target_label: namespace action: replace - source_labels: [__meta_kubernetes_pod_name] target_label: pod action: replace - job_name: doris/doriscluster-helm/be metrics_path: /metrics kubernetes_sd_configs: - role: endpoints namespaces: names: - doris relabel_configs: - source_labels: [__meta_kubernetes_service_name] regex: doriscluster-helm-be-service action: keep - source_labels: [__meta_kubernetes_endpoint_port_name] regex: webserver-port action: keep - source_labels: [__meta_kubernetes_namespace] target_label: namespace action: replace - source_labels: [__meta_kubernetes_pod_name] target_label: pod action: replace - job_name: doris/doriscluster-helm/be-brpc metrics_path: /brpc_metrics kubernetes_sd_configs: - role: endpoints namespaces: names: - doris relabel_configs: - source_labels: [__meta_kubernetes_service_name] regex: doriscluster-helm-be-service action: keep - source_labels: [__meta_kubernetes_endpoint_port_name] regex: brpc-port action: keep - source_labels: [__meta_kubernetes_namespace] target_label: namespace action: replace - source_labels: [__meta_kubernetes_pod_name] target_label: pod action: replacehttp-porthttp-porthttp-portwebserver-portwebserver-portwebserver-portbrpc-portbrpc-portbrpc-portThe internal Services (
*-fe-internal,*-be-internal) only exposequery-port/heartbeat-port, and are excluded by the Service name filter.FE
/metrics(viakubectl port-forward svc/doriscluster-helm-fe-service 18130:8030):476 lines, 56 metric families.
BE
/metrics(viakubectl port-forward svc/doriscluster-helm-be-service 18140:8040):682 lines, 169 metric families.
BE
/brpc_metrics(viakubectl port-forward svc/doriscluster-helm-be-service 18160:8060):2539 lines, 739 metric families.
Per Pod checks:
Check List (For Reviewer who merge this PR)