Skip to content

feat: generate prometheus-additional.yaml for Doris FE and BE - #558

Open
jiuker wants to merge 1 commit into
apache:masterfrom
jiuker:feat/issue-334-prometheus-additional
Open

jiuker wants to merge 1 commit into
apache:masterfrom
jiuker:feat/issue-334-prometheus-additional

Conversation

@jiuker

@jiuker jiuker commented Nov 21, 2026

Copy link
Copy Markdown

What problem does this PR solve?

Issue Number: close #334

Related PR: #xxx

Problem Summary:

Doris exposes Prometheus metrics on FE http_port (/metrics) and BE webserver_port (/metrics) / brpc_port (/brpc_metrics). prometheus-operator users normally describe this with a ServiceMonitor, but creating one from the operator requires the monitoring.coreos.com CRD to be installed first, which is not available on every Doris installation.

This PR avoids that dependency by letting the operator generate the raw scrape config instead:

  • add spec.enableServiceMonitor to DorisCluster (default false);
  • when enabled, create Secret <cluster>-prometheus-additional with key prometheus-additional.yaml;
  • the content is a scrape_configs list that can be referenced by spec.additionalScrapeConfigs of a Prometheus CR;
  • the Secret is owned by the DorisCluster and is deleted when the switch is disabled.

Generated jobs:

job service port path
<ns>/<cluster>/fe <cluster>-fe-service http-port /metrics
<ns>/<cluster>/be <cluster>-be-service webserver-port /metrics
<ns>/<cluster>/be-brpc <cluster>-be-service brpc-port /brpc_metrics

The ports are matched by Kubernetes Service port name (not by number), so custom http_port / webserver_port / brpc_port values in fe.conf / be.conf keep working.

Release note

Add spec.enableServiceMonitor to generate prometheus-additional.yaml for Doris FE/BE metrics.

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes. New optional field (default false); the operator now requires secrets write permission.
  • Does this need documentation?

    • No. The field is documented in the CRD and usage is described in this PR.
    • Yes.

Manual test

Environment: kind cluster, Kubernetes v1.32.2, namespace doris, cluster doriscluster-helm (3 FE, 1 BE).

  1. Enable the switch in DorisCluster:
spec:
  enableServiceMonitor: true
  1. The operator creates the Secret:
$ kubectl -n doris get secret doriscluster-helm-prometheus-additional
NAME                                      TYPE     DATA   AGE
doriscluster-helm-prometheus-additional   Opaque   1      3m

$ kubectl -n doris get secret doriscluster-helm-prometheus-additional \
    -o jsonpath='{.data.prometheus-additional\.yaml}' | base64 -d
- job_name: doris/doriscluster-helm/fe
  metrics_path: /metrics
  kubernetes_sd_configs:
    - role: endpoints
      namespaces:
        names:
          - doris
  relabel_configs:
    - source_labels: [__meta_kubernetes_service_name]
      regex: doriscluster-helm-fe-service
      action: keep
    - source_labels: [__meta_kubernetes_endpoint_port_name]
      regex: http-port
      action: keep
    - source_labels: [__meta_kubernetes_namespace]
      target_label: namespace
      action: replace
    - source_labels: [__meta_kubernetes_pod_name]
      target_label: pod
      action: replace
- job_name: doris/doriscluster-helm/be
  metrics_path: /metrics
  kubernetes_sd_configs:
    - role: endpoints
      namespaces:
        names:
          - doris
  relabel_configs:
    - source_labels: [__meta_kubernetes_service_name]
      regex: doriscluster-helm-be-service
      action: keep
    - source_labels: [__meta_kubernetes_endpoint_port_name]
      regex: webserver-port
      action: keep
    - source_labels: [__meta_kubernetes_namespace]
      target_label: namespace
      action: replace
    - source_labels: [__meta_kubernetes_pod_name]
      target_label: pod
      action: replace
- job_name: doris/doriscluster-helm/be-brpc
  metrics_path: /brpc_metrics
  kubernetes_sd_configs:
    - role: endpoints
      namespaces:
        names:
          - doris
  relabel_configs:
    - source_labels: [__meta_kubernetes_service_name]
      regex: doriscluster-helm-be-service
      action: keep
    - source_labels: [__meta_kubernetes_endpoint_port_name]
      regex: brpc-port
      action: keep
    - source_labels: [__meta_kubernetes_namespace]
      target_label: namespace
      action: replace
    - source_labels: [__meta_kubernetes_pod_name]
      target_label: pod
      action: replace
  1. Port names match the live Services:
job Secret port regex Service port name Service port EndpointSlice port name
fe http-port http-port 8030 http-port
be webserver-port webserver-port 8040 webserver-port
be-brpc brpc-port brpc-port 8060 brpc-port

The internal Services (*-fe-internal, *-be-internal) only expose query-port / heartbeat-port, and are excluded by the Service name filter.

  1. curl verification.

FE /metrics (via kubectl port-forward svc/doriscluster-helm-fe-service 18130:8030):

HTTP/1.1 200 OK
Content-Type: text/plain;charset=iso-8859-1
Content-Length: 27402
# HELP  jvm_heap_size_bytes jvm heap stat
# TYPE  jvm_heap_size_bytes gauge
jvm_heap_size_bytes{type="max"} 3435134976
jvm_heap_size_bytes{type="committed"} 3435134976
jvm_heap_size_bytes{type="used"} 952632552
...
doris_fe_query_total 11
doris_fe_query_total{user="root"} 11
doris_fe_report_queue_size 0
doris_fe_thrift_rpc_total{method="finishTask"} 22

476 lines, 56 metric families.

BE /metrics (via kubectl port-forward svc/doriscluster-helm-be-service 18140:8040):

HTTP/1.1 200 OK
Content-Type: text/plain; version=0.0.4
Content-Length: 32968
# TYPE doris_be_disks_avail_capacity gauge
doris_be_disks_avail_capacity{path="/opt/apache-doris/be/storage"} 278061080576
# TYPE doris_be_disks_local_used_capacity gauge
doris_be_disks_local_used_capacity{path="/opt/apache-doris/be/storage"} 0
# TYPE doris_be_disks_compaction_score gauge
doris_be_disks_compaction_score{path="/opt/apache-doris/be/storage"} 0
...
doris_be_streaming_load_current_processing 0
doris_be_http_stream_current_processing 0

682 lines, 169 metric families.

BE /brpc_metrics (via kubectl port-forward svc/doriscluster-helm-be-service 18160:8060):

HTTP/1.1 200 OK
Content-Length: 131681
Content-Type: text/plain
# HELP bthread_count
# TYPE bthread_count gauge
bthread_count 4
# HELP bthread_key_count
# TYPE bthread_key_count gauge
bthread_key_count 2
...
process_cpu_usage 0.052
process_cpu_usage_system 0.025
process_cpu_usage_user 0.027
process_context_switches_voluntary_second 2409

2539 lines, 739 metric families.

Per Pod checks:

doriscluster-helm-fe-0 /metrics       200
doriscluster-helm-fe-1 /metrics       200
doriscluster-helm-fe-2 /metrics       200
doriscluster-helm-be-0 /metrics       200
doriscluster-helm-be-0 /brpc_metrics  200

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

Add spec.enableServiceMonitor to DorisCluster. When enabled, the operator
creates a Secret named <cluster>-prometheus-additional with the key
prometheus-additional.yaml, containing scrape_configs for FE /metrics and
BE /metrics and /brpc_metrics. The Secret is owned by the DorisCluster and
is removed when the switch is disabled.

- add the scrape config builder and unit tests
- reconcile the Secret from the DorisCluster controller
- grant the operator secrets write RBAC
- regenerate CRDs and expose the field in the Doris helm chart

Closes apache#334
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] support monitor for doris

1 participant