Skip to content

TASK: Collect Kubernetes logs with Fluent Bit #94

Closed
digital-bank-java/infra-sit
#38

Description

@ramioooz

Parent: STORY: Implement centralized logging with OpenSearch

Collect service logs from Kubernetes stdout and stderr using Fluent Bit and route them to OpenSearch.

Acceptance criteria:

  • Fluent Bit runs as a Kubernetes DaemonSet.
  • Kubernetes metadata is attached to log events.
  • Structured JSON events are parsed according to the logging field contract.
  • Sensitive fields are filtered before they reach OpenSearch.
  • Buffering, retries, backpressure, and invalid-event handling are explicit and observable.
  • Delivery is tested using at least two platform services.
  • The output configuration can be changed from SIT OpenSearch to Amazon OpenSearch Service without application changes.

Activity

  1. added theissue type on Aug 2, 2026
  2. self-assigned this
    on Aug 30, 2026
  3. ramioooz commented on Sep 5, 2026

    @ramioooz
    MemberAuthor

    SIT verification checkpoint (2026-09-05). Fluent Bit DaemonSet is deployed and Ready 1/1. OpenSearch is healthy and the output plugin reports zero errors. Fluent Bit input and processed-output counters remain zero. A read-only node diagnostic found no files under the standard /var/log/containers or /var/log/pods paths exposed to the DaemonSet on Docker Desktop Kubernetes. The task remains open: this is a deployment-source-path limitation, not successful log-collection evidence. Do not mark Sprint 6 observability complete until the supported local SIT log source is available or the task scope explicitly records this platform limitation.

  4. ramioooz commented on Sep 5, 2026

    @ramioooz
    MemberAuthor

    Validation update (2026-09-06)

    The merged infrastructure checks pass:

    • bash tests/validate-fluent-bit.sh: chart lint/render and redaction validation passed.
    • bash tests/validate.sh: OpenSearch chart validation passed.
    • Fluent Bit health endpoint returned ok in digital-bank-sit.
    • Current Prometheus counters: tail.0 input_records=0, storage_backlog.1 input_records=0, opensearch.0 output_proc_records=0, output_errors=0, and output_retried_records=0.

    The collector is healthy and configured, but the zero input/output counters mean the delivery acceptance criterion is still unproven on Docker Desktop. Keep this task open until a supported local node log source is available and records from at least two services are searchable in OpenSearch.

  5. ramioooz commented on Sep 5, 2026

    @ramioooz
    MemberAuthor

    Node log-source probe (2026-09-06)

    A short read-only probe confirmed the Docker Desktop node exposes:

    • /var/log/containers and /var/log/pods directories, but no readable log files to the Fluent Bit DaemonSet;
    • Docker JSON log files under /var/lib/docker/containers//*-json.log;
    • Docker container configuration labels include Kubernetes pod, namespace, container, and log-path metadata.

    This confirms that raw Docker JSON collection is technically possible, but a path-only Fluent Bit input would lose the required Kubernetes metadata and structured-log enrichment. No partial collector was added. A future implementation must preserve the Docker-ID-to-Kubernetes-metadata mapping, or the local SIT runtime must expose standard Kubernetes log symlinks. The existing Fluent Bit acceptance remains open.

  6. ramioooz commented on Sep 5, 2026

    @ramioooz
    MemberAuthor

    Implemented in infra-sit PR #38. The Docker Desktop SIT fallback is disabled by default, enabled only through values-sit.yaml, mounts /var/lib/docker/containers read-only, extracts only selected Kubernetes labels from config.v2.json, and redacts structured, unstructured, and invalid records. Helm, existing validators, the Docker-mode Fluent Bit fixture, and CI pass. The container-ID matcher was corrected for the actual *-json.log tail tag and the fixture was rerun successfully. Keep this task open until the PR is merged and deployed SIT/OpenSearch evidence is repeated.

  7. ramioooz commented on Sep 6, 2026

    @ramioooz
    MemberAuthor

    SIT runtime evidence collected 2026-09-06 after the Docker Desktop fallback rollout: the Fluent Bit pod is Ready and the fluent-bit Helm release is deployed. Prometheus counters showed tail.0 (CRI) 372 records and tail.1 (Docker JSON) 397 records; OpenSearch outputs processed the same counts with zero errors, zero retries, and zero failed retries. An authenticated OpenSearch query found 1,749 logging_source=docker-json documents, including selected Kubernetes pod_name, namespace_name, container_name, pod_uid, and log_path metadata. The PR remains open for review; repeat this evidence after merge before closing the task.

  8. ramioooz commented on Sep 6, 2026

    @ramioooz
    MemberAuthor

    Post-merge SIT evidence (2026-09-06)

    • The merged infra-sit#38 Docker Desktop fallback is deployed in digital-bank-sit; the Fluent Bit DaemonSet is 1/1 Ready and the Helm release is deployed.
    • Prometheus metrics show CRI and Docker JSON inputs are active. OpenSearch outputs report zero output errors and zero failed retries. The Docker output has transient retry/backpressure activity while draining the existing node-log backlog, with no dropped records or abandoned batches.
    • A read-only authenticated OpenSearch query returned at least 10,000 logging_source=docker-json records retaining logging_agent=fluent-bit and kubernetes.namespace_name=digital-bank-sit; the selected pod, container, UID, and log-path metadata are present in the indexed documents. No secret values or raw credentials were recorded.

    This completes the runtime evidence for the already-closed task; the parent centralized-logging story remains governed by its remaining scope.

  9. ramioooz commented on Sep 6, 2026

    @ramioooz
    MemberAuthor

    Status update - 2026-09-06

    The existing Fluent Bit deployment exposed an OpenSearch TLS transport mismatch in SIT. Infra PR #40 is open for review: digital-bank-java/infra-sit#40

    The PR renders the existing TLS settings as canonical On/Off values, preserves the admin Secret wiring, and extends the existing validator. The task is In review pending merge and SIT rollout verification.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions