You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
SIT verification checkpoint (2026-09-05). Fluent Bit DaemonSet is deployed and Ready 1/1. OpenSearch is healthy and the output plugin reports zero errors. Fluent Bit input and processed-output counters remain zero. A read-only node diagnostic found no files under the standard /var/log/containers or /var/log/pods paths exposed to the DaemonSet on Docker Desktop Kubernetes. The task remains open: this is a deployment-source-path limitation, not successful log-collection evidence. Do not mark Sprint 6 observability complete until the supported local SIT log source is available or the task scope explicitly records this platform limitation.
Fluent Bit health endpoint returned ok in digital-bank-sit.
Current Prometheus counters: tail.0 input_records=0, storage_backlog.1 input_records=0, opensearch.0 output_proc_records=0, output_errors=0, and output_retried_records=0.
The collector is healthy and configured, but the zero input/output counters mean the delivery acceptance criterion is still unproven on Docker Desktop. Keep this task open until a supported local node log source is available and records from at least two services are searchable in OpenSearch.
A short read-only probe confirmed the Docker Desktop node exposes:
/var/log/containers and /var/log/pods directories, but no readable log files to the Fluent Bit DaemonSet;
Docker JSON log files under /var/lib/docker/containers//*-json.log;
Docker container configuration labels include Kubernetes pod, namespace, container, and log-path metadata.
This confirms that raw Docker JSON collection is technically possible, but a path-only Fluent Bit input would lose the required Kubernetes metadata and structured-log enrichment. No partial collector was added. A future implementation must preserve the Docker-ID-to-Kubernetes-metadata mapping, or the local SIT runtime must expose standard Kubernetes log symlinks. The existing Fluent Bit acceptance remains open.
Implemented in infra-sit PR #38. The Docker Desktop SIT fallback is disabled by default, enabled only through values-sit.yaml, mounts /var/lib/docker/containers read-only, extracts only selected Kubernetes labels from config.v2.json, and redacts structured, unstructured, and invalid records. Helm, existing validators, the Docker-mode Fluent Bit fixture, and CI pass. The container-ID matcher was corrected for the actual *-json.log tail tag and the fixture was rerun successfully. Keep this task open until the PR is merged and deployed SIT/OpenSearch evidence is repeated.
SIT runtime evidence collected 2026-09-06 after the Docker Desktop fallback rollout: the Fluent Bit pod is Ready and the fluent-bit Helm release is deployed. Prometheus counters showed tail.0 (CRI) 372 records and tail.1 (Docker JSON) 397 records; OpenSearch outputs processed the same counts with zero errors, zero retries, and zero failed retries. An authenticated OpenSearch query found 1,749 logging_source=docker-json documents, including selected Kubernetes pod_name, namespace_name, container_name, pod_uid, and log_path metadata. The PR remains open for review; repeat this evidence after merge before closing the task.
The merged infra-sit#38 Docker Desktop fallback is deployed in digital-bank-sit; the Fluent Bit DaemonSet is 1/1 Ready and the Helm release is deployed.
Prometheus metrics show CRI and Docker JSON inputs are active. OpenSearch outputs report zero output errors and zero failed retries. The Docker output has transient retry/backpressure activity while draining the existing node-log backlog, with no dropped records or abandoned batches.
A read-only authenticated OpenSearch query returned at least 10,000 logging_source=docker-json records retaining logging_agent=fluent-bit and kubernetes.namespace_name=digital-bank-sit; the selected pod, container, UID, and log-path metadata are present in the indexed documents. No secret values or raw credentials were recorded.
This completes the runtime evidence for the already-closed task; the parent centralized-logging story remains governed by its remaining scope.
The existing Fluent Bit deployment exposed an OpenSearch TLS transport mismatch in SIT. Infra PR #40 is open for review: digital-bank-java/infra-sit#40
The PR renders the existing TLS settings as canonical On/Off values, preserves the admin Secret wiring, and extends the existing validator. The task is In review pending merge and SIT rollout verification.
Activity
ramioooz commentedon Sep 5, 2026
SIT verification checkpoint (2026-09-05). Fluent Bit DaemonSet is deployed and Ready 1/1. OpenSearch is healthy and the output plugin reports zero errors. Fluent Bit input and processed-output counters remain zero. A read-only node diagnostic found no files under the standard /var/log/containers or /var/log/pods paths exposed to the DaemonSet on Docker Desktop Kubernetes. The task remains open: this is a deployment-source-path limitation, not successful log-collection evidence. Do not mark Sprint 6 observability complete until the supported local SIT log source is available or the task scope explicitly records this platform limitation.
ramioooz commentedon Sep 5, 2026
Validation update (2026-09-06)
The merged infrastructure checks pass:
bash tests/validate-fluent-bit.sh: chart lint/render and redaction validation passed.bash tests/validate.sh: OpenSearch chart validation passed.okindigital-bank-sit.tail.0 input_records=0,storage_backlog.1 input_records=0,opensearch.0 output_proc_records=0,output_errors=0, andoutput_retried_records=0.The collector is healthy and configured, but the zero input/output counters mean the delivery acceptance criterion is still unproven on Docker Desktop. Keep this task open until a supported local node log source is available and records from at least two services are searchable in OpenSearch.
ramioooz commentedon Sep 5, 2026
Node log-source probe (2026-09-06)
A short read-only probe confirmed the Docker Desktop node exposes:
This confirms that raw Docker JSON collection is technically possible, but a path-only Fluent Bit input would lose the required Kubernetes metadata and structured-log enrichment. No partial collector was added. A future implementation must preserve the Docker-ID-to-Kubernetes-metadata mapping, or the local SIT runtime must expose standard Kubernetes log symlinks. The existing Fluent Bit acceptance remains open.
ramioooz commentedon Sep 5, 2026
Implemented in infra-sit PR #38. The Docker Desktop SIT fallback is disabled by default, enabled only through
values-sit.yaml, mounts/var/lib/docker/containersread-only, extracts only selected Kubernetes labels fromconfig.v2.json, and redacts structured, unstructured, and invalid records. Helm, existing validators, the Docker-mode Fluent Bit fixture, and CI pass. The container-ID matcher was corrected for the actual*-json.logtail tag and the fixture was rerun successfully. Keep this task open until the PR is merged and deployed SIT/OpenSearch evidence is repeated.ramioooz commentedon Sep 6, 2026
SIT runtime evidence collected 2026-09-06 after the Docker Desktop fallback rollout: the Fluent Bit pod is Ready and the
fluent-bitHelm release is deployed. Prometheus counters showedtail.0(CRI) 372 records andtail.1(Docker JSON) 397 records; OpenSearch outputs processed the same counts with zero errors, zero retries, and zero failed retries. An authenticated OpenSearch query found 1,749logging_source=docker-jsondocuments, including selected Kubernetespod_name,namespace_name,container_name,pod_uid, andlog_pathmetadata. The PR remains open for review; repeat this evidence after merge before closing the task.ramioooz commentedon Sep 6, 2026
Post-merge SIT evidence (2026-09-06)
infra-sit#38Docker Desktop fallback is deployed indigital-bank-sit; the Fluent Bit DaemonSet is1/1Ready and the Helm release isdeployed.logging_source=docker-jsonrecords retaininglogging_agent=fluent-bitandkubernetes.namespace_name=digital-bank-sit; the selected pod, container, UID, and log-path metadata are present in the indexed documents. No secret values or raw credentials were recorded.This completes the runtime evidence for the already-closed task; the parent centralized-logging story remains governed by its remaining scope.
ramioooz commentedon Sep 6, 2026
Status update - 2026-09-06
The existing Fluent Bit deployment exposed an OpenSearch TLS transport mismatch in SIT. Infra PR #40 is open for review: digital-bank-java/infra-sit#40
The PR renders the existing TLS settings as canonical On/Off values, preserves the admin Secret wiring, and extends the existing validator. The task is In review pending merge and SIT rollout verification.