Skip to content

TASK: Add local SIT distributed tracing with Zipkin #242

Closed

Description

@ramioooz

Objective

Add a local-SIT distributed tracing baseline for the HTTP platform path so operators can correlate one request across API Gateway and downstream Spring services.

Parent

  • Parent story: #64
  • Sprint: Sprint 6 - Operational Resilience and Observability

Scope

  • Add the Micrometer tracing bridge and Zipkin reporter only to services that participate in the HTTP request path.
  • Add configurable SIT Zipkin endpoint and sampling settings through config-repo.
  • Add a local SIT Zipkin deployment and ClusterIP service in infra-sit.
  • Preserve existing correlation-ID behavior and keep trace/span identifiers out of business payloads.
  • Document local verification and explicitly defer AWS X-Ray/OpenTelemetry Collector mapping.

Acceptance Criteria

  • API Gateway and participating HTTP services emit trace spans when tracing is enabled.
  • SIT sends spans to an in-cluster Zipkin endpoint without public exposure.
  • Sampling and endpoint settings are environment-configurable and disabled or safely bounded outside SIT.
  • A controlled gateway request can be correlated by trace ID across at least two services.
  • Helm lint/template, affected Maven verification, and a focused smoke check pass.
  • No new CI workflow or broad redundant unit-test suite is introduced.

Dependencies and Rollout Notes

Activity

  1. self-assigned this
    on Sep 5, 2026
  2. added theissue type on Sep 5, 2026
  3. ramioooz commented on Sep 5, 2026

    @ramioooz
    MemberAuthor

    Zipkin tracing implementation PRs\n\nOpened the implementation as separate reviewable PRs:\n\n- infra-sit#37: local SIT Zipkin 3.6.1 ClusterIP deployment with in-memory storage\n- config-repo#45: shared SIT exporter endpoint, sampling, and safe default\n- api-gateway#25: managed Spring Boot Zipkin starter\n- customer-service#36: managed Spring Boot Zipkin starter\n- account-service#41: managed Spring Boot Zipkin starter\n- transaction-service#23: managed Spring Boot Zipkin starter\n- mfa-service#16: managed Spring Boot Zipkin starter\n- ledger-service#21: managed Spring Boot Zipkin starter\n\nLocal verification before PR creation:\n- Helm lint/template passed for the Zipkin chart.\n- Existing test suites passed: Gateway 20, Customer 7, Account 64, Transaction 89, MFA 46, Ledger 27.\n- No new tests or CI workflows were added.\n\nMerge order:\n1. infra-sit#37\n2. config-repo#45\n3. service PRs in any order: api-gateway#25, customer-service#36, account-service#41, transaction-service#23, mfa-service#16, ledger-service#21\n4. Roll out Zipkin and the rebuilt services in SIT, then verify one gateway request across at least two services.\n\nThe tracking item remains In progress until the SIT trace-correlation evidence is captured.

  4. ramioooz commented on Sep 5, 2026

    @ramioooz
    MemberAuthor

    The handoff record was refreshed in .github PR #241 with the tracing scope, PR links, verification counts, and post-merge SIT evidence required before closure.

  5. ramioooz commented on Sep 5, 2026

    @ramioooz
    MemberAuthor

    Follow-up hardening was pushed to the six service PRs: the Zipkin exporter is disabled by default for standalone/test execution and is enabled only by the shared SIT profile. Existing suites were rerun successfully with the same counts; no new tests or CI jobs were added.

  6. ramioooz commented on Sep 5, 2026

    @ramioooz
    MemberAuthor

    All implementation PR CI is now green and every PR is open, non-draft, and mergeable. The Zipkin PR set is ready for review. After merge, rollout order remains infra-sit#37, config-repo#45, then the six service PRs, followed by SIT trace-correlation verification.

  7. ramioooz commented on Sep 5, 2026

    @ramioooz
    MemberAuthor

    Tracing implementation wave is complete in reviewable PRs. The existing task remains In progress until the PRs are merged and SIT correlation is verified.

    New service coverage:

    Existing tracing wave:

    All new service PRs passed local ./mvnw verify, Helm lint/template validation, and git diff --check. GitHub Maven checks are still pending; Helm checks are green. No redundant tests, CI jobs, routes, or business behavior were added.

    After merge, deploy the Zipkin backend/config and all instrumented services to SIT, exercise one gateway-to-service request, and verify the trace appears in Zipkin before closing this task.

  8. ramioooz commented on Sep 6, 2026

    @ramioooz
    MemberAuthor

    SIT verification update (2026-09-06)

    • Zipkin candidate chart is deployed and healthy in digital-bank-sit; /health reports UP.
    • Rebuilt merged API Gateway and Auth Service tracing images from origin/main with temporary local tags and deployed them to SIT.
    • A controlled gateway login produced a single trace containing spans from both api-gateway and auth-service; Zipkin service discovery lists both.
    • Rollout exposed a chart security-context defect because the image user is symbolic; infra-sit PR #39 sets numeric UID/GID 1000 while retaining runAsNonRoot.
    • .github PR #245 records this evidence in the handoff.

    The task remains open until PR #39 is reviewed/merged and the trace is repeated with the normal release tags.

  9. ramioooz commented on Sep 6, 2026

    @ramioooz
    MemberAuthor

    2026-09-06 candidate verification update:

  10. ramioooz commented on Sep 6, 2026

    @ramioooz
    MemberAuthor

    Post-merge SIT verification (2026-09-06)

    • Merged infra-sit#39 was deployed successfully; Zipkin is 1/1 Ready with zero restarts.
    • Zipkin /health reports UP and its in-memory storage is healthy.
    • Zipkin service discovery currently includes account-service, api-gateway, auth-service, and transaction-service.
    • The normal release-tag deployment is therefore healthy after the numeric UID/GID correction.

    The task remains open until one fresh gateway request is correlated across at least two service spans after the merged release rollout. AWS tracing and collector mapping remain deferred as documented.

  11. ramioooz commented on Sep 6, 2026

    @ramioooz
    MemberAuthor

    Fresh post-merge trace evidence (2026-09-06)

    • Sent GET /account-service/actuator/health through the SIT API Gateway after deploying the merged release tags.
    • Response was 200 OK; Account Service reported both liveness and readiness groups UP.
    • Zipkin trace ID: 6a9d60564d687c479f9c8a6dc06da9fe.
    • The trace contains the API Gateway server span, the Gateway client span for route account-service-health, and the Account Service server span.

    This satisfies the local SIT cross-service trace-correlation acceptance criterion. AWS tracing and collector mapping remain deferred. The task can now be closed with the Sprint 6 observability work item.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions