Skip to content

feat(telemetry): trace each MCP request and tool call under the caller's trace - #103

Open
braghettos wants to merge 1 commit into
kagent-dev:mainfrom
braghettos:feat/trace-mcp-tool-calls
Open

braghettos wants to merge 1 commit into
kagent-dev:mainfrom
braghettos:feat/trace-mcp-tool-calls

Conversation

@braghettos

Copy link
Copy Markdown
Contributor

Part of #102

What

With tracing enabled, a tool call made over MCP HTTP now shows up in the caller's trace:

  • Server span per HTTP request. telemetry.HTTPMiddleware now wraps the handler in otelhttp.NewHandler (go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.72.0, the release that matches the otel v1.47.0 SDK already in go.mod). The span is a child of the incoming W3C traceparent. The standalone SSE stream (GET) is filtered out, because it stays open for the whole session and its span would last for hours and parent nothing. POST responses that stream SSE work as before: otelhttp wraps the writer with httpsnoop, so Flush is kept.
  • Tool-call span under the server span. ToolMiddleware (the single receiving middleware every tool already goes through) names its span tools/call <tool name> (was mcp.tool.<tool name>) and adds the MCP semconv attributes mcp.method.name and gen_ai.tool.name. The existing mcp.tool.* attributes are still there. Error status on an IsError result or a Go error is unchanged.

Why the tool span needed more than a server span

The SDK does not run handlers on the HTTP request's context. A stateful session's handlers run on the context of the request that opened the session (initialize). So even before this change, mcp.tool.* spans joined the trace of the session's initialize request, not the trace of the tools/call that triggered them. What does reach the handler for each request is that request's header (CallToolRequest.Extra.Header). So:

  • HTTPMiddleware clones the request and writes the server span's context into the clone's traceparent header. The caller's request is not modified.
  • ToolMiddleware drops whatever span is in the session context and extracts the parent from Extra.Header. With no header (stdio, in-memory) it keeps the old behaviour.

The middleware also no longer re-extracts trace context on top of an active span. That would have replaced the otelhttp server span with its remote parent.

What is not recorded

No request header, tool argument or tool result is added as a span attribute. Headers can carry credentials, and #66 removed header attributes for that reason. otelhttp's defaults record method, path, status, sizes, peer address and user_agent.original, and no other header.

Tests

cmd/tracing_test.go builds the HTTP handler the way run() does (new helper newMCPHTTPHandler) and records spans with tracetest.InMemoryExporter. It connects a real go-sdk client whose transport sets a different traceparent on initialize and on each tools/call, plus an Authorization header. It calls a succeeding tool, a tool that returns an error result, and a tool that returns a Go error. It asserts:

  • exactly one server span per tools/call request, with the incoming trace ID and a remote parent with the incoming span ID;
  • exactly one tools/call <name> span, a child of that server span and in the tools/call request's trace (not the initialize trace);
  • status Ok for the succeeding tool and Error for both failing tools;
  • mcp.method.name and gen_ai.tool.name are set;
  • no attribute key on any span contains header, authorization or token, and no attribute value contains the bearer credential or the tool argument.

When toolSpanParent is disabled, the test fails on the trace-ID and parent assertions, which confirms the session-context problem described above.

make lint (golangci-lint v2.15.0): 0 issues. go vet ./... is clean. go test -tags=test ./pkg/... ./internal/... ./cmd/... passes.

Follow-up (not in this PR)

This PR covers only the first half of #102. The second half is the k8s tools: they run kubectl as a subprocess, which cannot carry trace context. That is left for a follow-up, which would use client-go in-process for the read verbs, with an otelhttp transport, so traceparent reaches the API server.

Note

Dashboards or queries that match the old span name mcp.tool.<tool> need to switch to tools/call <tool> or filter on the mcp.tool.name / gen_ai.tool.name attribute.

🤖 Generated with Claude Code

https://claude.ai/code/session_016RAxvkp7wRXwXDx4Lx8STi

…r's trace

Wrap the MCP HTTP handler in otelhttp, so every request gets a server
span that is a child of the incoming W3C traceparent. The standalone SSE
stream (GET) is filtered out: it stays open for the whole session.

The tools/call span is now named "tools/call <tool>" and carries
mcp.method.name and gen_ai.tool.name next to the existing mcp.tool.*
attributes. It is parented on the server span of the request that made
the call. The SDK runs handlers on the session's context, which holds the
trace context of the initialize request, so the middleware writes the
server span's context into a cloned request's traceparent header and the
tool middleware extracts it from CallToolRequest.Extra.Header.

No header, argument or result values are added to spans.

Part of kagent-dev#102

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016RAxvkp7wRXwXDx4Lx8STi
Signed-off-by: Diego Braga <diego.braga86@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant