You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat: Add OpenTelemetry tracing for coordinator calls - #106
Every Molto.responses(), Molto.chat_completions(), and Molto.messages() call now emits one OpenTelemetry CLIENT span named chat {model}. It records the route Molto chose (native or translated), the downstream protocol and upstream endpoint, the provider and destination, the response identity, finish reasons, token usage, errors, and a molto.first_token event for streaming callers.
Molto depends only on opentelemetry-api and never configures a tracer provider. Without an SDK the API returns no-op spans and calls behave exactly as before; applications own the SDK, exporters, and OTEL_* configuration.
The coordinator owns the span. Adapters, normalizers, and upstream clients stay telemetry-free, and telemetry never changes routing, validation, request construction, projection, or errors, and never reads extra_body or extra_headers.
Molto.responses(body, endpoint, ...)
_resolve_provider() # unchanged
_traced(protocol, endpoint, provider, base_url, body, call)
start_attributes(route, body) # utils/telemetry.py, pure
start_span(kind=CLIENT) # parent: caller's active span
use_span(span) # current only while the upstream request is sent
_run_responses(..., recorder) # the previous coordinator body
observe(upstream events, recorder) # reads output for end attributes and first token
non-streaming -> end span
streaming -> MoltoStream ends it on exhaustion, error, aclose(), or garbage collection
Added docs/observability.md, the span contract: name, kind, status, parent and current-span rules, start and end attributes, error attributes, events, the content policy, and how applications enable export and httpx child spans.
Added opentelemetry-api>=1.36.0,<1.46 to core dependencies, and pinned opentelemetry-api==1.45.2, opentelemetry-sdk==1.45.2, and opentelemetry-instrumentation-httpx==0.66b2 in test. The floor is the lowest version on which the full suite passes unchanged, and the <1.46 ceiling stops just above 1.45.2, the newest release, on which it also passes. Each opentelemetry-sdk release requires the matching API version, so the ceiling must be re-derived with the suite when OpenTelemetry releases 1.46.
Added python/molto/utils/telemetry.py, which maps the resolved route, request, and upstream output to GenAI semantic-convention attributes. Keys are plain strings that follow the current spec, including gen_ai.usage.cache_write.input_tokens, rather than imports from the semconv package, which still uses older names.
Finish reasons keep each route's vocabulary: native spans report the provider's value (end_turn, stop, Responses completed, failed, or its incomplete reason), and translated spans report Molto's normalized termination reason. Native Messages input tokens include cached tokens, so native and translated spans report the same usage for the same call. Cache attributes appear only when non-zero.
error.type reports the normalized APIException class, so native and translated routes agree; http.response.status_code comes only from a real upstream error response of 400 or higher; molto.error.phase is setup before the coordinator returns, or stream after it returned a stream, including a failing aclose(). Exceptions are recorded without messages, which can quote request content: the event holds the class name and a frames-only stack trace across the __cause__ chain.
Every streaming span records how it ended in molto.stream.outcome: completed, closed (aclose() before the end), cancelled (the caller's task cancelled, including before the first event), error, or abandoned. Cancellation ends the span with UNSET status. A stream dropped without being consumed or closed ends its span when it is garbage-collected, as abandoned and at the time of its last event; the event loop's async-generator finalizer then closes the dropped source, which releases its upstream response.
Native output is not validated, so telemetry reads every field a provider may omit with a default. record_safely wraps every recording call, so output no extractor expects loses its attributes, with only the exception class logged at DEBUG through Molto's loguru logger, and never turns a successful call into a failure.
A native Responses stream that ends in response.failed without a separate error event returns normally; its span records the failed response with finish reason failed, as the same non-streaming response does.
Native passthrough now streams for any truthy stream, the rule the OpenAI and Anthropic SDKs use (stream=stream or False). It previously streamed only for True, so stream: 1 on a provider that accepted it returned an SDK stream to the non-streaming path and failed with TypeError: Responses request did not return a non-streaming response. A provider that rejects the value still raises its own error. One helper, _streams in core.py, holds each route's rule and decides both native routing and the span's gen_ai.request.stream, so they cannot drift: truthy on native routes, and on translated routes, which validate stream as Literal[True, False], true, 1, or 1.0. Telemetry never reads stream itself.
Ending a span skips error attributes and stack-trace formatting when the span is not recording, and native Chat streams set the response identity and finish reasons only when they change rather than on every chunk.
Added BaseClient.resolve_base_url(), so the clients and span attributes resolve the default destination in one place, and shared replay helpers in molto.utils.test.
Verification
python -m pytest on Python 3.12.13, 3.13.13, and 3.14.3: 2772 passed on each.
uv pip install -e . followed by python -c "import molto" (CI "Check import with core dependencies only"): passed on Python 3.12, 3.13, and 3.14.
opentelemetry-api floor: uv pip install -e ".[test]" --overrides with each candidate's matching opentelemetry-sdk and opentelemetry-instrumentation-httpx, then python -m pytest. 1.36.0 (with instrumentor 0.57b0) passed all 2743 tests on Python 3.12.13, 3.13.13, and 3.14.3, and all 2769 on Python 3.12.13 after the review fixes. 1.28.0 (with 0.49b0), pydantic-ai-slim's floor, passed 2739 on 3.12 and 3.14; the 4 failures are the test-only instrumentor's instrument_client() rejecting meter_provider.
ruff check --config pyproject.toml python tests and ruff format --diff --config pyproject.toml python tests: clean. A repository-wide run also flags test_molto.py, an untracked local file outside this change.
npx prettier@3.8 --check on the changed Markdown: passed.
tests/core/test_tracing.py replays committed cassettes through all three coordinators on native and translated routes, streaming and non-streaming. It covers span name, kind, parenting, start and end attributes, usage, status, the first-token event, every lifecycle edge (exhaustion, early and repeated aclose(), mid-stream failure, failing close, prefetch failure, cancellation of a real call and stream, and dropped streams collected with gc.collect()), each molto.stream.outcome value, error attributes on native and translated routes, and that spans hold only contract keys with no request content, extensions, headers, or credentials.
A fresh interpreter with no SDK produces identical stream events, responses, and errors, and never imports opentelemetry.sdk; another with a global SDK provider records the span under scope molto and the package version.
opentelemetry-instrumentation-httpx spans nest under the Molto span with global instrument(), on Molto's default client and an injected one created before instrumentation, and with instrument_client(); for streams, the HTTP span ends before the first token.
Each fix from the PR review fails its regression tests when reverted: dropping response.failed from the terminal events (1 test), the translated stream rule (2, before the rule moved into _streams), the observe guard (2), and the non-streaming response guard (1), and restoring the native is True stream rule (7).
Dropping an unconsumed, unclosed native or translated stream closes its upstream response body within five event-loop turns after gc.collect(), asserted by test_dropped_stream_releases_its_upstream_response.
Mutation checks: removing the end-on-exhaustion branch fails 6 tests, removing the end-once guard fails 7, removing the use_span call fails all 12 HTTP-span tests, reporting cancellation as error fails both cancellation tests, ending abandoned spans at collection time fails both end-time tests, and removing the __del__ guard fails its test.
ruff check --config pyproject.toml . and ruff format --diff --config pyproject.toml . (the repository-wide lint.yml commands): clean on every tracked file; the only findings are in test_molto.py, an untracked local file, which CI does not check out.
npx prettier@3.8 --check .: passed.
Design principles 1-5 checked against git diff origin/main...HEAD: no SDK import or set_tracer_provider under python/; start_span, use_span, and end() only in core.py; no OpenTelemetry or telemetry import in adapters/, upstream/, or types/; telemetry.py never references extra_body or extra_headers; no semantic-conventions import; no request or completion content read into an attribute.
Cassettes were replayed only; none were recorded or modified.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Molto.responses(),Molto.chat_completions(), andMolto.messages()call now emits one OpenTelemetryCLIENTspan namedchat {model}. It records the route Molto chose (native or translated), the downstream protocol and upstream endpoint, the provider and destination, the response identity, finish reasons, token usage, errors, and amolto.first_tokenevent for streaming callers.opentelemetry-apiand never configures a tracer provider. Without an SDK the API returns no-op spans and calls behave exactly as before; applications own the SDK, exporters, andOTEL_*configuration.extra_bodyorextra_headers.docs/observability.md, the span contract: name, kind, status, parent and current-span rules, start and end attributes, error attributes, events, the content policy, and how applications enable export andhttpxchild spans.opentelemetry-api>=1.36.0,<1.46to core dependencies, and pinnedopentelemetry-api==1.45.2,opentelemetry-sdk==1.45.2, andopentelemetry-instrumentation-httpx==0.66b2intest. The floor is the lowest version on which the full suite passes unchanged, and the<1.46ceiling stops just above 1.45.2, the newest release, on which it also passes. Eachopentelemetry-sdkrelease requires the matching API version, so the ceiling must be re-derived with the suite when OpenTelemetry releases 1.46.python/molto/utils/telemetry.py, which maps the resolved route, request, and upstream output to GenAI semantic-convention attributes. Keys are plain strings that follow the current spec, includinggen_ai.usage.cache_write.input_tokens, rather than imports from the semconv package, which still uses older names.end_turn,stop, Responsescompleted,failed, or its incomplete reason), and translated spans report Molto's normalized termination reason. Native Messages input tokens include cached tokens, so native and translated spans report the same usage for the same call. Cache attributes appear only when non-zero.error.typereports the normalizedAPIExceptionclass, so native and translated routes agree;http.response.status_codecomes only from a real upstream error response of 400 or higher;molto.error.phaseissetupbefore the coordinator returns, orstreamafter it returned a stream, including a failingaclose(). Exceptions are recorded without messages, which can quote request content: the event holds the class name and a frames-only stack trace across the__cause__chain.molto.stream.outcome:completed,closed(aclose()before the end),cancelled(the caller's task cancelled, including before the first event),error, orabandoned. Cancellation ends the span withUNSETstatus. A stream dropped without being consumed or closed ends its span when it is garbage-collected, asabandonedand at the time of its last event; the event loop's async-generator finalizer then closes the dropped source, which releases its upstream response.record_safelywraps every recording call, so output no extractor expects loses its attributes, with only the exception class logged atDEBUGthrough Molto'slogurulogger, and never turns a successful call into a failure.response.failedwithout a separateerrorevent returns normally; its span records the failed response with finish reasonfailed, as the same non-streaming response does.stream, the rule the OpenAI and Anthropic SDKs use (stream=stream or False). It previously streamed only forTrue, sostream: 1on a provider that accepted it returned an SDK stream to the non-streaming path and failed withTypeError: Responses request did not return a non-streaming response. A provider that rejects the value still raises its own error. One helper,_streamsincore.py, holds each route's rule and decides both native routing and the span'sgen_ai.request.stream, so they cannot drift: truthy on native routes, and on translated routes, which validatestreamasLiteral[True, False],true,1, or1.0. Telemetry never readsstreamitself.BaseClient.resolve_base_url(), so the clients and span attributes resolve the default destination in one place, and shared replay helpers inmolto.utils.test.Verification
python -m pyteston Python 3.12.13, 3.13.13, and 3.14.3: 2772 passed on each.uv pip install -e .followed bypython -c "import molto"(CI "Check import with core dependencies only"): passed on Python 3.12, 3.13, and 3.14.opentelemetry-apifloor:uv pip install -e ".[test]" --overrideswith each candidate's matchingopentelemetry-sdkandopentelemetry-instrumentation-httpx, thenpython -m pytest. 1.36.0 (with instrumentor 0.57b0) passed all 2743 tests on Python 3.12.13, 3.13.13, and 3.14.3, and all 2769 on Python 3.12.13 after the review fixes. 1.28.0 (with 0.49b0),pydantic-ai-slim's floor, passed 2739 on 3.12 and 3.14; the 4 failures are the test-only instrumentor'sinstrument_client()rejectingmeter_provider.ruff check --config pyproject.toml python testsandruff format --diff --config pyproject.toml python tests: clean. A repository-wide run also flagstest_molto.py, an untracked local file outside this change.npx prettier@3.8 --checkon the changed Markdown: passed.tests/core/test_tracing.pyreplays committed cassettes through all three coordinators on native and translated routes, streaming and non-streaming. It covers span name, kind, parenting, start and end attributes, usage, status, the first-token event, every lifecycle edge (exhaustion, early and repeatedaclose(), mid-stream failure, failing close, prefetch failure, cancellation of a real call and stream, and dropped streams collected withgc.collect()), eachmolto.stream.outcomevalue, error attributes on native and translated routes, and that spans hold only contract keys with no request content, extensions, headers, or credentials.opentelemetry.sdk; another with a global SDK provider records the span under scopemoltoand the package version.opentelemetry-instrumentation-httpxspans nest under the Molto span with globalinstrument(), on Molto's default client and an injected one created before instrumentation, and withinstrument_client(); for streams, the HTTP span ends before the first token.response.failedfrom the terminal events (1 test), the translated stream rule (2, before the rule moved into_streams), theobserveguard (2), and the non-streaming response guard (1), and restoring the nativeis Truestream rule (7).gc.collect(), asserted bytest_dropped_stream_releases_its_upstream_response.use_spancall fails all 12 HTTP-span tests, reporting cancellation aserrorfails both cancellation tests, ending abandoned spans at collection time fails both end-time tests, and removing the__del__guard fails its test.ruff check --config pyproject.toml .andruff format --diff --config pyproject.toml .(the repository-widelint.ymlcommands): clean on every tracked file; the only findings are intest_molto.py, an untracked local file, which CI does not check out.npx prettier@3.8 --check .: passed.git diff origin/main...HEAD: no SDK import orset_tracer_providerunderpython/;start_span,use_span, andend()only incore.py; no OpenTelemetry or telemetry import inadapters/,upstream/, ortypes/;telemetry.pynever referencesextra_bodyorextra_headers; no semantic-conventions import; no request or completion content read into an attribute.