Skip to content

fix(llm-client): keep context-window message out of client spans and request log - #1347

Open
deepujain wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
deepujain:fix/615-keep-upstream-error-content-out
Open

deepujain wants to merge 1 commit into
NVIDIA-NeMo:mainfrom
deepujain:fix/615-keep-upstream-error-content-out

Conversation

@deepujain

@deepujain deepujain commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

What

When a provider rejects a request for being too long, its message can quote the prompt back. Two places still wrote that message out:

  • The libsy.client_call span, for both buffered and streamed calls. It recorded the error's full Display text.
  • The server's terminal switchyard_server::request log line, which recorded the response message.

Both now record only the error class. The span keeps error.type = context_window_exceeded and adds the model name. The request log records context_length_exceeded.

The error itself is unchanged. Fallback and retries still see the full typed error, and the client still gets the provider's message in the HTTP response.

Why

Part of #615. That issue asks for upstream error content to stay out of logs and traces. Earlier PRs covered most sinks: #611 (inner client-call span for HTTP errors), #794 (Advisor fail-open reports) and #826 (server request log and SSE errors for upstream HTTP errors). Checking each sink on current main showed that ContextWindowExceeded still reached the client span and the request log through its message. The Advisor path, the judge warnings, the runner and Relay failure summaries and the SSE error log already report only the kind and model.

This PR does not close #615. The other free-form variants (InvalidRequest, Configuration, translation errors and boxed sources) still carry their text into the server request log. They come from the router or translator rather than the provider's reply, so I left them out. A maintainer can decide whether they need the same treatment.

Notes for reviewers

Start with crates/libsy-llm-client/src/observability.rs: a ContextWindowExceeded arm in the buffered path and one in the stream observer, next to the existing UpstreamHttp arms. In crates/switchyard-server/src/lib.rs, client_error replaces the request-log text for this variant, the same way upstream_error does.

Tests, written before the fix:

  • context_window_message_is_redacted_from_the_client_call_span and streamed_context_window_message_is_redacted_from_the_client_call_span in crates/libsy-llm-client/tests/observability.rs. Both failed on main with the quoted text in the span error field.
  • context_window_error_redacts_request_log_error in crates/switchyard-server/src/lib.rs. It failed on main with the quoted text in the log line, and checks that the client response still carries the message on all three wire formats.

Validation run: cargo fmt --all --check, cargo clippy --locked -p switchyard-llm-client -p switchyard-server --all-targets -- -D warnings, cargo test --locked -p switchyard-llm-client and cargo test --locked -p switchyard-server. All passed.

Live check: I sent the same overlong prompt through the native server on the base and on this head, once buffered and once streamed, over a passthrough route to the NVIDIA-hosted Gemma 2 9B model. On the base, the request log and the client trace both carried the provider's full error. On this head, the request log says context_length_exceeded and the client trace says the call failed because the context window was exceeded. Both callers still got HTTP 400 and the full provider message. The result is in a comment on this PR.

Not covered live: this provider rejects the request before any streamed content, so an error that arrives inside an already open stream was not exercised. The stream observer arm is covered only by streamed_context_window_message_is_redacted_from_the_client_call_span.

Not run: the full workspace test suite, the prefill-router feature build and the Python tests. CI has not run on this head yet because the fork workflow runs are waiting for maintainer approval.

Summary by CodeRabbit

  • Bug Fixes
    • Context-window errors are now represented by fixed classifications in request logs and monitoring data, so quoted request content is not included in those records.
    • API responses continue to include the provider’s error message and the context_length_exceeded code. The error classification is also retained for streamed requests.

…request log

Signed-off-by: Deepak Jain <deepujain@gmail.com>
@deepujain
deepujain marked this pull request as ready for review November 19, 2026 01:34
@deepujain
deepujain requested a review from a team as a code owner November 19, 2026 01:34
@coderabbitai

coderabbitai Bot commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

No actionable comments were generated in the recent review.

Walkthrough

Context-window failures now use fixed descriptions in client spans and a fixed classification in server request logs. The typed error remains available to callers, and HTTP 400 responses retain the provider message.

Changes

Context-window error logging

Layer / File(s) Summary
Client span classification
crates/libsy-llm-client/src/observability.rs, crates/libsy-llm-client/tests/observability.rs
Client-call and streaming spans record context-window error descriptions without the leaked content. Tests check the span fields and confirm the typed error retains its message.
Server request log classification
crates/switchyard-server/src/lib.rs
Context-window errors receive the fixed context_length_exceeded log classification. Tests check all three wire formats and confirm that the response retains the provider message.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Severity of issue fixed: Medium

Merge Risk: Minimal · up to 40d3d

This change keeps provider-supplied context-window error text out of client spans and server request logs. Callers still receive the original error message, and the HTTP 400 response is unchanged. No merge-blocking risk is evident.

Security architecture risk: Minimal · up to 40d3d

The change reduces disclosure of request content in diagnostics without changing caller-visible errors or failure handling. No introduced or worsened architecture-level risk was identified.

Retained concerns
No architecture-level concerns identified.

Security review details

Security Blast Radius

  • inferred — The changed information flow is limited to diagnostic handling of typed context-window failures. It reduces a path by which provider messages quoting request content could reach trace and request-log consumers. Caller-facing access to the original message is preserved; downstream retention and tenant isolation were not established by the inspected evidence.

Trust Boundaries and Controls

  • observed — The new control avoids formatting the provider-message field at the affected diagnostic sinks while retaining error classification. Added regression tests assert message exclusion from buffered and streamed client spans and request logs across three wire formats, plus preservation of caller error content. These assertions were inspected, not executed.

Resilience and Maintainability Implications

  • observed — The redaction branches preserve existing terminal-state handling. A stream error marks observation failed, releases the observer, and forwards the error item; normal completion and abandonment retain their separate outcomes. Request-log guards consume their context once for either response emission or cancellation. The new server override is constructed synchronously before response publication.
Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title names the main change: the context-window message is kept out of the client-call span and the request log in the llm-client crate. The raw summary shows the client span and server request-lo…
Linked Issues check Passed [#615] The issue names ContextWindowExceeded as the remaining unredacted case. The diff covers it at both final sinks. In crates/libsy-llm-client/src/observability.rs, observe_client_call adds a Conte…
Out of Scope Changes check Passed All source changes apply to the ContextWindowExceeded variant. The observability.rs hunks add arms next to the existing UpstreamHttp arms and change only the recorded span text. The lib.rs hunk change…
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Full details: Title check

Explanation

The title names the main change: the context-window message is kept out of the client-call span and the request log in the llm-client crate. The raw summary shows the client span and server request-log changes, and the objectives confirm this is the core intent. The title is concise and specific. It is an accurate, single-sentence summary, so it passes.

Full details: Linked Issues check

Explanation

[#615] The issue names ContextWindowExceeded as the remaining unredacted case. The diff covers it at both final sinks. In crates/libsy-llm-client/src/observability.rs, observe_client_call adds a ContextWindowExceeded arm that records a fixed message with the target and no provider text. ClientStreamObserver adds a matching arm that records the model name and no provider text. In crates/switchyard-server/src/lib.rs, client_error attaches a RequestLogError('context_length_exceeded') extension, so the terminal request log omits the message. The returned LlmClientError keeps its message, so retries, fallback, and the HTTP response are unchanged. The added tests cover both client-call paths and the request log on all three wire formats, and check that the client response still carries the message. The PR does not claim to close #615. It states that InvalidRequest, Configuration, translation errors, and boxed sources remain. The objectives given for this review also limit scope to ContextWindowExceeded. Not re-verified in this review: the libsy.llm_call and libsy.run spans and the Advisor, runner, and Relay paths for ContextWindowExceeded. The PR author reports these already record only kind and model.

Full details: Out of Scope Changes check

Explanation

All source changes apply to the ContextWindowExceeded variant. The observability.rs hunks add arms next to the existing UpstreamHttp arms and change only the recorded span text. The lib.rs hunk changes only the ContextWindowExceeded branch of client_error, and the new test covers that branch. The added tests in crates/libsy-llm-client/tests/observability.rs support the same change. No unrelated files or behavior changes appear in the reviewed diffs. The diffs for the two source files were read directly. The tests in tests/observability.rs were not read in this review, but the change summary describes them as supporting tests for the same variant.

  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

A rabbit hops past traces bright,
The quoted words stay out of sight.
A fixed error tag is all logs show,
The caller keeps the text they know.
Three wire formats pass the night.

Recent review info

Run configuration

Commits

Reviewing files that changed from the base of the PR and between 1e0a94e and 40d3df1.

Files selected for processing: 3

  • crates/libsy-llm-client/src/observability.rs
  • crates/libsy-llm-client/tests/observability.rs
  • crates/switchyard-server/src/lib.rs

Included review availability: This review used your included allowance. Your plan provides up to 12 included reviews per hour; 11 remain after this review.

@deepujain

Copy link
Copy Markdown
Contributor Author

I sent the same overlong prompt through the native Switchyard server on both revisions, once as a buffered request and once with streaming enabled. Both used a passthrough route to the live NVIDIA-hosted Gemma 2 9B model. I checked the HTTP reply, the terminal request log and the exported client trace.

Before (the base of this PR): the request log and the client trace both included the provider's full error, including "Context Window exceeded for given call. No models have context window large enough for this call."

After (this PR): the request log says context_length_exceeded, and the client trace says client call to target "overflow" failed: context window exceeded. Both callers still get HTTP 400 and the same full provider message.

This provider did not quote the prompt in its error. It rejected both requests before any streamed content, so errors that arrive inside an open stream were not exercised live.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[bug] Keep upstream error content out of logs and traces

1 participant