Repository navigation
Conversation
…request log Signed-off-by: Deepak Jain <deepujain@gmail.com>
|
No actionable comments were generated in the recent review. WalkthroughContext-window failures now use fixed descriptions in client spans and a fixed classification in server request logs. The typed error remains available to callers, and HTTP 400 responses retain the provider message. ChangesContext-window error logging
Estimated code review effort: 3 (Moderate) | ~20 minutes Severity of issue fixed: Medium Merge Risk:
|
|
I sent the same overlong prompt through the native Switchyard server on both revisions, once as a buffered request and once with streaming enabled. Both used a passthrough route to the live NVIDIA-hosted Gemma 2 9B model. I checked the HTTP reply, the terminal request log and the exported client trace. Before (the base of this PR): the request log and the client trace both included the provider's full error, including "Context Window exceeded for given call. No models have context window large enough for this call." After (this PR): the request log says This provider did not quote the prompt in its error. It rejected both requests before any streamed content, so errors that arrive inside an open stream were not exercised live. |
What
When a provider rejects a request for being too long, its message can quote the prompt back. Two places still wrote that message out:
libsy.client_callspan, for both buffered and streamed calls. It recorded the error's fullDisplaytext.switchyard_server::requestlog line, which recorded the response message.Both now record only the error class. The span keeps
error.type = context_window_exceededand adds the model name. The request log recordscontext_length_exceeded.The error itself is unchanged. Fallback and retries still see the full typed error, and the client still gets the provider's message in the HTTP response.
Why
Part of #615. That issue asks for upstream error content to stay out of logs and traces. Earlier PRs covered most sinks: #611 (inner client-call span for HTTP errors), #794 (Advisor fail-open reports) and #826 (server request log and SSE errors for upstream HTTP errors). Checking each sink on current main showed that
ContextWindowExceededstill reached the client span and the request log through its message. The Advisor path, the judge warnings, the runner and Relay failure summaries and the SSE error log already report only the kind and model.This PR does not close #615. The other free-form variants (
InvalidRequest,Configuration, translation errors and boxed sources) still carry their text into the server request log. They come from the router or translator rather than the provider's reply, so I left them out. A maintainer can decide whether they need the same treatment.Notes for reviewers
Start with
crates/libsy-llm-client/src/observability.rs: aContextWindowExceededarm in the buffered path and one in the stream observer, next to the existingUpstreamHttparms. Incrates/switchyard-server/src/lib.rs,client_errorreplaces the request-log text for this variant, the same wayupstream_errordoes.Tests, written before the fix:
context_window_message_is_redacted_from_the_client_call_spanandstreamed_context_window_message_is_redacted_from_the_client_call_spanincrates/libsy-llm-client/tests/observability.rs. Both failed on main with the quoted text in the spanerrorfield.context_window_error_redacts_request_log_errorincrates/switchyard-server/src/lib.rs. It failed on main with the quoted text in the log line, and checks that the client response still carries the message on all three wire formats.Validation run:
cargo fmt --all --check,cargo clippy --locked -p switchyard-llm-client -p switchyard-server --all-targets -- -D warnings,cargo test --locked -p switchyard-llm-clientandcargo test --locked -p switchyard-server. All passed.Live check: I sent the same overlong prompt through the native server on the base and on this head, once buffered and once streamed, over a passthrough route to the NVIDIA-hosted Gemma 2 9B model. On the base, the request log and the client trace both carried the provider's full error. On this head, the request log says
context_length_exceededand the client trace says the call failed because the context window was exceeded. Both callers still got HTTP 400 and the full provider message. The result is in a comment on this PR.Not covered live: this provider rejects the request before any streamed content, so an error that arrives inside an already open stream was not exercised. The stream observer arm is covered only by
streamed_context_window_message_is_redacted_from_the_client_call_span.Not run: the full workspace test suite, the
prefill-routerfeature build and the Python tests. CI has not run on this head yet because the fork workflow runs are waiting for maintainer approval.Summary by CodeRabbit
context_length_exceededcode. The error classification is also retained for streamed requests.