Repository navigation
Conversation
Coverage Report (Python 3.12) •
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
Coverage Report (Python 3.13) •
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
jiahuei
left a comment
There was a problem hiding this comment.
Review of 19c60a5 against 6036a7f: ten findings, each in its own inline thread.
- Two were reproduced with probes at this commit: the Chat projection error for OpenRouter summaries, and the forwarding of explicit null message fields.
- Six are behaviour findings from reading the code, and two are cleanups.
- The offline replay suite passes at this commit (
pytest, Python 3.12: 2837 passed), so none of these is caught by existing tests. - No violations of
AGENTS.mdwere found; all touched tests have docstrings.
koladefaj
left a comment
There was a problem hiding this comment.
I reviewed the provider integrations and multi-turn conversation handling and found four issues I'd like your input on before merge. These findings are against 19c60a5.
- Messages requests through Gemini and Fireworks Chat fail with HTTP 400. Molto forwards
include_obfuscation, which the tested endpoints reject. The SDK controls succeed when that option is omitted. - Gemini tool turns return the wrong Messages stop reason. After removing the option above, a response containing a tool call returns
end_turninstead oftool_use. This can cause clients to exit their tool loop without executing the tool. - Chat history now includes unused null fields. Passing SDK-shaped assistant history forwards fields previously omitted, including
audio,function_call, andrefusal. Live checks confirmed that both providers reject history containing these unused null fields and accept it when those fields are omitted. - OpenRouter reasoning summaries fail conversion to Chat output. The synthetic unsigned-summary reproduction raises an error despite the documented support for this conversion. This demonstrates a conversion defect, but does not establish that the recorded OpenRouter model produces those summaries.
The inline comments and replies below include the code paths, reproduction evidence, and suggested regression tests.
I also have two contract questions:
- Should we document OpenRouter reasoning-history replay as a known limitation for both Chat and Messages?
- What should omitted tool
strictmean when translating Responses requests to Chat? It currently becomesfalse, which differs from the native Responses behavior observed in the experiment.
Verification: 114 focused offline replay tests passed (tests/core/test_unsigned_reasoning.py, tests/upstream/test_openrouter.py, and tests/upstream/test_multi_turn.py). Live checks used the documented provider configurations. One Gemini control initially returned 503 and succeeded on retry. The full suite was not rerun. Live observations are limited to the tested models and settings.
Removing the formal request-changes status at the reviewer request. The feedback comments remain for discussion.
jiahuei
left a comment
There was a problem hiding this comment.
Review at e378cc0
Merged main (232659e, #105) into the branch as e378cc0. Four documentation files conflicted: docs/protocols/openai-chat-completions.md, python/ARCHITECTURE.md, python/CONVENTIONS.md, and tests/cassettes/README.md. Each resolution keeps both sides. From main: the include=["web_search_call.action.sources"] omission exception, the "translated Chat requests reject n" wording, and the anthropic/web_search/ directory. From this branch: the format="unknown" exception, the Gemini limitation sentence, and the recorder masking text. Source, test, and cassette files merged cleanly.
No blocking defects found in the source. One inline item should be fixed before this leaves draft: the stale Gemini contract in CONVENTIONS.md. The other inline comments are migration notes, documentation gaps, and questions.
Not anchorable inline
strict: nullrelative tomain. Thestrict: truedefault follows the resolved contract, so this is not reopening it. Compared withmain, though, a translated Responses function tool withstrict: nullused to sendstrict: falseupstream and now sendsstrict: true. I checked this with a mock transport forresponses(endpoint="chat/completions", provider="openai"); the Anthropic upstream likewise adds"strict": true. Omittedstrictwas a local validation error onmain. Suggest listing thenullchange as a behavior change in the PR summary.- The PR description's Verification section still cites
af4c215.
Still open from the previous round. The merge did not change the code these cite:
- Unregistered native Messages routes send to a doubled
/v1path - Reasoning text plus a signature splits one Gemini assistant turn
- Fireworks Responses default reasoning stops tool loops at the second turn
Checked, no finding
- Credentials in the 97 new cassettes: every retained
authorizationvalue isBearer ***; the Bedrock Messages captures keep noauthorizationorx-api-key; there are no query parameters; no key-shaped strings, AWS account ARNs, or email addresses. scripts/recorder_proxy.pymasking, including echoed and query credentials, and saving on every exit path;scripts/smoke_provider.pyrequest budget and failure output;.env.exampleplaceholders are empty.- Shared Chat normalizer changes that also affect OpenAI and vLLM: covered by
test_tool_lifecycle_closes_before_following_contentandtest_closed_tool_call_cannot_resume. serialize_messages(exclude_none=False): the translated message builders never insertNone, so no new nulls reach the wire.- All 81 test functions added by this PR have docstrings.
Verification at e378cc0, Python 3.12.9, with main's pinned test extra (openai 2.53.0, anthropic 0.121.0, pytest 8.4.2, pytest-asyncio 1.4.0), offline replay only:
OPENAI_API_KEY=DUMMY_KEY python -m pytest: 3,215 passed;git status --porcelain -- tests/cassettesshowed no cassette changes.ruff check --output-format github --config pyproject.toml .: passed.ruff format --diff --config pyproject.toml .: passed, 103 files already formatted.npx prettier@3.8 --check .: passed.gh pr checks 96: all five checks passed one378cc0.- Not reviewed line by line:
tests/core/test_provider_history_routes.pyandtests/test_hosted_providers.pywere skimmed, and cassette payloads were checked for secrets and headers but not audited file by file.
Restores public SDK parity coverage for OpenRouter Responses summaries reaching Chat Completions callers, removed from the unsigned-reasoning parity cases in a004b58. The existing Messages test now runs for both downstream protocols in both response modes.
Summary
reasoning. Setchat_reasoning_field="reasoning_content"for older vLLM servers. Setchat_signature_namespaceto carry opaque signatures through a compatible Chat extension. Gemini usesgoogleby default. Translated Gemini text streams immediately, and late message signatures and per-tool signatures return to their recorded attachments on continuation.None, distinct from a provider-reported zero.body_bytesfrom 59 older committed recordings after comparing each copy with its retained JSON body. The reasoning guide and provider matrix describe projection, continuation, and the tested models and settings.Migration
base_url. Selectprovider="openrouter"orprovider="gemini"when translating those providers' extensions through a compatible-server URL.strict: truewhen strictness is omitted or null. Previouslystrict: nullsentfalse. Explicitfalseremains available. Molto passes the supplied schema through and does not retry in non-strict mode.reasoning_text. Its live probe established acceptance of that request shape. The provider's use of the two histories may differ. Bedrock Chat uses raw classification for the tested Mantle GPT-OSS configuration. Bedrock Messages preserves Claude's signed thinking. A client that declaresREQUIRES_REASONING_REPRESENTATIONrequires an explicit selection for translated calls only.Related to #78.
Verification
OPENAI_API_KEY=DUMMY_KEY COVERAGE_FILE=/tmp/pr96-final-coverage /tmp/pr96-final-replay-venv/bin/python -m pytest --cov=python/molto --cov-report=term-missing --cov-report=xml:/tmp/pr96-final-coverage.xml --junitxml=/tmp/pr96-final-results.xmlpassed all 3,452 tests ated6ef11with 95.4% line coverage on Python 3.12.9 and the pinned test extra. The existing deliberately invalid Gemini tool-index serialization test produced one warning.uv run --extra lint ruff check --output-format github --config pyproject.toml .,uv run --extra lint ruff format --diff --config pyproject.toml ., andnpx prettier@3.8 --check .passed. All 157 checked relative links in the 11 changed guides and cassette documentation resolve.git diff origin/main...HEAD --checkpassed. All 31 added YAML files are byte-identical to the sanitized recordings. The reasoning update contains no changes to previously committed cassette payloads.OPENAI_API_KEY=DUMMY_KEY .venv/bin/python -m pytestpassed all 3,456 tests at87b6406, which adds public Chat Completions coverage for OpenRouter summary projection, on Python 3.12.9 with the pinned test extra. Cassette files were unchanged after the run.87b6406, including offline replay on Python 3.12, 3.13 and 3.14, Ruff, and Prettier. Hosted replay, hosted lint