Skip to content

feat: Add provider clients and multi-turn replay coverage - #96

Draft
jiahuei wants to merge 38 commits into
mainfrom
feat/provider-compatibility-evidence
Draft

jiahuei wants to merge 38 commits into
mainfrom
feat/provider-compatibility-evidence

Conversation

@jiahuei

@jiahuei jiahuei commented Nov 3, 2026 •

Copy link
Copy Markdown
Member

Summary

  • Added registered Responses, Chat Completions, and Messages clients for Bedrock, Fireworks, Together AI, OpenRouter, and Gemini on their supported endpoints. Both native and translated calls select an explicit provider/endpoint pair. Bedrock uses the caller's regional URL and supports Anthropic-native Messages for Claude.
  • Responses preserves raw reasoning, summaries, and encrypted state independently. Chat and Messages classify incoming unlabeled reasoning with the provider profile. For outgoing history they prefer raw content when both fields exist. When streaming either protocol, each reasoning part uses its first nonempty representation. Both upstream request builders for Responses and Messages accept unsigned summaries.
  • Chat recognizes both reasoning field names and Google's thought marker. vLLM and Bedrock Chat default to reasoning. Set chat_reasoning_field="reasoning_content" for older vLLM servers. Set chat_signature_namespace to carry opaque signatures through a compatible Chat extension. Gemini uses google by default. Translated Gemini text streams immediately, and late message signatures and per-tool signatures return to their recorded attachments on continuation.
  • Chat history keeps independent reasoning parts in separate assistant messages. Within an assistant message it groups text and tool calls while retaining order within each field and preserving tool-result associations. Streams emit the latest signature per reasoning run at the next text or tool boundary, or at completion. OpenAI SDK collection concatenates message-level signatures from different runs at the same extension path.
  • Added actual-output conversation replay with independent SDK comparisons and exact outgoing history across the eight registered providers. The 31 new recordings include Gemini text and sequential tools with signed, marked history. A four-call live comparison verified an assistant message carrying visible text, its original signature, and reasoning from a separate generation. New Oregon Bedrock Mantle and Fireworks Responses recordings cover reasoning and tool continuation. An omitted cache-write count remains None, distinct from a provider-reported zero.
  • Preserved provider credential isolation, added bounded opt-in smoke tooling, and limited Gemini's native tool-index repair to missing indices. Removed redundant body_bytes from 59 older committed recordings after comparing each copy with its retained JSON body. The reasoning guide and provider matrix describe projection, continuation, and the tested models and settings.

Migration

  • Unregistered endpoint pairs fail before HTTP, including calls with an explicit base_url. Select provider="openrouter" or provider="gemini" when translating those providers' extensions through a compatible-server URL.
  • Translated Responses tool declarations now send strict: true when strictness is omitted or null. Previously strict: null sent false. Explicit false remains available. Molto passes the supplied schema through and does not retry in non-strict mode.
  • Chat and Messages history loses the original Responses reasoning label. For example, Fireworks can emit a Responses summary that returns through a raw profile as reasoning_text. Its live probe established acceptance of that request shape. The provider's use of the two histories may differ. Bedrock Chat uses raw classification for the tested Mantle GPT-OSS configuration. Bedrock Messages preserves Claude's signed thinking. A client that declares REQUIRES_REASONING_REPRESENTATION requires an explicit selection for translated calls only.

Related to #78.

Verification

  • OPENAI_API_KEY=DUMMY_KEY COVERAGE_FILE=/tmp/pr96-final-coverage /tmp/pr96-final-replay-venv/bin/python -m pytest --cov=python/molto --cov-report=term-missing --cov-report=xml:/tmp/pr96-final-coverage.xml --junitxml=/tmp/pr96-final-results.xml passed all 3,452 tests at ed6ef11 with 95.4% line coverage on Python 3.12.9 and the pinned test extra. The existing deliberately invalid Gemini tool-index serialization test produced one warning.
  • The 12 control/combined and 10 mixed-mode Gemini replay cases passed. Live calls used the pinned OpenAI SDK, zero retries, bounded token and request budgets, and the recorder proxy outside pytest. Native conversation run, mixed-mode run, four-call control comparison
  • uv run --extra lint ruff check --output-format github --config pyproject.toml ., uv run --extra lint ruff format --diff --config pyproject.toml ., and npx prettier@3.8 --check . passed. All 157 checked relative links in the 11 changed guides and cassette documentation resolve.
  • git diff origin/main...HEAD --check passed. All 31 added YAML files are byte-identical to the sanitized recordings. The reasoning update contains no changes to previously committed cassette payloads.
  • OPENAI_API_KEY=DUMMY_KEY .venv/bin/python -m pytest passed all 3,456 tests at 87b6406, which adds public Chat Completions coverage for OpenRouter summary projection, on Python 3.12.9 with the pinned test extra. Cassette files were unchanged after the run.
  • All five hosted checks passed at 87b6406, including offline replay on Python 3.12, 3.13 and 3.14, Ruff, and Prettier. Hosted replay, hosted lint

@jiahuei jiahuei changed the title feat: Add provider compatibility and recording evidence feat: Add provider-specific protocol clients and compatibility evidence Nov 5, 2026
@jiahuei jiahuei changed the title feat: Add provider-specific protocol clients and compatibility evidence feat: Add provider clients and multi-turn replay coverage Nov 5, 2026
@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

Coverage

Coverage Report (Python 3.14) •
FileStmtsMissCoverMissing
configs.py41490%45, 49, 57, 61
core.py167696%124, 137, 186, 195, 222, 228
providers.py130100% 
adapters
   anthropic.py4433891%147, 157, 195, 223, 260, 311, 316, 320, 327, 331, 335, 341, 347, 351–353, 356–358, 361, 369, 371, 376, 382, 389, 413–414, 519, 581, 607, 631, 712, 738, 798, 818, 843, 847, 922
   base.py87495%85, 166, 175, 180
   openai.py6745592%114, 195–197, 199, 208, 359, 364, 376, 423, 425, 460, 469, 484, 495, 520, 535, 560, 592, 733, 772, 835, 838, 892, 894, 908, 944–947, 956, 960, 1017, 1092, 1162, 1195, 1201, 1237, 1243, 1254, 1264, 1311, 1320, 1337, 1339, 1449, 1498, 1500, 1519, 1573, 1588, 1592, 1606, 1617, 1624
types
   normalized.py187199%298
   openai_chat_completions.py150199%93
   openai_responses.py183199%443
upstream
   anthropic.py5052695%98, 104, 122–123, 246, 350, 367, 469, 549, 569, 615–616, 719–720, 723, 729, 740, 749, 792, 809, 822, 826, 878–879, 912, 914
   base.py71494%136, 141, 155, 160
   bedrock.py230100% 
   fireworks.py170100% 
   gemini.py360100% 
   openai.py8193296%159, 161, 181, 191, 203, 267–268, 277, 423, 441, 595, 603, 605, 608, 642–643, 766, 842, 968, 987, 1219, 1251, 1295–1296, 1335–1336, 1489, 1521, 1539, 1543, 1571, 1573
   openrouter.py170100% 
   registry.py110100% 
   together.py70100% 
   vllm.py60100% 
utils
   test.py1650100% 
TOTAL387018095% 

Tests Skipped Failures Errors Time
3456 0 💤 0 ❌ 0 🔥 38.874s ⏱️

@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

Coverage

Coverage Report (Python 3.12) •
FileStmtsMissCoverMissing
configs.py41490%45, 49, 57, 61
core.py167696%124, 137, 186, 195, 222, 228
providers.py140100% 
adapters
   anthropic.py4433891%147, 157, 195, 223, 260, 311, 316, 320, 327, 331, 335, 341, 347, 351–353, 356–358, 361, 369, 371, 376, 382, 389, 413–414, 519, 581, 607, 631, 712, 738, 798, 818, 843, 847, 922
   base.py87495%85, 166, 175, 180
   openai.py6745592%114, 195–197, 199, 208, 359, 364, 376, 423, 425, 460, 469, 484, 495, 520, 535, 560, 592, 733, 772, 835, 838, 892, 894, 908, 944–947, 956, 960, 1017, 1092, 1162, 1195, 1201, 1237, 1243, 1254, 1264, 1311, 1320, 1337, 1339, 1449, 1498, 1500, 1519, 1573, 1588, 1592, 1606, 1617, 1624
types
   normalized.py230199%298
   openai_chat_completions.py150199%93
   openai_responses.py183199%443
upstream
   anthropic.py5052695%98, 104, 122–123, 246, 350, 367, 469, 549, 569, 615–616, 719–720, 723, 729, 740, 749, 792, 809, 822, 826, 878–879, 912, 914
   base.py73495%136, 141, 155, 160
   bedrock.py230100% 
   fireworks.py170100% 
   gemini.py360100% 
   openai.py8193296%159, 161, 181, 191, 203, 267–268, 277, 423, 441, 595, 603, 605, 608, 642–643, 766, 842, 968, 987, 1219, 1251, 1295–1296, 1335–1336, 1489, 1521, 1539, 1543, 1571, 1573
   openrouter.py170100% 
   registry.py110100% 
   together.py70100% 
   vllm.py60100% 
utils
   test.py1650100% 
TOTAL391618095% 

Tests Skipped Failures Errors Time
3456 0 💤 0 ❌ 0 🔥 59.610s ⏱️

@github-actions

github-actions Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

Coverage

Coverage Report (Python 3.13) •
FileStmtsMissCoverMissing
configs.py41490%45, 49, 57, 61
core.py167696%124, 137, 186, 195, 222, 228
providers.py140100% 
adapters
   anthropic.py4433891%147, 157, 195, 223, 260, 311, 316, 320, 327, 331, 335, 341, 347, 351–353, 356–358, 361, 369, 371, 376, 382, 389, 413–414, 519, 581, 607, 631, 712, 738, 798, 818, 843, 847, 922
   base.py87495%85, 166, 175, 180
   openai.py6745592%114, 195–197, 199, 208, 359, 364, 376, 423, 425, 460, 469, 484, 495, 520, 535, 560, 592, 733, 772, 835, 838, 892, 894, 908, 944–947, 956, 960, 1017, 1092, 1162, 1195, 1201, 1237, 1243, 1254, 1264, 1311, 1320, 1337, 1339, 1449, 1498, 1500, 1519, 1573, 1588, 1592, 1606, 1617, 1624
types
   normalized.py230199%298
   openai_chat_completions.py150199%93
   openai_responses.py183199%443
upstream
   anthropic.py5052695%98, 104, 122–123, 246, 350, 367, 469, 549, 569, 615–616, 719–720, 723, 729, 740, 749, 792, 809, 822, 826, 878–879, 912, 914
   base.py73495%136, 141, 155, 160
   bedrock.py230100% 
   fireworks.py170100% 
   gemini.py360100% 
   openai.py8193296%159, 161, 181, 191, 203, 267–268, 277, 423, 441, 595, 603, 605, 608, 642–643, 766, 842, 968, 987, 1219, 1251, 1295–1296, 1335–1336, 1489, 1521, 1539, 1543, 1571, 1573
   openrouter.py170100% 
   registry.py110100% 
   together.py70100% 
   vllm.py60100% 
utils
   test.py1650100% 
TOTAL391618095% 

Tests Skipped Failures Errors Time
3456 0 💤 0 ❌ 0 🔥 1m 8s ⏱️

@jiahuei jiahuei left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review of 19c60a5 against 6036a7f: ten findings, each in its own inline thread.

  • Two were reproduced with probes at this commit: the Chat projection error for OpenRouter summaries, and the forwarding of explicit null message fields.
  • Six are behaviour findings from reading the code, and two are cleanups.
  • The offline replay suite passes at this commit (pytest, Python 3.12: 2837 passed), so none of these is caught by existing tests.
  • No violations of AGENTS.md were found; all touched tests have docstrings.

Comment thread python/molto/adapters/openai.py Resolved
Comment thread python/molto/upstream/openai.py Resolved
Comment thread python/molto/upstream/openai.py Resolved
Comment thread python/molto/upstream/gemini.py Resolved
Comment thread python/molto/core.py Resolved
Comment thread python/molto/adapters/openai.py Resolved
Comment thread scripts/recorder_proxy.py Resolved
Comment thread python/molto/core.py Resolved
Comment thread python/molto/upstream/gemini.py Resolved

@koladefaj koladefaj left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I reviewed the provider integrations and multi-turn conversation handling and found four issues I'd like your input on before merge. These findings are against 19c60a5.

  • Messages requests through Gemini and Fireworks Chat fail with HTTP 400. Molto forwards include_obfuscation, which the tested endpoints reject. The SDK controls succeed when that option is omitted.
  • Gemini tool turns return the wrong Messages stop reason. After removing the option above, a response containing a tool call returns end_turn instead of tool_use. This can cause clients to exit their tool loop without executing the tool.
  • Chat history now includes unused null fields. Passing SDK-shaped assistant history forwards fields previously omitted, including audio, function_call, and refusal. Live checks confirmed that both providers reject history containing these unused null fields and accept it when those fields are omitted.
  • OpenRouter reasoning summaries fail conversion to Chat output. The synthetic unsigned-summary reproduction raises an error despite the documented support for this conversion. This demonstrates a conversion defect, but does not establish that the recorded OpenRouter model produces those summaries.

The inline comments and replies below include the code paths, reproduction evidence, and suggested regression tests.

I also have two contract questions:

  • Should we document OpenRouter reasoning-history replay as a known limitation for both Chat and Messages?
  • What should omitted tool strict mean when translating Responses requests to Chat? It currently becomes false, which differs from the native Responses behavior observed in the experiment.

Verification: 114 focused offline replay tests passed (tests/core/test_unsigned_reasoning.py, tests/upstream/test_openrouter.py, and tests/upstream/test_multi_turn.py). Live checks used the documented provider configurations. One Gemini control initially returned 503 and succeeded on retry. The full suite was not rerun. Live observations are limited to the tested models and settings.

Comment thread python/molto/upstream/registry.py Resolved
Comment thread tests/upstream/test_multi_turn.py Resolved
Comment thread python/molto/providers.py Resolved
Comment thread python/molto/types/openai_responses.py Resolved
@koladefaj
koladefaj dismissed their stale review November 10, 2026 16:08

Removing the formal request-changes status at the reviewer request. The feedback comments remain for discussion.

Comment thread python/molto/core.py Resolved
Comment thread python/molto/upstream/gemini.py Resolved

@jiahuei jiahuei left a comment

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review at e378cc0

Merged main (232659e, #105) into the branch as e378cc0. Four documentation files conflicted: docs/protocols/openai-chat-completions.md, python/ARCHITECTURE.md, python/CONVENTIONS.md, and tests/cassettes/README.md. Each resolution keeps both sides. From main: the include=["web_search_call.action.sources"] omission exception, the "translated Chat requests reject n" wording, and the anthropic/web_search/ directory. From this branch: the format="unknown" exception, the Gemini limitation sentence, and the recorder masking text. Source, test, and cassette files merged cleanly.

No blocking defects found in the source. One inline item should be fixed before this leaves draft: the stale Gemini contract in CONVENTIONS.md. The other inline comments are migration notes, documentation gaps, and questions.

Not anchorable inline

  • strict: null relative to main. The strict: true default follows the resolved contract, so this is not reopening it. Compared with main, though, a translated Responses function tool with strict: null used to send strict: false upstream and now sends strict: true. I checked this with a mock transport for responses(endpoint="chat/completions", provider="openai"); the Anthropic upstream likewise adds "strict": true. Omitted strict was a local validation error on main. Suggest listing the null change as a behavior change in the PR summary.
  • The PR description's Verification section still cites af4c215.

Still open from the previous round. The merge did not change the code these cite:

Checked, no finding

  • Credentials in the 97 new cassettes: every retained authorization value is Bearer ***; the Bedrock Messages captures keep no authorization or x-api-key; there are no query parameters; no key-shaped strings, AWS account ARNs, or email addresses.
  • scripts/recorder_proxy.py masking, including echoed and query credentials, and saving on every exit path; scripts/smoke_provider.py request budget and failure output; .env.example placeholders are empty.
  • Shared Chat normalizer changes that also affect OpenAI and vLLM: covered by test_tool_lifecycle_closes_before_following_content and test_closed_tool_call_cannot_resume.
  • serialize_messages(exclude_none=False): the translated message builders never insert None, so no new nulls reach the wire.
  • All 81 test functions added by this PR have docstrings.

Verification at e378cc0, Python 3.12.9, with main's pinned test extra (openai 2.53.0, anthropic 0.121.0, pytest 8.4.2, pytest-asyncio 1.4.0), offline replay only:

  • OPENAI_API_KEY=DUMMY_KEY python -m pytest: 3,215 passed; git status --porcelain -- tests/cassettes showed no cassette changes.
  • ruff check --output-format github --config pyproject.toml .: passed.
  • ruff format --diff --config pyproject.toml .: passed, 103 files already formatted.
  • npx prettier@3.8 --check .: passed.
  • gh pr checks 96: all five checks passed on e378cc0.
  • Not reviewed line by line: tests/core/test_provider_history_routes.py and tests/test_hosted_providers.py were skimmed, and cassette payloads were checked for secrets and headers but not audited file by file.

Comment thread python/CONVENTIONS.md Resolved
Comment thread python/ARCHITECTURE.md Resolved
Comment thread python/molto/upstream/openai.py Resolved
Comment thread python/molto/upstream/gemini.py Resolved
Comment thread python/molto/providers.py Resolved
Comment thread python/molto/upstream/gemini.py Resolved
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants