[JackyHsu-X]

What does this PR do?

When fetching an observations page exhausts its retries, batch evaluation returns an incomplete result and resume token before flushing scores from earlier pages. Flush before that return so the failure path honors the same end-of-run delivery behavior as successful runs.

Extend the existing recovery test to cover first-page and later-page failures, flushing before return, and successful resumption without skipping observations. A public-API reproduction against a local HTTP server produced two scores but delivered zero before return on unchanged code; with this change both are delivered and the score queue is empty.

Fixes langfuse/langfuse#19419

Type of change

• Bug fix
• New feature
• Breaking change
• Refactor
• Documentation update
• Tooling, CI, or repo maintenance

Verification

uv sync --locked --python 3.13
uv run --frozen pytest tests/unit/test_batch_evaluation.py
uv run --frozen ruff check .
uv run --frozen ruff format --check langfuse/batch_evaluation.py tests/unit/test_batch_evaluation.py
uv run --frozen mypy langfuse --no-error-summary
uv run --frozen pre-commit run --files langfuse/batch_evaluation.py tests/unit/test_batch_evaluation.py

• The two failure-path cases fail on unchanged production code; all 18 batch-evaluation tests pass with the fix.
• Full unit suite on Linux/Python 3.12: 973 passed, 2 skipped. Dummy LANGFUSE_* settings were supplied for the prompt fixtures; no live Langfuse or model service was used.
• The public synchronous API was also exercised with the real API client, background score consumer and a loopback HTTP server (page one succeeds, page two returns 503).
• Full Windows/Python 3.13 suite: 951 passed, 2 skipped, 3 failed and 19 setup errors. All 22 unsuccessful cases reproduce on untouched main: prompt credentials are absent, child processes resolve a different Python, and two tests assume POSIX paths/time.tzset. These unrelated tests are unchanged.
• Ruff, mypy and applicable pre-commit hooks pass. Live-backend e2e tests, live-provider tests and the remaining Python matrix were not run locally.

Checklist

• I self-reviewed the diff using code_review.md.
• I added or updated tests for behavior changes.
• I updated docs, examples, or .env.template if needed. Existing public documentation already promises automatic flushing at the end of batch evaluation.
• I did not hand-edit generated files; if generated files changed, I used the upstream regeneration path.
• I did not commit secrets or credentials.

AI assistance: OpenAI Codex investigated, generated and self-reviewed the patch, and ran the local checks under the account owner's authorization. No separate human review is claimed.

Confidence Score: 5/5

This PR appears safe to merge.

Summary

Flushes queued scores before returning an incomplete batch result after page fetching fails.

• Batch evaluation flushes scores before returning after a fetch failure.

_{Reviews (1) · Last reviewed commit: "fix(evaluations): flush scores when batc..." · Reviewed by Greptile}

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[CLAassistant]

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution.
1 out of 2 committers have signed the CLA.

✅ JackyHsu-X
❌ hassiebp
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[hassiebp]

thanks for your contribution!