[Ovi Poddar (Ovi-Poddar)]

Summary

A finite JSON reward such as 1e40 survives event storage and rollout conversion, then becomes infinity when the training adapter creates float32 outcome scores. The training reward metric still reports the finite original value, making the invalid training tensor easy to miss.

Reject non-finite rewards and fallbacks with the rollout ID, and reject out-of-range training rewards before their float32 conversion. Preserve valid rewards and configured finite fallbacks. Validation metrics remain float64 and accept large finite rewards that do not undergo the training cast.

Reproduction

On main bc3a9e80417d731744040df5d37117f9cc48005f, a CPU reproduction through the real local event API, rollout manager and training adapter produced:

source_reward_finite: true
scores_finite: false
scores: [[Infinity, 0.0]]
training/reward: 1e40

The new public-API regression pins this path and checks that the error names the affected rollout.

Verification

• Real local event API → rollout manager → CPU DataProto reproduction on main confirms the overflow.
• 24 reward regression tests pass, including the API path, both aggregation modes, finite boundaries and fallback controls.
• Related adapter/manager tests: 111 passed.
• Whole Windows CPU suite: 256 passed, 4 skipped, 9 failures in unchanged POSIX controller timeout/shutdown tests.
• Pyright with the Linux target: 0 errors, 0 warnings. Native Linux runtime validation has not been completed.
• Ruff lint/format, headers, pre-commit, strict docs and wheel/sdist builds pass.

No model inference, GPU training or learning-gain experiment was performed. The guard prevents invalid score inputs; it does not guarantee finite arithmetic throughout optimization. Invalid rewards raise rather than being silently clipped or replaced by zero.

AI assistance

Codex investigated the code, prepared the patch and regression tests, and ran the listed checks. No independent human review is claimed.

[Copilot]

🟢 Approval recommended

There are no active review issues blocking the reward-validation changes.

No open issues 🎉

What changed in this PR

This PR prevents invalid outcome rewards from entering the verl training adapter’s float32 score tensors while preserving large finite rewards in validation metrics.

Changes:

• Reject non-finite rewards and missing-reward fallbacks, with errors identifying the rollout.
• Reject training rewards outside the finite float32 range before conversion.
• Add regression coverage and document the validation rules and their limits.

File	Description
tests/​verl/​test_reward_range.py	Tests invalid rewards, finite boundaries, fallbacks, and the event API-to-training path.
docs/​20-trainer-configuration.md	Documents reward constraints and distinguishes training from validation behavior.
agentlightning/​verl/​rollout_adapter.py	Adds shared finiteness checks and training-only float32 range checks.

_{Review effort: Balanced}

[Ovi Poddar (Ovi-Poddar)]

@microsoft-github-policy-service agree