Repository navigation
[NVBug 6938144] Qualify the extension fallback in the vLLM example docs - #3068
shengliangxu wants to merge 1 commit into
Conversation
Without a writable TORCH_EXTENSIONS_DIR only the quantizers with an eager implementation, such as FP8, warn and fall back. NVFP4 on GPUs below compute capability 8.9 has no Triton kernel and calls get_cuda_ext_mx(raise_if_failed=True), so it raises instead of running slower. Say so in the README and the changelog entry, and note that the extension cache directory can be kept or deleted. Signed-off-by: Shengliang Xu <shengliangx@nvidia.com>
|
Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually. Contributors can view more details about this message here. |
|
No actionable comments were generated in the recent review.
 |
| Layer / File(s) | Summary |
|---|---|
Build and cache behavior examples/vllm_serve/README.md, CHANGELOG.rst |
The documentation explains that quantizers with eager implementations warn and fall back to slower fake quantization when extensions cannot be built. Quantizers without eager implementations fail. It also states that ext_cache contains built extensions and can be retained or deleted. |
Priority:
Estimated code review effort: 1 (Trivial) | ~5 minutes
Change: Other
Possibly related PRs
- NVIDIA/Model-Optimizer#2997: Changes
/workspaceownership and writable-path behavior that this PR documents. - NVIDIA/Model-Optimizer#2905: Changes Torch extension JIT cache directories and CI cache handling.
Suggested reviewers: sugunav14
Merge Risk: 
45f66
-
This documentation-only change is ready to merge; its documented fallback and failure behavior is consistent with the implementation.
Pre-merge checks: 
-
Check name Status Explanation Description Check Check skipped - CodeRabbit’s high-level summary is enabled. Title check The title clearly identifies the main documentation change: clarifying extension fallback behavior in the vLLM example. It is concise and related to the changeset. Docstring Coverage No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Documentation-only edits and deletions within survi… Linked Issues check Check skipped because no linked issues were found for this pull request. Out of Scope Changes check Check skipped because no linked issues were found for this pull request. Security Anti-Patterns PASS: The pull request changes only CHANGELOG.rstandexamples/vllm_serve/README.md. It adds no Python changes, dependency changes, or security-sensitive code patterns covered by this check.Full details: Docstring Coverage
-
Explanation
No eligible functions found in the changed files. Skipping docstring coverage check. Docstring coverage is scoped to functions touched by this diff. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 0 functions across 0 files. (2 skipped: 2 unsupported.)
-
Finishing Touches
-
🧪 Generate unit tests (beta)
-
- Commit to this branch
- Create a new PR
-
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Comment @coderabbitai help to get the list of available commands.
|
/ok to test 45f6654 |
|
@coderabbitai review |
✅ Action performed
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #3068 +/- ##
=======================================
Coverage 79.95% 79.95%
=======================================
Files 661 661
Lines 75663 75663
=======================================
Hits 60497 60497
Misses 15166 15166
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
What does this PR do?
Type of change: documentation
Follow-up to #2997 (merged), which stopped making
/workspaceworld-writable in the vLLM serve example image. Two review comments arrived as it merged:TORCH_EXTENSIONS_DIRModelOpt "falls back to slower eager fake quantization". That holds for quantizers with an eager implementation, such as FP8 (get_cuda_ext_fp8(raise_if_failed=False)falls back tofp8_eager). It does not hold for NVFP4 on GPUs below compute capability 8.9, such as A100: there_dynamic_block_quantize_implhas no Triton kernel and callsget_cuda_ext_mx(raise_if_failed=True), and nothing inTensorQuantizer._fake_quantizecatches theRuntimeError, so serving fails instead of running slower. Both texts now say so.ext_cachehas to be cleaned up. The README now says that it only holds the built extensions and can be kept or deleted at any time.Usage
No code change.
Testing
tensor_quant.py,extensions.py,TensorQuantizer._fake_quantize)Before your PR is "Ready for review"
Make sure you read and follow Contributor guidelines and your commits are signed (
git commit -s -S).Make sure you read and follow the Security Best Practices (e.g. avoiding hardcoded
trust_remote_code=True,torch.load(..., weights_only=False),pickle, etc.).CONTRIBUTING.md: N/AAdditional Information
NVBug 6938144. Targets the 0.48.0 release: it edits the changelog entry that #2997 adds, so on
release/0.48.0cherry-pick #2997's commit first and then this one.Summary by CodeRabbit