Repository navigation
Conversation
An E8M0 scale of 0xFF is NaN (docs/designs/tensor_core_wgmma_engine.md), and the TFR datapath and its TfrFedp model make every element it scales NaN. mxfp8_to_f32, mxbf8_to_f32 and mxfp4_to_f32 instead multiplied by ldexp(1.0f, 128), which is +Inf, so a nonzero element became Inf and only a zero one NaN. These decodes are SimX's MX datapath for the non-TFR tensor-core types and the reference of the sgemm_tcu_mx, sgemm_tcu_sp_mx and sgemm_tcu_wg_mx tests. They now return NaN for that scale, as TFR does. Other scales are unchanged.
The DPI FEDP applied the two E8M0 block scales by adding their exponents into bits [30:23] of each group's fp32 product sum with an 8-bit add. When the scaled group left the fp32 exponent range the field wrapped: a group below the normal range became a huge value and an overflowing group a tiny one. A NaN or Inf product also became a finite number whenever the scale was not 1, and a 0xFF scale gave Inf instead of NaN. Each element is now multiplied by its scale as the fp32 value 2^(scale-127), the way SimX and the test references decode MX elements (rv_mxfp8tof_s and friends). Overflow goes to Inf, underflow is gradual, NaN propagates, and a 0xFF scale is NaN, as in SimX and the TFR datapath. While the scaled values stay in the normal range the result is the same as before, since scaling by a power of two is exact there.
-u scales the first K block of A and B by 2^-70 before quantization. That block then gets E8M0 scales near 2^-78 and its products fall below the fp32 normal range, so with -k64 every output is the sum of the second block plus a negligible term. The DPI backend used to wrap the exponent of such a block into a huge value. tensor_mx runs it for mxfp8, mxbf8 and mxfp4 at NT=8 on simx and on rtlsim with the DPI backend.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #491.
Summary
The DPI FEDP added the E8M0 block scales into the exponent field of each group's fp32 sum. That wrapped when the scaled sum left the fp32 range and made NaN and Inf products finite. Each MXFP8, MXBF8 and MXFP4 element is now multiplied by its scale as the fp32 value 2^(scale-127), as SimX and the test references decode MX elements. The summation order is unchanged, so in-range results stay bit for bit the same. The first commit makes those decodes (
sw/common/softfloat_ext.cpp) return NaN for a 0xFF scale, as TFR anddocs/designs/tensor_core_wgmma_engine.mdhave it; they gave Inf when every scaled element was nonzero. The last commit addssgemm_tcu_mx -uand three-k64 -ucases totensor_mx.yaml.Impact and evidence
Impact class: R2
Contracts touched: none
tensor_mx,tensor_sp_mx,tensor_wgpasses, cycles equal to mastermodel_paritymodel_parity-mxfp8(gap 0.26 %),-lsu-lanes(3.41 %), same as masterperf_gateperf_gate-mxfp8: cycles equal to master; not compared with the baseline (toolchain v3.0.1 here)logic_gateRun through
ci/blackbox.sh, not pytest. Not run: xrt, the fullci/regression.sh.known_issue:, wider tolerance, smaller args, removed case) without a root-cause plan.