Repository navigation
Add bfloat16 and float8e4m3 to the spec - #956
mklimenko-nv wants to merge 2 commits into
Conversation
fdwr
left a comment
There was a problem hiding this comment.
✌️ Looks good Mikhail. Thanks. Minor comments...
| : <dfn>bfloat16</dfn> | ||
| :: 16-bit "brain" floating point, with 1 sign bit, 8 exponent bits (exponent bias 127) and 7 trailing significand bits. It has the same exponent range as {{MLOperandDataType/"float32"}}, and infinities, NaNs, signed zeros and subnormal values are represented as in the binary interchange formats of [[IEEE-754-2019]]. The bit layout of a value in this format is identical to the 16 most significant bits of the binary32 encoding of the same value. | ||
| : <dfn>float8e4m3</dfn> | ||
| :: 8-bit floating point in the E4M3 format of the OCP 8-bit Floating Point Specification [[OCP-OFP8]], with 1 sign bit, 4 exponent bits (exponent bias 7) and 3 trailing significand bits. Unlike the other floating point data types, this format has no representation for infinities; the largest finite magnitude is 448, and NaN is encoded with all exponent and significand bits set to 1. Signed zeros and subnormal values are supported. Conversions to this format are saturating: values whose magnitude exceeds 448, including infinities, convert to the largest finite value with the same sign, see {{MLGraphBuilder/cast()}}. |
There was a problem hiding this comment.
this format has no representation for infinities
Oh yeah, I thought this was the one with infinity for a moment (float8m3e4s1_t), but it's actually (float8m3e4s1fn_t) from the earlier table.
- float8m3e4s1_t - mantissa: 3 bits, exponent: 4 bits, sign: 1 bit, infinities, NaNs
- ⭐ float8m3e4s1fn_t - mantissa: 3 bits, exponent: 4 bits, sign: 1 bit, infinities, NaNs
- float8m3e4s1fnuz_t - mantissa: 3 bits, exponent: 4 bits, sign: 1 bit, no infinities, NaN as -0
Double checking D3D12_LINEAR_ALGEBRA_DATATYPE_FLOAT8_E4M3FN and ONNX TensorProto.DataType.FLOAT8E4M3FN and PyTorch, indeed there is no infinity. Additionally verified here:
- https://onnx.ai/onnx/technical/float8.html
- https://www.opencompute.org/documents/ocp-microscaling-formats-mx-v1-0-spec-final-pdf
- https://asawicki.info/articles/fp8_tables.php
- https://arxiv.org/pdf/2209.05433
(resolve me)
There was a problem hiding this comment.
Yes, it's the fn flavor. To be on the safe side, let's pin the encoding down (no infinities, NaN = S.1111.111, max 448), so the enum value is unambiguous whatever we call it.
Personally, I'd prefer to keep the unsuffixed version to match the approach in OCP spec itself, TensorRT, OpenVINO, etc. There is a camp that prefers explicit suffixes (PyTorch, ONNX, CoreML), so both approaches have industry backing. Since we specify explicitly what the data type means, there shouldn't be misreadings, but we can adjust it.
There was a problem hiding this comment.
I'd prefer to keep the unsuffixed version to match the approach in OCP spec itself, FP8 (E4M3)
...
There is a camp that prefers explicit suffixes (PyTorch, ONNX, CoreML {and Direct3D})
Hmm, I wager a reason why people appended the "fn" is because the spec nomenclature was ambiguous and surprising, given that same spec with "E4M3" also has "E5M2", with no visible distinction that they differ not only it bit allocation (which is apparent), but also infinity support and NaN representations. The type is an odd outlier in that regard compared to every other WebNN float type. Anyway, a minor naming change can be a separate issue.
|
|
||
| {{MLOperandDataType/"bfloat16"}} and {{MLOperandDataType/"float8e4m3"}} are not among the [=/required data types=] of any operation. Support can be queried per operation using {{MLContext/opSupportLimits()}}. | ||
|
|
||
| Note: The {{MLOperandDataType/"float8e4m3"}} format corresponds to the `float8_e4m3fn` type of PyTorch and the `FLOAT8E4M3FN` type of ONNX. It is distinct from E4M3 encodings that reserve codes for infinities and from the "FNUZ" variants without negative zero; neither is represented by this enum. |
There was a problem hiding this comment.
You think we should make that more apparent in the naming, that this type is finite and doesn't support the usual norms expected of a floating-point type?
float8e4m3 -> float8e4m3fn
(just seeing float8e4m3 by itself, I would have expected infinity too)
| 1. If |a|'s [=MLOperand/dataType=] is not {{MLOperandDataType/"uint8"}}, then [=exception/throw=] a {{TypeError}}. | ||
| 1. If |op| is one of "isNaN", "isInfinite", then: | ||
| 1. If |a|'s [=MLOperand/dataType=] is not one of « {{MLOperandDataType/"float32"}}, {{MLOperandDataType/"float16"}} », then [=exception/throw=] a {{TypeError}}. | ||
| 1. If |a|'s [=MLOperand/dataType=] is not one of « {{MLOperandDataType/"float32"}}, {{MLOperandDataType/"float16"}}, {{MLOperandDataType/"bfloat16"}} », then [=exception/throw=] a {{TypeError}}. |
There was a problem hiding this comment.
Shouldn't float8 types also be present for isNaN(), since float8e4m3fn supports NaN's?
We could also add a helper function (like with blockwise broadcastable https://webmachinelearning.github.io/webnn/#blockwise-broadcastable) that returns true for floating point types, rather than listing them all out 🤷♂️.
Now, isInfinite is more interesting, since that particular float8 format doesn't support infinities, but maybe the function should still succeed (since isNaN and isInfinite should generically work with any floating point data type) but just return a false tensor in that case 🤔.
There was a problem hiding this comment.
Yes, sounds fair, fp8 is now allowed in both isNaN() and isInfinite().
On the topic of helper function I'm not sure, since the scope of fp8 is somewhat limited. But I don't have a strong opinion on that and it should be straightforward to add.
This PR adds WebNN spec wording to support
bfloat16andfloat8e4m3, according to #938Detailed changes:
MLOperandDataType: new values"bfloat16"and"float8e4m3", plus a definition list describing every value.float8e4m3is the OCP OFP8 E4M3 encoding (1/4/3 bits, bias 7, no infinities, max ±448, NaN =S.1111.111), mirroring the PyTorchfloat8_e4m3fnand ONNXFLOAT8E4M3FN. Neither new type is a required data type for any operation, support is discoverable throughopSupportLimits().bfloat16is added to the allowed data types whereverfloat16is allowed. The required columns are untouched.float8e4m3is allowed ondequantizeLinear(input,zeroPoint),quantizeLinear(zeroPoint, output) andcast, and is implicitly allowed on operands whose allowed types areany. It is intentionally not listed on compute operators:fp8compute is expressed through the Q/DQ pattern and realized by backend fusion, which a Note next to the enum spells out.cast(): the behavior table gainsbfloat16and afloat8e4m3target column. Conversions tofloat8e4m3saturate (out-of-range and infinite values become ±448, NaN is preserved), matching OFP8's saturating mode and the ONNXCastdefault. NaN handling is now stated in every float-source cell.ConvertToFloat(used forMLNumbercasting) takes the data type instead of a bit length and handles the two new formats, RNE and saturation match OFP8 1.1 Appendix A.quantizeLinear/dequantizeLinear: states the type promotion (arithmetic happens in the scale/input float type) and the float8 formulacast(input / scale + cast(zeroPoint, input.dataType), zeroPoint.dataType). ThequantizeLinearemulation branches on integer vs. floating point output and clamps to the output type's range instead of a hardcoded 0...255. A note points out thatfp8quantization is typically symmetric (zeroPoint= zeros).bfloat16<->Uint16Array,float8e4m3<->Uint8Array(native-endian bit patterns;Uint8Arrayremains the generic view for any type).validate buffer with descriptorneeded no change since it is table-driven. The stale "Float16Array is Stage 3" note is refreshed.Also fixed while here: the isNaN/isInfinite tensor-limits table said
anywhile the method steps restricted to floating point types; the table now matches the steps.Preview | Diff