Repository navigation
time: fix parse_float with ten or more fractional digits - #5575
Conversation
NumericTimeParser#parse_float sliced 10 characters out of the fraction (nsec_s[0..9]), so a value with 10 or more decimal digits stored a nanosecond field of 1_000_000_000 or more. EventTime assumes nsec fits in a second, so such a value reports the wrong second everywhere: to_f and to_r are off, to_time normalises into the next second, %N prints ten digits, and the pair is forwarded as-is through to_msgpack_ext, so a downstream fluentd inherits it. It also truncated the whole part towards zero. EventTime keeps nsec non-negative, so a negative timestamp has to floor to the second below and carry the remainder, exactly like Time#to_i and Time#nsec. Instead "-1234.5" became (-1234, 500000000) -- one second late, and "-0.5" became (0, 500000000), i.e. +0.5. Every pre-1970 timestamp parsed with time_type float was wrong, and the parser disagreed with the string based parser for the same instant. Both are now checked against the decomposition Time.at(Rational(value)) gives: a randomised corpus of 200_000 positive and negative values with 1 to 9 fraction digits had 98_614 wrong decompositions before this change and 0 after, and nsec stays in [0, 1_000_000_000) for fractions of up to 20 digits. Signed-off-by: Qingyuan Wang <162230229@nuaa.edu.cn>
Watson1978
left a comment
There was a problem hiding this comment.
Thanks for finding this. I reviewed 606851a on Ruby 4.0.9 (Linux).
Keeping nine digits of the fraction looks good. It fixes the out-of-range nsec.
You are right that master parses -0.5 as +0.5. However, the negative-timestamp adjustment changes behavior in ways that interact badly with existing formatting and MessagePack serialization. Details are in the inline comments. Let's keep this PR focused on the nine-digit limit and address negative timestamps separately.
Negative timestamps require a separate discussion of formatting and serialization compatibility. EventTime is also defined in the Forward Protocol, so changing its encoding affects other implementations too.
- MUST: Drop the negative-timestamp adjustment and its test. Both inline comments have suggestions for this.
- MUST: Update the PR description and the release note to reflect the narrower scope.
With the two suggestions applied, test/test_time_parser.rb (31 tests) and test/test_time_formatter.rb (77 tests) pass.
Per review: dropping the negative-timestamp adjustment again. It changed how negative values interact with out_file formatting and MessagePack serialization, and EventTime is also defined in the Forward protocol, so changing its encoding needs a separate discussion. This keeps the PR focused on the nine-digit fraction limit. Signed-off-by: NUAAwqy <NUAAwqy@users.noreply.github.com>
|
Thanks for the thorough review — both MUSTs are applied in 5173c74:
|
Watson1978
left a comment
There was a problem hiding this comment.
Thanks for the quick update.
LGTM
…gits (#5575) (#5580) **Which issue(s) this PR fixes**: Backport #5575 None — self-found while auditing `NumericTimeParser#parse_float`. **What this PR does / why we need it**: `NumericTimeParser#parse_float` (`lib/fluent/time.rb`) kept **ten** characters of the fractional part (`nsec_s[0..9]`), so a timestamp with ten or more decimal digits stored an `nsec` of `1_000_000_000` or more. `EventTime` assumes `nsec` fits inside a second: `to_f`/`to_r` report the wrong instant, `to_time` normalises into the next second, `%N` prints ten digits, and `to_msgpack_ext` forwards the oversized field as-is, so other fluentd nodes inherit it. Of fraction lengths 1..20, eleven produce an out-of-range `nsec`: ``` Fluent::NumericTimeParser.new(:float).parse("1700000000.1234567890").to_f # => 1700000001.2345679 (should be 1700000000.123456789) ``` **The fix**: keep the first nine fraction digits (`nsec_s[0, 9]`, sub-nanosecond digits dropped, which is what `Time#nsec` does). Padding and caching behaviour are untouched, so the hot path keeps its shape. Per the review, the negative-timestamp adjustment introduced earlier in this PR was dropped again: it changed how negative values interact with existing formatting and MessagePack serialization, and `EventTime` is also defined in the [Forward protocol](https://github.com/fluent/fluentd/wiki/Forward-Protocol-Specification-v1#eventtime-ext-format), so changing its encoding needs a separate discussion. Negative values now parse exactly as before this PR. **Validation** (Ruby 3.3.6, `TZ=UTC`): | check | result | | --- | --- | | new test (`test/test_time_parser.rb`) | 31 tests, **0 failures, 0 errors** | | regression slice: test_time_formatter 77, test_event_time 22, test_formatter 36, test_mixin 21, plugin/test_parser 23, plugin/test_parser_json 66, plugin/test_parser_apache 3, plugin_helper/test_compat_parameters 10, plugin/test_out_exec_filter 22 | all **0 failures, 0 errors** | | rubocop 1.92.1 + rubocop-performance 1.27.0 | no offenses | `plugin/test_in_exec` reports 2 failures (`can handle non-ascii characters`) that reproduce identically at `f984d65` with the fix reverted — the flaky encoding tests pinned in #5562/#5572, unrelated to time parsing. **Docs Changes**: None needed. **Release Note**: time: fix parse_float with ten or more fractional digits * With `time_type float`, a time string with ten or more fractional digits was parsed into a wrong time. Digits beyond nanoseconds are now dropped. Signed-off-by: Qingyuan Wang <162230229@nuaa.edu.cn> Signed-off-by: NUAAwqy <NUAAwqy@users.noreply.github.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Qingyuan Wang <146524024+NUAAwqy@users.noreply.github.com> Co-authored-by: NUAAwqy <NUAAwqy@users.noreply.github.com>
Which issue(s) this PR fixes:
None — self-found while auditing
NumericTimeParser#parse_float.What this PR does / why we need it:
NumericTimeParser#parse_float(lib/fluent/time.rb) kept ten characters of the fractional part (nsec_s[0..9]), so a timestamp with ten or more decimal digits stored annsecof1_000_000_000or more.EventTimeassumesnsecfits inside a second:to_f/to_rreport the wrong instant,to_timenormalises into the next second,%Nprints ten digits, andto_msgpack_extforwards the oversized field as-is, so other fluentd nodes inherit it. Of fraction lengths 1..20, eleven produce an out-of-rangensec:The fix: keep the first nine fraction digits (
nsec_s[0, 9], sub-nanosecond digits dropped, which is whatTime#nsecdoes). Padding and caching behaviour are untouched, so the hot path keeps its shape.Per the review, the negative-timestamp adjustment introduced earlier in this PR was dropped again: it changed how negative values interact with existing formatting and MessagePack serialization, and
EventTimeis also defined in the Forward protocol, so changing its encoding needs a separate discussion. Negative values now parse exactly as before this PR.Validation (Ruby 3.3.6,
TZ=UTC):test/test_time_parser.rb)plugin/test_in_execreports 2 failures (can handle non-ascii characters) that reproduce identically atf984d65with the fix reverted — the flaky encoding tests pinned in #5562/#5572, unrelated to time parsing.Docs Changes:
None needed.
Release Note:
time: fix parse_float with ten or more fractional digits
time_type float, a time string with ten or more fractional digits was parsed into a wrong time. Digits beyond nanoseconds are now dropped.