Conversation
Use byte-parser normalization for comparison and hashing while retaining permissive explicit conversions. Reject impossible numeric characters before renting scratch buffers for strings and fragmented sequences. Add storage, conversion, length and allocation regression coverage.
|
In the PR body (so: here) can we please be very clear and explicit exactly what scenario(s) this is addressing? Specifically, any change is a potential break to someone, so it would be good to be very clear to see what we're looking to change here. In particular, I wonder whether what we actually want here is an equality comparer instance, like |
|
If the point is to defer any lossy conversions until compare time: that's great, I just think we should be super clear about what actually changes in the PR body. |
| } | ||
| if (forComparison || seq.Length <= Format.MaxDoubleTextLen) | ||
| { | ||
| int len = checked((int)seq.Length); |
There was a problem hiding this comment.
I think we need to be mindful of what scenarios we need to allow/disallow here, and what means for the inputs; for example, if we can't simplify something that is over-long, it seems unlikely that we'd ever need the lease here - we can presumably discount such values immediately. So: what forComparison scenario forces this lease? Something with pathological numbers of leading zeros... probably isn't a useful case. I also wonder about [+-]inf etc.
There was a problem hiding this comment.
The lease preserves numeric comparison for overlong inputs, including leading zeroes; it isn't needed for inf, +inf, -inf or nan, which remain text for equality even though double casts can parse them. I tested removing it with a uniform 40-byte cutoff, but that also breaks existing long byte-array comparisons, so I haven't pushed that change. The body now gives the round-trip mismatch and breaking changes explicitly: would you prefer the bounded default behavior or an opt-in comparer?
There was a problem hiding this comment.
Again, it is hard for me to opine because the PR body doesn't really make it clear what it intends to change, behaviour-wise, in concrete terms / examples.
|
I wonder if we can simplify the intent here. For example, what if the non-comparing simplify only handled strict integers (no commas etc) and floating points that parse and round-trip back to the exact same bytes/chars, so it is never lossy, and the compare parse does what it has always done. Does that solve the underlying issue more simply? Is that what this PR is trying to do? |
|
leaving comparison parsing unchanged would still leave #3233: "1,000" equals "1000" as strings, but not its own UTF-8 bytes after a Redis round trip. This PR targets that comparison mismatch; a lossless non-comparing Simplify change is narrower and would also need separate fallbacks to preserve existing numeric casts. Would you prefer that narrower change here and a separate comparison fix? |
|
What I'd prefer is for the PR body to be very clear, with concrete examples, what it proposed to change, to what. |
|
Updated the description with concrete before/after examples, including why over-long inputs still need the rented buffer. |
Fixes #3233.
Today default equality depends on how a
RedisValueis stored. Strings are parsed withdouble.TryParse(..., NumberStyles.Any, ...), but bytes (what comes back from the server) go through the byte parser, which rejects those forms. So"1,000"equals"1000", but not its own UTF-8 bytes after a round trip.This PR makes
==/Equals,GetHashCodeandCompareTouse the byte parser for strings,byte[]/memory andReadOnlySequence<byte>alike. Explicit conversions ((long),(int),(double), ...) are left alone.Before and after
bytes "x"isEncoding.UTF8.GetBytes("x"). Hash equality matches==in every row.x == ybeforeMath.Sign(x.CompareTo(y))before"1,000""1000"or1000L"1,000""1,000""(5)""-5"or-5L"(5)""(5)"" 5 ""5"or5L" 5 "" 5 ""1,000""999""000...001"(41 chars)"1""1,000""1,000"So
new HashSet<RedisValue> { "1,000", "1000" }has 1 item today and 2 with this PR, while{ "1,000", bytes "1,000" }goes from 2 to 1. Hash codes of the affected strings change.Unchanged
"1000", bytes"1000"and1000Lare all equal, and"5." == "5","+5" == "5","1e2" == "100"still hold."000...001"(41 chars, string orbyte[]) equals"1"today and still does. That's why inputs longer thanFormat.MaxDoubleTextLenare still parsed, and whySimplifyrents a buffer for them. Strings and sequences are only copied and parsed if every char is one of0-9 + - . e E."inf","+inf"and"nan"don't equaldouble.PositiveInfinityordouble.NaN. The ordering fallback is unchanged too, so"+inf".CompareTo(double.PositiveInfinity)is still 0 while==is false.(long)"1,000"is 1000,(int)" 5 "is 5,(long)"(5)"is -5,(double)"inf"is infinity, and(long)of bytes"1,000"still throwsInvalidCastException.RedisValue.EqualityComparer.Binary.This is a behaviour change to default comparison. If you'd rather keep the default and ship this as an opt-in comparer next to
Binary, I can rework the PR that way.I got the table by running the same small program against
main(0a92ae4) and this branch (2f7af8e) on net10.0.dotnet test tests/StackExchange.Redis.Tests -c Release -f net10.0 --filter "FullyQualifiedName~RedisValue"passes on this branch (261 tests, including the newRedisValueNumericNormalizationTests). I haven't run other target frameworks or the integration suite against a real server.Checklist