Repository navigation
Optimize numeric vectors in Cereal, YAS, and Boost binary - #740
Merged
Merged
Conversation
Preserve each adapter's array length and parent metadata while transferring numeric vector payloads in bulk. Keep Cereal portable byte swapping and restrict Boost optimization to standard binary archives. Add wire-compatibility and error-path regression tests, plus deterministic numeric-vector and World benchmarks for all three formats. Validated 141 format tests, four focused Cereal size-type tests, and 180 benchmark case runs.
Collaborator
|
That's a great contribution. Thanks! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cereal, YAS, and Boost.Serialization binary currently serialize numeric vectors one element at a time. Transfer their contiguous numeric payloads in one archive operation while preserving the adapters’ existing array lengths and parent metadata.
size_tlength prefix and uses typedbinary_data, including per-element byte swapping for portable archives.size_tprefix and uses block I/O for its existing uncompressed host-endian binary archive.binary_iarchive/binary_oarchive, retaining theuint64_tlength prefix and usingload_binary/save_binary. Other archive types keep the generic parser.vector<bool>,Vectorstring, and non-arithmetic element types keep their existing paths. Schema generation remains inherited fromVectorParser.Validation:
CEREAL_SIZE_TYPE=uint32_t.NoFieldNames, truncated payloads, and excessive lengths.Added
bulk_numeric_vectorsto the existing all-format benchmark executable and Linux/macOS/Windows benchmark summaries. The results below cover a deterministicstd::vector<uint64_t>with 1.5 million elements, using ordinary reflect-cpp APIs without custom serializers or field annotations.To isolate the incremental benefit, both measured variants include #738 and the reserve-capacity changes from #739 (
ce5db49). The optimized variant adds only this PR’s bulk-I/O changes. This branch itself is based on upstream main and does not depend on #739.Run the benchmark:
./build/benchmarks/all/reflect-cpp-all-format-benchmarks \ --benchmark_filter='bulk_numeric_vectors/[^/]+/ids/' \ --benchmark_min_warmup_time=0.1 \ --benchmark_min_time=0.2s \ --benchmark_repetitions=5