Repository navigation
Conversation
Collaborator
|
Review requested:
|
`parsePackageJSON()` copies its input into a `std::string` with no spare capacity. `simdjson::pad()` then appends `SIMDJSON_PADDING` spaces, so every call reallocated and copied the whole document. Reserve room for the padding when the input is copied so that the append happens in place. The padding stays within the string's size, so simdjson never reads past the end of the string. Add a benchmark that parses the package.json files under deps/. Assisted-by: an AI coding agent Signed-off-by: Daniel Lemire <daniel@lemire.me>
anonrig
approved these changes
Nov 19, 2026
lemire
force-pushed
the
src-reserve-simdjson-padding
branch
from
November 19, 2026 18:36
9d623d0 to
2e4265a
Compare
Codecov Reportβ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #67623 +/- ##
=======================================
Coverage 90.71% 90.72%
=======================================
Files 801 801
Lines 280923 280925 +2
Branches 54240 54243 +3
=======================================
+ Hits 254839 254856 +17
+ Misses 16254 16249 -5
+ Partials 9830 9820 -10
π New features to boost your workflow:
|
Collaborator
yiguolei
pushed a commit
to apache/doris
that referenced
this pull request
Nov 21, 2026
### Background The v2 JSON reader (`be/src/format_v2/json/json_reader.cpp`) parses each document with the simdjson On-Demand parser, which needs the input followed by `SIMDJSON_PADDING` (64) readable bytes. The reader keeps a reusable `_padding_buffer` for that. In routine load and stream load (`FILE_STREAM`), a document is one pipe message, or several chunks of a chunked HTTP transfer. When `read_json_by_line` is false, a document is the whole file split. ### What problem does this PR solve? Issue Number: None Related PR: None PR Type: Performance Problem Summary: Pipe messages and whole-file documents were first copied into `_document_buffer`, then copied again into `_padding_buffer` before parsing. Whole-file reads also zero-filled a freshly allocated `_document_buffer` the size of the file. In addition, `open()` zero-filled an 8 MB `_padding_buffer` in every reader, even for small documents. This PR reads and copies documents straight into `_padding_buffer`, and grows that buffer to the largest document instead of allocating 8 MB up front. Results and error handling are unchanged. ### Details - **Why it was slow:** The cost follows from the code. For each pipe message, `_read_one_document_from_pipe` does `_document_buffer.assign(...)`, then `_parse_next_json` does `memcpy(_padding_buffer, _document_view, size)`. For a whole-file document, `_document_buffer.resize(n)` allocates and zeroes `n` bytes, `read_at` fills them, and the same `memcpy` follows. `open()` runs `_padding_buffer.resize(8 MB + 64)` every time. - **Optimization:** `_read_one_document_from_pipe` copies messages, and appends chunks, directly into `_padding_buffer`. Whole-file mode calls `read_at` directly into `_padding_buffer`. `_parse_next_json` copies only when the document is not already at the start of the buffer. That is line mode, where the line reader's span is still borrowed, and a skipped UTF-8 BOM, where it uses `memmove` because the ranges overlap. `_document_buffer` and `_padded_size` are removed. The buffer grows with `std::string::resize`, which keeps the contents and grows capacity geometrically. - **Results:** A standalone program reproduces the old and new buffer handling with simdjson On-Demand and a full traversal of every value. These are not Doris end-to-end numbers. Apple M4, clang, `-O3`, best of 30 iterations, two runs (range shown): | Workload | Before | After | |---|---:|---:| | Pipe, 20,000 messages of 89 B | 1.05β1.07 GB/s | 1.15β1.16 GB/s | | Pipe, 2,000 messages of 6.3 KB (twitter statuses) | 3.42 GB/s | 3.56β3.58 GB/s | | Whole file, twitter.json (632 KB) | 2.93β3.04 GB/s | 3.28β3.30 GB/s | | Whole file, citm_catalog.json (1.7 MB) | 3.28β3.36 GB/s | 3.75β3.96 GB/s | | Whole file, 40 MB array | 1.93β1.95 GB/s | 2.16β2.19 GB/s | A new reader's 8 MB zero-fill costs about 49 Β΅s on the same machine, while a 1 KB document parses in about 1 Β΅s. - **Applicability and cost:** Applies to `FILE_STREAM` (routine load, stream load) and to whole-file JSON. Line mode (`read_json_by_line`) keeps its single copy and only benefits from the smaller initial allocation. The padding buffer is now sized to the largest document seen by the reader, plus 64 bytes. Before, it was at least 8 MB. A reader that sees a large document grows the buffer once, as before. No new config. ### Testing - `be/test/format_v2/json/json_reader_test.cpp`: the line-mode test now checks that the document is still borrowed from the line reader and that the padding buffer fits the longest line plus padding (not 8 MB). The whole-file outer-array test checks that the document is read in place and that the buffer is the file size plus padding. - `./run-be-ut.sh --run --filter='JsonReader*'` in the `apache/doris:build-env-ldb-toolchain-latest` image (ASAN build): 35 tests from `JsonReaderTest`, `JsonReaderSplitTest` and `JsonReaderPrefetchTest` pass. - The pipe path has no unit test in this file. Its buffer logic (chunked appends that grow the buffer, a BOM in whole-file and line mode, a long line after short ones) was checked in a standalone program under ASan and UBSan. ### Release note Faster JSON routine load, stream load, and whole-file JSON load, with less memory per JSON reader. ### Check List (For Author) - Behavior changed: No - Does this need documentation: No - Pre-submission status: Partially verified: the performance numbers come from a standalone program that reproduces the reader's buffer handling, not from a Doris benchmark. ### AI Disclosure - Tool: Claude Code (Claude Opus 5.5) - Generated: code, tests, and this description - Human review: all code reviewed by Daniel - Prompts: The author asked to apply the single-copy fix of nodejs/node#67623 to Doris JSON loading, then to also look at the 8 MB buffer allocated in `open()` and at whether a newer simdjson would help (left for a separate proposal).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The
parsePackageJSON()function copies its input into astd::stringwith no sparecapacity. The
simdjson::pad()function then appendsSIMDJSON_PADDINGspaces.This change reserves room for the padding when the input is copied, so the append happens in
place.
This adds
benchmark/module/parse-package-json.js, which parses theunique package.json files under
deps/, grouped by file size.Benchmarks
benchmark/compare.js, 30 runs, old =main(108cbef), new = this PR,Welch's t-test.
Intel Xeon Gold 6548N (Emerald Rapids), Linux, clang 23:
Intel Xeon Gold 6338 (Ice Lake), Linux, clang 23:
Apple M4 Max, macOS, Apple clang 17:
Refs: #55584
Refs: #55591
Refs: #67620