You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
feat: Opt-in pluggable allocators (draft for feedback) - #504
Draft for feedback. This prototype is shared to get review on the overall design before it
is split into smaller PRs. Several parts are independent and could land separately; see Plan.
What this does
Lets callers choose how Arrow memory is allocated, and makes the behaviours that cause problems
today opt-out rather than unavoidable. Code that passes no allocator behaves as it does now:
the new behaviours are opted into by passing an allocator, so working applications that depend
on today's behaviour are not affected.
PooledManagedAllocator: reuses arrays, bounded by PooledManagedAllocatorOptions
(default: up to 16 arrays of each power-of-two size up to 1 MiB, about 32 MiB at most), with an
option to poison disposed buffers to expose use after disposal.
CallerOwnedNativeAllocator: 64-byte-aligned native memory with no finalizer and no
memory pressure (NativeMemory.AlignedAlloc on .NET 6+).
SharedMemoryHandle keeps its finalizer only for Finalized allocators, so the default and
existing custom allocators behave as before.
Builders
Builders have a temporary allocator for growth and a long-term one for the built buffers:
No allocator (all existing code): grow in managed memory, copy once into MemoryAllocator.Default on Build. This is today's behaviour, verified to make one native
allocation per buffer.
Allocator X: grow in X and hand the working memory to the built array instead of
copying it. A builder created with its final capacity allocates exactly once:
usingvarbuilder=newInt32Array.Builder(100,allocator);Span<int>values=builder.GetSpan(100);// fill directlyusingInt32Arrayarray=builder.Advance(100).Build();
Also: capacity and allocator constructors (shipped constructors are unchanged), GetSpan and Advance on fixed-width builders, IDisposable on array builders, a validity bitmap that is
created only at the first null, and child builders, Clone, concatenation, the RecordBatch
column helpers and variant shredding building directly in their target allocator.
Contexts for IPC and Flight
Operations take an ArrowContext (allocator, codec factory, extension registry); builders take a MemoryAllocator.
IPC writers get context constructors. A codec factory in IpcOptions wins; the context's
is a fallback.
Flight: FlightClient, FlightServer, FlightSqlServer and the stream readers and
writers accept a context. It also works when the server is created by dependency injection, and
when the client is created by Grpc.Net.ClientFactory (see Questions).
ExtensionTypeRegistry.Clone is now copy-on-write, so a context's snapshot of the default
registry is cheap and never sees later changes.
Behaviour and API changes
IDisposable on array builders. Binary- and source-compatible; projects with CA2000
enabled will see new warnings. Disposal is only required with caller-owned allocators.
New dependency: Apache.Arrow.Flight now references Microsoft.Extensions.DependencyInjection.Abstractions (6.0.0) for [ActivatorUtilitiesConstructor].
Bug fixes that change output:
The builder factory honours a timestamp element type's unit and time zone; a List<timestamp[us, UTC]> builder used to produce millisecond values with no time zone.
Flight decodes schemas with the reader's extension registry instead of always the default.
Set and Swap on primitive builders check the index before writing; the exception type
is unchanged.
With an explicit allocator, buffer builder capacity is rounded up to 64 bytes.
Small builds allocate about 30% more managed memory, without being slower.
Opt-in hand-over makes a builder given its final capacity allocate once.
The remaining per-call cost of GetSpan/Advance is about 3 ns; both inline fully.
Benchmarks ending in Build() with the default native allocator are dominated by
memory-pressure GCs (Consider batching calls to GC.AddMemoryPressure #261) and are bimodal between launches, so use several launches.
Questions for reviewers
IDisposable on builders. Is that acceptable? The alternative is a second set of
builders.
PooledManagedAllocator defaults (1 MiB, 16 per size). And should lifetime become
per-buffer, to allow composite allocators such as pooled small buffers with native large ones?
The hand-over keeps up to 50% unused capacity rather than copying. Is that the right
trade-off?
Not yet done
Capacity constructors for Boolean, Binary/String, Decimal, FixedSizeBinary and nested builders
(they have allocator constructors), and GetSpan on the converting builders (Date, Time,
Timestamp).
In-place Resize for the native allocators, and trimming the pool over time.
Plan
Once the shape is agreed, split along the commits here: the independent fixes first, then
allocators and lifetimes, builders, IPC contexts, Flight, and benchmarks.
Testing
New tests check where every buffer comes from and that everything is released after disposal,
using a counting caller-owned allocator. The key behaviours were checked by deliberately breaking
them and confirming the tests fail. All suites pass on net462, net472 and net8.0, including the
pyarrow C Data tests.
Clone now shares the definitions with the original registry and copies them only when either
registry is next modified. Taking a snapshot, as every ArrowContext does of the default
registry, therefore costs almost nothing, and a snapshot never observes later changes to the
registry it was taken from. Each registry has its own lock object, since the dictionary it
guards can now be replaced.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds MemoryAllocator.Lifetime, which says what happens to a buffer that is never disposed, and
three allocators that users can opt into:
- ManagedMemoryAllocator: plain managed arrays. Spans keep their memory alive and disposal
releases nothing, which avoids the hazard in apache#397 without requiring disposal.
- PooledManagedAllocator: managed arrays kept for reuse, bounded by
PooledManagedAllocatorOptions (largest pooled size, arrays per size, total bytes), with an
option to poison disposed buffers so that use after disposal is visible.
- CallerOwnedNativeAllocator: aligned native memory with no finalizer (NativeMemory on .NET 6+).
Buffers must be disposed.
SharedMemoryHandle keeps its finalizer for allocators whose lifetime is Finalized, so the
default allocator and existing custom allocators behave as before; other lifetimes opt out of
release by collection.
Also adds a virtual MemoryAllocator.Resize, and fixesapache#50: NativeMemoryAllocator now removes
exactly the memory pressure it added when its alignment is not the default.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A builder given an allocator grows its working memory in that allocator and hands it to the
array it builds instead of copying it, so a builder created with its final capacity allocates
exactly once. A builder given no allocator behaves exactly as before: it grows in managed memory
and copies once into MemoryAllocator.Default when built.
- Buffer and array builders take an optional allocator; constructors that already shipped are
unchanged. Fixed-width builders also get capacity constructors and GetSpan/Advance for writing
values in bulk.
- Array builders are now IDisposable, which matters only for caller-owned allocators.
- Child builders, the RecordBatch column helpers, Clone, the run-end concatenation path and
variant shredding all build in the allocator their result belongs to.
- The validity bitmap of a primitive builder is created only when the first null is appended.
- Fixes the builder factory ignoring a timestamp element type's unit and time zone.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e writing
- ArrowStreamWriter and ArrowFileWriter get constructors taking an ArrowContext. A compression
codec factory in IpcOptions still takes precedence; the context's is a fallback.
- The in-memory stream reader now uses the context's allocator, and delta dictionaries keep the
historical behaviour when no allocator was given.
- Buffers the writer and decompressor allocate carry their allocator's lifetime.
- Fixes a use-after-free reported on apache#397: the writer now keeps the record batch alive until its
buffers have been written, so a collection during the write cannot free them.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- FlightServer and FlightSqlServer accept a context, which reaches every reader and writer the
server creates, including when the server is created by dependency injection.
- FlightClient gets a (CallInvoker, ArrowContext) constructor marked
[ActivatorUtilitiesConstructor], so Grpc.Net.ClientFactory can still create clients and passes
them any ArrowContext registered for dependency injection. This adds a dependency on
Microsoft.Extensions.DependencyInjection.Abstractions.
- Flight writers use the context's allocator.
- Fixes schema decoding in Flight readers, FlightInfo and FlightClient.GetSchema always using
the default extension type registry.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
DefaultPathBuilderBenchmark uses only long-standing APIs, so it can be run against earlier
versions to check the default path for regressions. FixedWidthBuilderBenchmark compares ways of
filling a fixed-width builder whose final length is known, with and without explicit allocators.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
Unresolved buffer lifetime and initialization defects can cause native use-after-free and expose uninitialized values, alongside allocation leaks and capacity overflow.
This draft adds opt-in allocator control across Arrow’s builders, memory ownership, IPC, and Flight APIs, while aiming to preserve existing behavior when no allocator is supplied.
Changes:
Introduces managed, pooled-managed, and caller-owned native allocators with explicit lifetime contracts.
Adds allocator-aware builders, buffer hand-over, direct span filling, and deterministic disposal.
Threads contexts through IPC and Flight, improves registry snapshots and memory-pressure accounting, and adds regression tests and benchmarks.
- A growing BitmapBuilder.Resize explicitly unsets the bits it adds, instead of relying on the
allocator to zero memory (GarbageMemoryAllocator does not). Clear() after a build and the
padding of copied buffers are now zeroed explicitly for the same reason.
- Appending a span that is part of the builder's own memory copies it first when the append
has to grow, since growing releases that memory. Applies to both buffer builders.
- A builder built again after a shrinking Resize no longer reuses the earlier buffer, which
would have exposed the removed values in its padding.
- BuilderMemory keeps itself alive while copying its working memory, so a collection during
allocation cannot free the source.
- Run-end concatenation releases its builders if it throws partway through.
- TimestampArray.Builder(type, capacity, allocator) checks for a null type before allocating.
- BitmapBuilder.Capacity no longer overflows for very large bitmaps.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
Approval is blocked by four active critical issues involving a cross-assembly constructor access error, incorrect buffer lifetime propagation, a potential use-after-free during copying, and disposal of pinned native memory.
- Concatenated buffers now carry their allocator's lifetime, as built buffers do. They were
created through a helper that always used the Finalized lifetime, so buffers from caller-owned
or pooled allocators could still be released by collection.
- CallerOwnedNativeAllocator tracks pins. Disposing a pinned buffer marks it disposed but
defers freeing the memory until the last pin is released, so a pinned pointer never dangles.
It also counts live allocations, for tests.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Unresolved critical memory-lifetime and disposal defects can cause use-after-free or allocator leaks, and the added constructor overload breaks source compatibility.
No open issues 🎉
4 previously missed issues
In code that hasn't changed since last review
Bitmap overlap check can free shared source memory
This has the same shared-builder lifetime problem as the fixed-width append path: source.Overlaps(Span) evaluates the writable Span property, which detaches and releases the retained buffer when the bitmap was already built. If the previously built buffer is not otherwise retained, a span captured before Build can point at finalized native memory that is freed before the later append copies it. Perform the overlap test without detaching (or copy conservatively while shared) and retain the old owner through the copy.
Overlap check can free shared source memory before copying
src/Apache.Arrow/ArrowBuffer.Builder.cs:160
The overlap check reads Span, but Span delegates to BuilderMemory.Memory, whose getter detaches and releases the builder's retained shared buffer after a prior Build. If the caller takes a span, builds (and discards) the returned buffer, then appends that span after growth, this check can release the only owner of finalized native memory before source.CopyTo runs; the raw source span can then read freed memory. Check overlap without detaching (or conservatively copy the source whenever the builder is shared) and keep the old owner alive until the copy completes.
When decompression throws or returns a length different from uncompressedLength, outputData is abandoned without being disposed. That is a leak for a context using CallerOwnedNativeAllocator (and can retain a pooled array for other allocators), so malformed or corrupt compressed input permanently consumes the allocator's memory. Wrap the allocation/use in a try/catch that disposes outputData before rethrowing, while leaving it owned by the returned ArrowBuffer on success.
Nullable context overload breaks derived constructors using base(null)
Adding the FlightSqlServer(ArrowContext?, ILoggerFactory?) overload makes existing derived constructors that call base(null) fail to compile because null now matches both the old ILoggerFactory? overload and the new ArrowContext? overload. Preserve source compatibility by avoiding two nullable reference-type overloads that can be selected by a null literal (for example, use a differently named/factory-based context entry point or require an explicit compatibility cast).
… leaks
- Appending a span that points into memory the builder is about to release now copies it
first in both buffer builders. Besides growth, this covers the first write after a build:
that releases the builder's reference to the built buffer, which may be the last one. The
overlap check reads the current memory without detaching, since detaching would release it.
- A failed decompression disposes its output buffer instead of leaking it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Responses to the third Copilot review, which listed four issues in its summary without inline threads:
Overlap checks reading Span (both buffer builders): agreed, and fixed in 4a02294. It was broader than described: the first write after a build releases the builder's reference to the built buffer, so a span captured before Build could be read after release even without growth, because CopyTo evaluates Span first. The check now reads the builder's current memory without detaching, and copies the source first whenever writing could release that memory (growth, or memory handed to a built buffer). Covered by two new tests that use a poisoning pooled allocator.
Decompression failures leak the output buffer: agreed, fixed in 4a02294. The output is now disposed if decompression throws or returns the wrong length. A new test reads corrupt compressed data through a caller-owned allocator and checks that nothing is left outstanding.
FlightSqlServer(ArrowContext?, ILoggerFactory?) makes base(null) ambiguous: I don't think this applies. The new overload has no optional parameters, so it isn't a candidate for a one-argument call; base(null) binds only to the existing FlightSqlServer(ILoggerFactory? factory = null). I confirmed that a derived server calling base(null) (and base()) compiles without error.
On the earlier macOS Verify failure: it wasn't caused by this PR, and a rerun passed. In the failing run, every project was restored under both /var/folders/… and /private/var/folders/… (on macOS, /var is a symlink to /private/var), so MSBuild treated each project as two and built them concurrently into the same outputs. Every passing run, including on main, uses /private/var only. The failure coincided with the newer runner image 20261103.629, while the passing runs and the rerun were on 20261028.624. So main's macOS release verification may start failing the same way as that image rolls out. Resolving the temporary directory in dev/release/verify_rc.sh to a canonical path (for example with pwd -P after mktemp) should prevent it; I'll leave that to a separate change.
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Human assessment is needed because the change contains multiple critical exception-safety leaks with caller-owned allocators, plus a moderate overlapping-memory corruption risk in bitmap appends.
With a caller-owned allocator these paths leaked permanently; with the default allocator a
finalizer eventually reclaimed the memory.
- The writer allocates all of a batch's temporary buffers before writing and released each
only after writing it, so a write that failed partway through (for example on a broken
connection) leaked the rest. The batch builder now releases any buffers not yet released when
it is disposed, and preparing a batch releases what it allocated if it throws. Buffers are
still released as soon as they are written.
- The readers released a message body only by handing it to a record batch. A truncated stream,
a malformed message or a failed decompression leaked it, as did a non-empty body on a schema or
unrecognized message. CreateArrowObjectFromMessage now owns the body and releases it unless the
result refers to it, and the stream readers release it if reading it fails.
- BitmapBuilder copies a source that overlaps its own memory whenever it appends it: appending
bit by bit could otherwise overwrite source bits before reading them.
Dictionary batch bodies are still never released; that needs a separate design.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A dictionary's arrays are views into its message body, but nothing owned the body: it was
reclaimed only by finalization, so with a caller-owned allocator every dictionary message
leaked its body permanently. It cannot simply be released when the dictionary is replaced or
the reader is disposed, because every record batch that uses the dictionary shares it and can
outlive both.
- Dictionary buffers now count references to the body (a MessageBody hands out sliced
SharedMemoryHandles), including buffers stored uncompressed in a compressed stream.
- Each record batch retains its own reference to the dictionary it uses, so disposing a batch
releases only that batch's references.
- The reader owns the dictionaries in its memo: it releases one when it is replaced, a delta and
the dictionary it extends once they are concatenated, and all of them when it is disposed.
- Arrays already built are released if building a batch or dictionary fails.
With the default allocator the body is now released as soon as the reader and every batch using
it are disposed, rather than at finalization.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The latest review reported no open issues, but its summary mentioned "exception-safety leaks with caller-owned allocators" and "overlapping-memory corruption risk in bitmap appends" without locations. Following those up found real problems, all of which predate this PR: with the default allocator, finalizers reclaimed the memory, so the leaks only became permanent with caller-owned allocators. Two new commits fix them, each with tests that failed before the fix.
f8346b5: release IPC buffers when reading or writing fails
Writer: all of a batch's temporary buffers (rewritten offsets, compression output) are allocated before writing, and each was released only after it was written. A stream that threw partway through (for example on a broken connection) leaked the rest, as did an exception while preparing the batch. The batch builder's Dispose now releases anything not yet released, and buffers are still released as soon as they're written. The test fails the stream at every write the writer makes, sync and async.
Readers: a message body was released only by handing it to a record batch, so a truncated stream, a failed decompression or a malformed message leaked it, as did a non-empty body on a schema or unrecognized message. CreateArrowObjectFromMessage now owns the body and releases it unless the result refers to it, and the stream readers release it if reading it fails.
Bitmap appends: appending bit by bit could overwrite source bits before reading them when the source was the builder's own memory and extended past its length. BitmapBuilder now copies a source that overlaps its memory whenever it appends it.
This one also leaked when nothing went wrong. A dictionary's arrays are views into its message body, but nothing owned the body, so every dictionary message read through a caller-owned allocator leaked it. Releasing it when the dictionary is replaced or the reader is disposed would be a use-after-free, because every batch using the dictionary shares it and can outlive both. So it now uses the same reference counting as shared buffers:
Dictionary buffers hold counted references to slices of the body, including buffers stored uncompressed in a compressed stream.
Each record batch retains its own reference to its dictionary (ArrayData.Retain(), no data copied), so disposing a batch releases only that batch's references.
The reader owns the dictionaries it keeps: it releases one when it's replaced, a delta and the dictionary it extends once they're merged, and all of them when the reader is disposed.
Tests read consecutive dictionaries through a counting caller-owned allocator and check nothing is left outstanding, and check that both batches' dictionaries stay intact after the reader is disposed and the dictionary replaced, using an allocator that overwrites freed memory. The delta path isn't covered by a test, because the .NET writer can't produce delta dictionaries.
One behaviour change with the default allocator: a dictionary body is now freed once the reader and every batch using it are disposed, rather than at finalization. Code that keeps using a dictionary after disposing both its batch and the reader would now read freed memory, which is the rule that already applies to a batch's own columns.
All tests pass on net462, net472 and net8.0, including the pyarrow interop tests.
## What's wrong
On macOS, `verify_rc.sh` can fail with dozens of spurious compile errors
in the test step: "type or namespace name 'FlightTicket' could not be
found", `K4os` not found in the compression tests, and so on, even
though the same projects built successfully moments earlier in the build
step.
The cause is the temporary directory. On macOS it's under `/var`, which
is a symlink to `/private/var`, and the script works under the symlinked
path. Tools can then see the same files under both `/var/folders/…` and
`/private/var/folders/…`. When that happens, MSBuild treats each project
as two different projects and builds them concurrently into the same
output folder, so builds read references that another build is still
writing.
## Evidence
From the `Verify (8.0.x, macos-latest)` job of the RC workflow on #504:
| Run | Runner image | Restores under `/var` | Restores under
`/private/var` | Result |
|---|---|---:|---:|---|
| [Failing
run](https://github.com/apache/arrow-dotnet/actions/runs/43776728836/job/133004801520)
| `20261103.629` | 25 | 20 | 127 compile errors |
| [Rerun of the same
commit](https://github.com/apache/arrow-dotnet/actions/runs/43776728836/attempts/2)
| `20261028.624` | 0 | 34 | Passed |
| Earlier runs, and the latest run on `main` | `20261028.624` | 0 | 34 |
Passed |
Every run reports its sandbox under `/var/folders/…`, but only the run
on the newer image ended up with both paths in use. So this is likely to
start failing on `main` too once that image becomes the default for
`macos-latest`.
## The fix
`setup_tmpdir` now resolves `VERIFY_TMPDIR` to its physical path with
`pwd -P` before anything works in it, so the shell's `$PWD` and the
operating system's view of the current directory agree. This applies
both when the script creates the directory with `mktemp` and when the
caller supplies `VERIFY_TMPDIR`. The cleanup trap removes the same
directory either way. On Linux and Windows, where the path normally has
no symlinks, nothing changes.
## Testing
- With a real symlinked directory, both branches of `setup_tmpdir`
(`mktemp` and a caller-supplied `VERIFY_TMPDIR`) resolve to the physical
path, `$PWD` is physical after the `cd`, and the cleanup trap still
removes the directory.
- `shellcheck` 0.10.0 (the version pinned in `.pre-commit-config.yaml`)
and `shfmt` report no issues.
- I couldn't run the script on macOS locally; this PR's own RC workflow
run exercises it on `macos-latest`.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
Approval is blocked by active allocator-related issues involving unnecessary bitmap allocation and potential buffer leaks during primitive array construction and IPC field loading.
If reading a field failed after some of its buffers or children were created (for example, a
decompression failure on a later buffer), the reader dropped the partially built field, leaking
its decompressed buffers and references to a shared message body. LoadField and GetChildren now
release what they built before rethrowing.
Also build the shredded variant's top-level validity bitmap with the supplied allocator, so the
result is handed over rather than copied, and dispose the builder.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Human assessment is needed because allocator-lifetime and exception-path ownership issues across multiple builders can leak buffers or retain backing allocations after disposal.
After handing its working memory to a built buffer, a builder kept its own reference to that
buffer so it could keep appending. With a caller-owned allocator, a builder that was never
disposed therefore leaked what it built permanently, even once the built array was disposed,
and the usual chained style (new Builder(allocator).Append(x).Build()) never disposes it.
A builder now holds a reference only for allocators whose lifetime is Finalized, where
collection reclaims it, so contents still survive a built array that is dropped without being
disposed. With other lifetimes it takes a reference only while reading the built buffer, which
it can do while the buffer is alive. Once the buffer has been disposed the builder's contents
are gone: using them throws ObjectDisposedException, and Clear makes the builder usable again.
Also release what the list, list-view, fixed-size list, map and binary-view builders have built
when a later part of Build fails, for example a child fixed-size list of the wrong length.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The reason will be displayed to describe this comment to others. Learn more.
🟡 Changes recommended
Several allocator-backed array builders still leak caller-owned buffers or child arrays when later allocation or construction steps fail, leaving active moderate findings that require cleanup before approval.
Builders that build several buffers or child arrays in sequence dropped the earlier ones if a
later step threw. With a pluggable allocator that is not only running out of memory: an
allocator that enforces a budget refuses allocations by throwing, and with a caller-owned
allocator the dropped buffers leaked permanently.
The primitive, Boolean, binary, large binary, fixed-size binary, run-end encoded, timestamp
with offset and variant builders now release what they built before rethrowing. A new test
builds each into an allocator that refuses its n-th allocation, for every n a successful build
makes, and checks that nothing is left outstanding in either allocator.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The reason will be displayed to describe this comment to others. Learn more.
🔵 Needs a closer look
Human assessment is needed because a critical lifetime issue in ArrowBuffer.Clone may allow copying from freed native memory, making the change too complex to approve automatically.
Both read the buffer through a span, which doesn't root the handle that
owns the memory. A buffer that was otherwise unreachable could have its
handle finalized, freeing native memory, partway through the copy or
comparison. This predates the allocator work; only buffers whose
lifetime is Finalized are affected, since other lifetimes have no
finalizer.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The review on 34dc888 reported no open issues, but its summary mentioned a lifetime issue in ArrowBuffer.Clone. It's real, though narrow and older than this PR: Clone (and Equals) read the buffer through a span, which doesn't root the handle that owns the memory. So a buffer that is otherwise unreachable, with the default Finalized lifetime, could be finalized and its native memory freed partway through the copy. 399ab94 keeps the handle alive until the read finishes. Other lifetimes weren't affected, because they have no finalizer.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this does
Lets callers choose how Arrow memory is allocated, and makes the behaviours that cause problems
today opt-out rather than unavoidable. Code that passes no allocator behaves as it does now:
the new behaviours are opted into by passing an allocator, so working applications that depend
on today's behaviour are not affected.
It addresses, fully or as an opt-in:
NativeMemoryManagerand
SharedMemoryHandlerelease memory from finalizers.GC.AddMemoryPressurecalls trigger excessive collections.Build).NativeMemoryon modern .NET.Design
Allocators and lifetimes
MemoryAllocator.Lifetimedescribes what happens to a buffer that is never disposed:AllocationLifetimeFinalizedNativeMemoryAllocator(the default, unchanged)GarbageCollectedManagedMemoryAllocator,PooledManagedAllocatorCallerOwnedCallerOwnedNativeAllocatorWhatever the lifetime, spans must not be used after their buffer is disposed. The new allocators:
ManagedMemoryAllocator: plain managed arrays, no pooling. The safe opt-out from ArrowBuffer spans do not keep native memory alive: NativeMemoryManager's finalizer can free a buffer while a Span over it is in use #397: nodisposal needed, and spans can never see freed or reused memory.
PooledManagedAllocator: reuses arrays, bounded byPooledManagedAllocatorOptions(default: up to 16 arrays of each power-of-two size up to 1 MiB, about 32 MiB at most), with an
option to poison disposed buffers to expose use after disposal.
CallerOwnedNativeAllocator: 64-byte-aligned native memory with no finalizer and nomemory pressure (
NativeMemory.AlignedAllocon .NET 6+).SharedMemoryHandlekeeps its finalizer only forFinalizedallocators, so the default andexisting custom allocators behave as before.
Builders
Builders have a temporary allocator for growth and a long-term one for the built buffers:
No allocator (all existing code): grow in managed memory, copy once into
MemoryAllocator.DefaultonBuild. This is today's behaviour, verified to make one nativeallocation per buffer.
Allocator X: grow in X and hand the working memory to the built array instead of
copying it. A builder created with its final capacity allocates exactly once:
Also: capacity and allocator constructors (shipped constructors are unchanged),
GetSpanandAdvanceon fixed-width builders,IDisposableon array builders, a validity bitmap that iscreated only at the first null, and child builders,
Clone, concatenation, theRecordBatchcolumn helpers and variant shredding building directly in their target allocator.
Contexts for IPC and Flight
Operations take an
ArrowContext(allocator, codec factory, extension registry); builders take aMemoryAllocator.IpcOptionswins; the context'sis a fallback.
FlightClient,FlightServer,FlightSqlServerand the stream readers andwriters accept a context. It also works when the server is created by dependency injection, and
when the client is created by
Grpc.Net.ClientFactory(see Questions).ExtensionTypeRegistry.Cloneis now copy-on-write, so a context's snapshot of the defaultregistry is cheap and never sees later changes.
Behaviour and API changes
IDisposableon array builders. Binary- and source-compatible; projects with CA2000enabled will see new warnings. Disposal is only required with caller-owned allocators.
Microsoft.Extensions.DependencyInjection.Abstractions(6.0.0) for[ActivatorUtilitiesConstructor].List<timestamp[us, UTC]>builder used to produce millisecond values with no time zone.SetandSwapon primitive builders check the index before writing; the exception typeis unchanged.
Bug fixes that could land on their own
ArrowStreamWriternow keeps the record batch alive while copying itsbuffers. A Release-mode reproduction crashed the runtime without the fix and is clean with it.
Debug builds extend lifetimes, so there is no unit test for it.
snapshots.
Performance
Measured with
DefaultPathBuilderBenchmark, which uses only long-standing APIs so it runsagainst
maintoo.mainin every case.Append(span)on 100 values (+9%, ~30 ns).GetSpan/Advanceis about 3 ns; both inline fully.Build()with the default native allocator are dominated bymemory-pressure GCs (Consider batching calls to GC.AddMemoryPressure #261) and are bimodal between launches, so use several launches.
Questions for reviewers
IDisposableon builders. Is that acceptable? The alternative is a second set ofbuilders.
FlightClient.Create(CallInvoker, ArrowContext)factory. Both are currently present.Create(feat: add ICompressionCodecFactory support to Flight record batch readers #285) is unreleased and now redundant; should it be removed?
ArrowContext?Builders already work this way internally.
INativeAllocationTrackeronNativeMemoryAllocator. Should it be in scope?PooledManagedAllocatordefaults (1 MiB, 16 per size). And should lifetime becomeper-buffer, to allow composite allocators such as pooled small buffers with native large ones?
trade-off?
Not yet done
(they have allocator constructors), and
GetSpanon the converting builders (Date, Time,Timestamp).
Resizefor the native allocators, and trimming the pool over time.Plan
Once the shape is agreed, split along the commits here: the independent fixes first, then
allocators and lifetimes, builders, IPC contexts, Flight, and benchmarks.
Testing
New tests check where every buffer comes from and that everything is released after disposal,
using a counting caller-owned allocator. The key behaviours were checked by deliberately breaking
them and confirming the tests fail. All suites pass on net462, net472 and net8.0, including the
pyarrow C Data tests.
🤖 Generated with Claude Code
Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com