Skip to content

feat: Opt-in pluggable allocators (draft for feedback) - #504

Draft
CurtHagenlocher wants to merge 15 commits into
apache:mainfrom
CurtHagenlocher:prototype/pluggable-allocator
Draft

CurtHagenlocher wants to merge 15 commits into
apache:mainfrom
CurtHagenlocher:prototype/pluggable-allocator

Conversation

@CurtHagenlocher

Copy link
Copy Markdown
Contributor

Draft for feedback. This prototype is shared to get review on the overall design before it
is split into smaller PRs. Several parts are independent and could land separately; see
Plan.

What this does

Lets callers choose how Arrow memory is allocated, and makes the behaviours that cause problems
today opt-out rather than unavoidable. Code that passes no allocator behaves as it does now:
the new behaviours are opted into by passing an allocator, so working applications that depend
on today's behaviour are not affected.

It addresses, fully or as an opt-in:

Design

Allocators and lifetimes

MemoryAllocator.Lifetime describes what happens to a buffer that is never disposed:

AllocationLifetime Undisposed buffer Example
Finalized Released by a finalizer, even while a span is in use (#397) NativeMemoryAllocator (the default, unchanged)
GarbageCollected Collected; a span keeps it alive ManagedMemoryAllocator, PooledManagedAllocator
CallerOwned Leaked; must be disposed CallerOwnedNativeAllocator

Whatever the lifetime, spans must not be used after their buffer is disposed. The new allocators:

SharedMemoryHandle keeps its finalizer only for Finalized allocators, so the default and
existing custom allocators behave as before.

Builders

Builders have a temporary allocator for growth and a long-term one for the built buffers:

  • No allocator (all existing code): grow in managed memory, copy once into
    MemoryAllocator.Default on Build. This is today's behaviour, verified to make one native
    allocation per buffer.

  • Allocator X: grow in X and hand the working memory to the built array instead of
    copying it. A builder created with its final capacity allocates exactly once:

    using var builder = new Int32Array.Builder(100, allocator);
    Span<int> values = builder.GetSpan(100);   // fill directly
    using Int32Array array = builder.Advance(100).Build();

Also: capacity and allocator constructors (shipped constructors are unchanged), GetSpan and
Advance on fixed-width builders, IDisposable on array builders, a validity bitmap that is
created only at the first null, and child builders, Clone, concatenation, the RecordBatch
column helpers and variant shredding building directly in their target allocator.

Contexts for IPC and Flight

Operations take an ArrowContext (allocator, codec factory, extension registry); builders take a
MemoryAllocator.

  • IPC writers get context constructors. A codec factory in IpcOptions wins; the context's
    is a fallback.
  • Flight: FlightClient, FlightServer, FlightSqlServer and the stream readers and
    writers accept a context. It also works when the server is created by dependency injection, and
    when the client is created by Grpc.Net.ClientFactory (see Questions).
  • ExtensionTypeRegistry.Clone is now copy-on-write, so a context's snapshot of the default
    registry is cheap and never sees later changes.

Behaviour and API changes

  • IDisposable on array builders. Binary- and source-compatible; projects with CA2000
    enabled will see new warnings. Disposal is only required with caller-owned allocators.
  • New dependency: Apache.Arrow.Flight now references
    Microsoft.Extensions.DependencyInjection.Abstractions (6.0.0) for
    [ActivatorUtilitiesConstructor].
  • Bug fixes that change output:
    • The builder factory honours a timestamp element type's unit and time zone; a
      List<timestamp[us, UTC]> builder used to produce millisecond values with no time zone.
    • Flight decodes schemas with the reader's extension registry instead of always the default.
  • Set and Swap on primitive builders check the index before writing; the exception type
    is unchanged.
  • With an explicit allocator, buffer builder capacity is rounded up to 64 bytes.

Bug fixes that could land on their own

Performance

Measured with DefaultPathBuilderBenchmark, which uses only long-standing APIs so it runs
against main too.

  • Default path (no allocator): parity or faster than main in every case.
    • Single-threaded: 2–83% faster, except Append(span) on 100 values (+9%, ~30 ns).
    • Parallel builder growth: about 25% faster.
    • Small builds allocate about 30% more managed memory, without being slower.
  • Opt-in hand-over makes a builder given its final capacity allocate once.
  • The remaining per-call cost of GetSpan/Advance is about 3 ns; both inline fully.
  • Benchmarks ending in Build() with the default native allocator are dominated by
    memory-pressure GCs (Consider batching calls to GC.AddMemoryPressure #261) and are bimodal between launches, so use several launches.

Questions for reviewers

  1. IDisposable on builders. Is that acceptable? The alternative is a second set of
    builders.
  2. The DI abstractions dependency for Flight, versus offering only the existing
    FlightClient.Create(CallInvoker, ArrowContext) factory. Both are currently present. Create
    (feat: add ICompressionCodecFactory support to Flight record batch readers #285) is unreleased and now redundant; should it be removed?
  3. Should the temporary and long-term allocators be exposed publicly, or on ArrowContext?
    Builders already work this way internally.
  4. Consider batching calls to GC.AddMemoryPressure #261 batching could be added later as an opt-in INativeAllocationTracker on
    NativeMemoryAllocator. Should it be in scope?
  5. PooledManagedAllocator defaults (1 MiB, 16 per size). And should lifetime become
    per-buffer, to allow composite allocators such as pooled small buffers with native large ones?
  6. The hand-over keeps up to 50% unused capacity rather than copying. Is that the right
    trade-off?

Not yet done

  • Capacity constructors for Boolean, Binary/String, Decimal, FixedSizeBinary and nested builders
    (they have allocator constructors), and GetSpan on the converting builders (Date, Time,
    Timestamp).
  • The construction of arrays should be more flexible #97's allocation without zero-filling, read-write arrays, and more flexible nested builders.
  • In-place Resize for the native allocators, and trimming the pool over time.

Plan

Once the shape is agreed, split along the commits here: the independent fixes first, then
allocators and lifetimes, builders, IPC contexts, Flight, and benchmarks.

Testing

New tests check where every buffer comes from and that everything is released after disposal,
using a counting caller-owned allocator. The key behaviours were checked by deliberately breaking
them and confirming the tests fail. All suites pass on net462, net472 and net8.0, including the
pyarrow C Data tests.

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 5.5 noreply@anthropic.com

CurtHagenlocher and others added 6 commits November 15, 2026 14:23
Clone now shares the definitions with the original registry and copies them only when either
registry is next modified. Taking a snapshot, as every ArrowContext does of the default
registry, therefore costs almost nothing, and a snapshot never observes later changes to the
registry it was taken from. Each registry has its own lock object, since the dictionary it
guards can now be replaced.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Adds MemoryAllocator.Lifetime, which says what happens to a buffer that is never disposed, and
three allocators that users can opt into:

- ManagedMemoryAllocator: plain managed arrays. Spans keep their memory alive and disposal
  releases nothing, which avoids the hazard in apache#397 without requiring disposal.
- PooledManagedAllocator: managed arrays kept for reuse, bounded by
  PooledManagedAllocatorOptions (largest pooled size, arrays per size, total bytes), with an
  option to poison disposed buffers so that use after disposal is visible.
- CallerOwnedNativeAllocator: aligned native memory with no finalizer (NativeMemory on .NET 6+).
  Buffers must be disposed.

SharedMemoryHandle keeps its finalizer for allocators whose lifetime is Finalized, so the
default allocator and existing custom allocators behave as before; other lifetimes opt out of
release by collection.

Also adds a virtual MemoryAllocator.Resize, and fixes apache#50: NativeMemoryAllocator now removes
exactly the memory pressure it added when its alignment is not the default.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A builder given an allocator grows its working memory in that allocator and hands it to the
array it builds instead of copying it, so a builder created with its final capacity allocates
exactly once. A builder given no allocator behaves exactly as before: it grows in managed memory
and copies once into MemoryAllocator.Default when built.

- Buffer and array builders take an optional allocator; constructors that already shipped are
  unchanged. Fixed-width builders also get capacity constructors and GetSpan/Advance for writing
  values in bulk.
- Array builders are now IDisposable, which matters only for caller-owned allocators.
- Child builders, the RecordBatch column helpers, Clone, the run-end concatenation path and
  variant shredding all build in the allocator their result belongs to.
- The validity bitmap of a primitive builder is created only when the first null is appended.
- Fixes the builder factory ignoring a timestamp element type's unit and time zone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…e writing

- ArrowStreamWriter and ArrowFileWriter get constructors taking an ArrowContext. A compression
  codec factory in IpcOptions still takes precedence; the context's is a fallback.
- The in-memory stream reader now uses the context's allocator, and delta dictionaries keep the
  historical behaviour when no allocator was given.
- Buffers the writer and decompressor allocate carry their allocator's lifetime.
- Fixes a use-after-free reported on apache#397: the writer now keeps the record batch alive until its
  buffers have been written, so a collection during the write cannot free them.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- FlightServer and FlightSqlServer accept a context, which reaches every reader and writer the
  server creates, including when the server is created by dependency injection.
- FlightClient gets a (CallInvoker, ArrowContext) constructor marked
  [ActivatorUtilitiesConstructor], so Grpc.Net.ClientFactory can still create clients and passes
  them any ArrowContext registered for dependency injection. This adds a dependency on
  Microsoft.Extensions.DependencyInjection.Abstractions.
- Flight writers use the context's allocator.
- Fixes schema decoding in Flight readers, FlightInfo and FlightClient.GetSchema always using
  the default extension type registry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
DefaultPathBuilderBenchmark uses only long-standing APIs, so it can be run against earlier
versions to check the default path for regressions. FixedWidthBuilderBenchmark compares ways of
filling a fixed-width builder whose final length is known, with and without explicit allocators.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Unresolved buffer lifetime and initialization defects can cause native use-after-free and expose uninitialized values, alongside allocation leaks and capacity overflow.

7 open issues
What changed in this PR

This draft adds opt-in allocator control across Arrow’s builders, memory ownership, IPC, and Flight APIs, while aiming to preserve existing behavior when no allocator is supplied.

Changes:

  • Introduces managed, pooled-managed, and caller-owned native allocators with explicit lifetime contracts.
  • Adds allocator-aware builders, buffer hand-over, direct span filling, and deterministic disposal.
  • Threads contexts through IPC and Flight, improves registry snapshots and memory-pressure accounting, and adds regression tests and benchmarks.
File Description
test/​Apache.Arrow.Tests/​ExtensionTypeRegistryTests.cs Tests registry snapshot isolation and concurrency.
test/​Apache.Arrow.Tests/​DefaultAllocationBehaviorTests.cs Tests default allocation and lifetime compatibility.
test/​Apache.Arrow.Tests/​BuilderHandOverTests.cs Tests ownership transfer, builder reuse, and allocation counts.
test/​Apache.Arrow.Tests/​ArrowArrayConcatenatorTests.cs Updates builder calls for allocator parameters.
test/​Apache.Arrow.Flight.TestWeb/​TestFlightServer.cs Accepts an injected server context.
test/​Apache.Arrow.Flight.Tests/​TestWebFactory.cs Supports context registration in test hosting.
test/​Apache.Arrow.Flight.Tests/​FlightContextTests.cs Tests Flight context propagation and allocator cleanup.
test/​Apache.Arrow.Benchmarks/​FixedWidthBuilderBenchmark.cs Compares allocator-aware fixed-width filling strategies.
test/​Apache.Arrow.Benchmarks/​DefaultPathBuilderBenchmark.cs Benchmarks existing default builder usage.
src/​Apache.Arrow/​RecordBatch.Builder.cs Passes allocators to column builders and disposes them.
src/​Apache.Arrow/​Memory/​SlicedMemoryOwner.cs Exposes an owned buffer prefix without copying.
src/​Apache.Arrow/​Memory/​SharedMemoryOwner.cs Records whether shared handles should finalize.
src/​Apache.Arrow/​Memory/​SharedMemoryHandle.cs Suppresses finalization for opt-in lifetimes.
src/​Apache.Arrow/​Memory/​PooledManagedAllocatorOptions.cs Defines pool limits and disposal poisoning options.
src/​Apache.Arrow/​Memory/​PooledManagedAllocator.cs Adds bounded managed-array pooling.
src/​Apache.Arrow/​Memory/​NativeMemoryAllocator.cs Corrects alignment-dependent memory-pressure removal.
src/​Apache.Arrow/​Memory/​MemoryAllocator.cs Adds lifetime metadata and resizing support.
src/​Apache.Arrow/​Memory/​ManagedMemoryAllocator.cs Adds a non-pooling managed allocator.
src/​Apache.Arrow/​Memory/​CallerOwnedNativeAllocator.cs Adds explicitly disposed aligned native allocation.
src/​Apache.Arrow/​Memory/​BuilderMemory.cs Implements allocator-backed growth and buffer hand-over.
src/​Apache.Arrow/​Memory/​AllocationLifetime.cs Defines allocation lifetime contracts.
src/​Apache.Arrow/​Ipc/​DecompressingBufferCreator.cs Preserves allocator lifetime on decompressed buffers.
src/​Apache.Arrow/​Ipc/​ArrowStreamWriter.cs Adds contexts and keeps batches alive during writes.
src/​Apache.Arrow/​Ipc/​ArrowStreamReader.cs Forwards context allocation to memory readers.
src/​Apache.Arrow/​Ipc/​ArrowReaderImplementation.cs Preserves allocator intent for dictionary deltas.
src/​Apache.Arrow/​Ipc/​ArrowMemoryReaderImplementation.cs Accepts an allocator for memory-backed reading.
src/​Apache.Arrow/​Ipc/​ArrowFileWriter.cs Adds context-based writer construction.
src/​Apache.Arrow/​ExtensionTypeRegistry.cs Implements copy-on-write registry snapshots.
src/​Apache.Arrow/​ArrowContext.cs Documents allocator, codec, and registry behavior.
src/​Apache.Arrow/​ArrowBuffer.cs Carries allocation lifetimes and optimizes cloning.
src/​Apache.Arrow/​ArrowBuffer.Builder.cs Adds allocator-backed storage, span filling, and disposal.
src/​Apache.Arrow/​ArrowBuffer.BitmapBuilder.cs Adds allocator-aware bitmap storage and disposal.
src/​Apache.Arrow/​Arrays/​VariantArray.cs Adds allocator propagation and builder disposal.
src/​Apache.Arrow/​Arrays/​UInt8Array.cs Adds capacity and allocator construction.
src/​Apache.Arrow/​Arrays/​UInt64Array.cs Adds capacity and allocator construction.
src/​Apache.Arrow/​Arrays/​UInt32Array.cs Adds capacity and allocator construction.
src/​Apache.Arrow/​Arrays/​UInt16Array.cs Adds capacity and allocator construction.
src/​Apache.Arrow/​Arrays/​TimestampWithOffsetArray.cs Propagates allocation to child builders and disposes them.
src/​Apache.Arrow/​Arrays/​TimestampArray.cs Adds capacity and allocator construction.
src/​Apache.Arrow/​Arrays/​Time64Array.cs Adds capacity and allocator construction.
src/​Apache.Arrow/​Arrays/​Time32Array.cs Adds capacity and allocator construction.
src/​Apache.Arrow/​Arrays/​StringViewArray.cs Adds allocator-aware builder construction.
src/​Apache.Arrow/​Arrays/​StringArray.cs Adds allocator-aware builder construction.
src/​Apache.Arrow/​Arrays/​PrimitiveArrayBuilder.cs Adds lazy validity, span filling, disposal, and bounds checks.
src/​Apache.Arrow/​Arrays/​MapArray.cs Adds allocator-aware child builders and disposal.
src/​Apache.Arrow/​Arrays/​ListViewArray.cs Adds allocator propagation and builder disposal.
src/​Apache.Arrow/​Arrays/​ListArray.cs Adds allocator propagation and builder disposal.
src/​Apache.Arrow/​Arrays/​LargeStringArray.cs Adds allocator-aware builder construction.
src/​Apache.Arrow/​Arrays/​LargeListViewArray.cs Adds allocator propagation and builder disposal.
src/​Apache.Arrow/​Arrays/​LargeListArray.cs Adds allocator propagation and builder disposal.
src/​Apache.Arrow/​Arrays/​LargeBinaryArray.cs Adds allocator-backed builders and disposal.
src/​Apache.Arrow/​Arrays/​IntervalArray.cs Adds capacity and allocator constructors for intervals.
src/​Apache.Arrow/​Arrays/​Int8Array.cs Adds capacity and allocator construction.
src/​Apache.Arrow/​Arrays/​Int64Array.cs Adds capacity and allocator construction.
src/​Apache.Arrow/​Arrays/​Int32Array.cs Adds capacity and allocator construction.
src/​Apache.Arrow/​Arrays/​Int16Array.cs Adds capacity and allocator construction.
src/​Apache.Arrow/​Arrays/​HalfFloatArray.cs Adds capacity and allocator construction.
src/​Apache.Arrow/​Arrays/​GuidArray.cs Adds allocator-aware builder construction.
src/​Apache.Arrow/​Arrays/​FloatArray.cs Adds capacity and allocator construction.
src/​Apache.Arrow/​Arrays/​FixedSizeListArray.cs Adds allocator-aware child construction and disposal.
src/​Apache.Arrow/​Arrays/​FixedSizeBinaryArray.cs Adds allocator support and disposal to the builder base.
src/​Apache.Arrow/​Arrays/​DurationArray.cs Adds capacity and allocator construction.
src/​Apache.Arrow/​Arrays/​DoubleArray.cs Adds capacity and allocator construction.
src/​Apache.Arrow/​Arrays/​DelegatingArrayBuilder.cs Forwards disposal to the inner builder.
src/​Apache.Arrow/​Arrays/​Decimal64Array.cs Adds allocator-aware builder construction.
src/​Apache.Arrow/​Arrays/​Decimal32Array.cs Adds allocator-aware builder construction.
src/​Apache.Arrow/​Arrays/​Decimal256Array.cs Adds allocator-aware builder construction.
src/​Apache.Arrow/​Arrays/​Decimal128Array.cs Adds allocator-aware builder construction.
src/​Apache.Arrow/​Arrays/​Date64Array.cs Adds capacity and allocator construction.
src/​Apache.Arrow/​Arrays/​Date32Array.cs Adds capacity and allocator construction.
src/​Apache.Arrow/​Arrays/​BooleanArray.cs Adds allocator-backed bitmaps and builder disposal.
src/​Apache.Arrow/​Arrays/​Bool8Array.cs Adds capacity and allocator construction.
src/​Apache.Arrow/​Arrays/​BinaryViewArray.cs Adds allocator-backed builders and disposal.
src/​Apache.Arrow/​Arrays/​BinaryArray.cs Adds allocator-backed builders and disposal.
src/​Apache.Arrow/​Arrays/​ArrowArrayBuilderFactory.cs Propagates allocators and preserves timestamp types.
src/​Apache.Arrow/​Arrays/​ArrayDataConcatenator.cs Uses allocator-backed run-end builders.
src/​Apache.Arrow.Operations/​Shredding/​ShreddedVariantArrayBuilder.cs Builds directly in the target allocator and disposes builders.
src/​Apache.Arrow.Flight/​Server/​Internal/​FlightServerImplementation.cs Passes server contexts to stream readers and writers.
src/​Apache.Arrow.Flight/​Server/​FlightServerRecordBatchStreamWriter.cs Adds context-aware writer construction.
src/​Apache.Arrow.Flight/​Server/​FlightServerRecordBatchStreamReader.cs Documents context-aware reader construction.
src/​Apache.Arrow.Flight/​Server/​FlightServer.cs Exposes a server-level Arrow context.
src/​Apache.Arrow.Flight/​Internal/​RecordBatchReaderImplementation.cs Uses the reader’s registry for schema decoding.
src/​Apache.Arrow.Flight/​Internal/​FlightMessageSerializer.cs Accepts registries when decoding schemas.
src/​Apache.Arrow.Flight/​Internal/​FlightDataStream.cs Passes context settings to the IPC writer.
src/​Apache.Arrow.Flight/​FlightRecordBatchStreamWriter.cs Carries contexts into Flight data streams.
src/​Apache.Arrow.Flight/​FlightInfo.cs Decodes schemas with a supplied registry.
src/​Apache.Arrow.Flight/​Client/​FlightClientRecordBatchStreamWriter.cs Forwards the client context to the writer base.
src/​Apache.Arrow.Flight/​Client/​FlightClient.cs Propagates contexts and supports dependency-injected construction.
src/​Apache.Arrow.Flight/​Apache.Arrow.Flight.csproj Adds dependency-injection abstractions.
src/​Apache.Arrow.Flight.Sql/​FlightSqlServer.cs Adds context-aware server construction.
Directory.Packages.props Pins the dependency-injection abstractions version.

Review effort: Balanced

Comment thread src/Apache.Arrow/ArrowBuffer.Builder.cs Resolved
- A growing BitmapBuilder.Resize explicitly unsets the bits it adds, instead of relying on the
  allocator to zero memory (GarbageMemoryAllocator does not). Clear() after a build and the
  padding of copied buffers are now zeroed explicitly for the same reason.
- Appending a span that is part of the builder's own memory copies it first when the append
  has to grow, since growing releases that memory. Applies to both buffer builders.
- A builder built again after a shrinking Resize no longer reuses the earlier buffer, which
  would have exposed the removed values in its padding.
- BuilderMemory keeps itself alive while copying its working memory, so a collection during
  allocation cannot free the source.
- Run-end concatenation releases its builders if it throws partway through.
- TimestampArray.Builder(type, capacity, allocator) checks for a null type before allocating.
- BitmapBuilder.Capacity no longer overflows for very large bitmaps.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Approval is blocked by four active critical issues involving a cross-assembly constructor access error, incorrect buffer lifetime propagation, a potential use-after-free during copying, and disposal of pinned native memory.

4 open issues
7 resolved since last review

Review effort: Lite

- Concatenated buffers now carry their allocator's lifetime, as built buffers do. They were
  created through a helper that always used the Finalized lifetime, so buffers from caller-owned
  or pooled allocators could still be released by collection.
- CallerOwnedNativeAllocator tracks pins. Disposing a pinned buffer marks it disposed but
  defers freeing the memory until the last pin is released, so a pinned pointer never dangles.
  It also counts live allocations, for tests.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Unresolved critical memory-lifetime and disposal defects can cause use-after-free or allocator leaks, and the added constructor overload breaks source compatibility.

No open issues 🎉

4 previously missed issues

In code that hasn't changed since last review

High severity Bitmap overlap check can free shared source memory

src/​Apache.Arrow/​ArrowBuffer.BitmapBuilder.cs:134

This has the same shared-builder lifetime problem as the fixed-width append path: source.Overlaps(Span) evaluates the writable Span property, which detaches and releases the retained buffer when the bitmap was already built. If the previously built buffer is not otherwise retained, a span captured before Build can point at finalized native memory that is freed before the later append copies it. Perform the overlap test without detaching (or copy conservatively while shared) and retain the old owner through the copy.

High severity Overlap check can free shared source memory before copying

src/​Apache.Arrow/​ArrowBuffer.Builder.cs:160

The overlap check reads Span, but Span delegates to BuilderMemory.Memory, whose getter detaches and releases the builder's retained shared buffer after a prior Build. If the caller takes a span, builds (and discards) the returned buffer, then appends that span after growth, this check can release the only owner of finalized native memory before source.CopyTo runs; the raw source span can then read freed memory. Check overlap without detaching (or conservatively copy the source whenever the builder is shared) and keep the old owner alive until the copy completes.

High severity Decompression failures leak the output buffer

src/​Apache.Arrow/​Ipc/​DecompressingBufferCreator.cs:61

When decompression throws or returns a length different from uncompressedLength, outputData is abandoned without being disposed. That is a leak for a context using CallerOwnedNativeAllocator (and can retain a pooled array for other allocators), so malformed or corrupt compressed input permanently consumes the allocator's memory. Wrap the allocation/use in a try/catch that disposes outputData before rethrowing, while leaving it owned by the returned ArrowBuffer on success.

Medium severity Nullable context overload breaks derived constructors using base(null)

src/​Apache.Arrow.Flight.Sql/​FlightSqlServer.cs:227

Adding the FlightSqlServer(ArrowContext?, ILoggerFactory?) overload makes existing derived constructors that call base(null) fail to compile because null now matches both the old ILoggerFactory? overload and the new ArrowContext? overload. Preserve source compatibility by avoiding two nullable reference-type overloads that can be selected by a null literal (for example, use a differently named/factory-based context entry point or require an explicit compatibility cast).

4 resolved since last review

Review effort: Lite

… leaks

- Appending a span that points into memory the builder is about to release now copies it
  first in both buffer builders. Besides growth, this covers the first write after a build:
  that releases the builder's reference to the built buffer, which may be the last one. The
  overlap check reads the current memory without detaching, since detaching would release it.
- A failed decompression disposes its output buffer instead of leaking it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@CurtHagenlocher

Copy link
Copy Markdown
Contributor Author

Responses to the third Copilot review, which listed four issues in its summary without inline threads:

  • Overlap checks reading Span (both buffer builders): agreed, and fixed in 4a02294. It was broader than described: the first write after a build releases the builder's reference to the built buffer, so a span captured before Build could be read after release even without growth, because CopyTo evaluates Span first. The check now reads the builder's current memory without detaching, and copies the source first whenever writing could release that memory (growth, or memory handed to a built buffer). Covered by two new tests that use a poisoning pooled allocator.
  • Decompression failures leak the output buffer: agreed, fixed in 4a02294. The output is now disposed if decompression throws or returns the wrong length. A new test reads corrupt compressed data through a caller-owned allocator and checks that nothing is left outstanding.
  • FlightSqlServer(ArrowContext?, ILoggerFactory?) makes base(null) ambiguous: I don't think this applies. The new overload has no optional parameters, so it isn't a candidate for a one-argument call; base(null) binds only to the existing FlightSqlServer(ILoggerFactory? factory = null). I confirmed that a derived server calling base(null) (and base()) compiles without error.

On the earlier macOS Verify failure: it wasn't caused by this PR, and a rerun passed. In the failing run, every project was restored under both /var/folders/… and /private/var/folders/… (on macOS, /var is a symlink to /private/var), so MSBuild treated each project as two and built them concurrently into the same outputs. Every passing run, including on main, uses /private/var only. The failure coincided with the newer runner image 20261103.629, while the passing runs and the rerun were on 20261028.624. So main's macOS release verification may start failing the same way as that image rolls out. Resolving the temporary directory in dev/release/verify_rc.sh to a canonical path (for example with pwd -P after mktemp) should prevent it; I'll leave that to a separate change.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because the change contains multiple critical exception-safety leaks with caller-owned allocators, plus a moderate overlapping-memory corruption risk in bitmap appends.

No open issues 🎉

Review effort: Lite

CurtHagenlocher and others added 2 commits November 15, 2026 16:39
With a caller-owned allocator these paths leaked permanently; with the default allocator a
finalizer eventually reclaimed the memory.

- The writer allocates all of a batch's temporary buffers before writing and released each
  only after writing it, so a write that failed partway through (for example on a broken
  connection) leaked the rest. The batch builder now releases any buffers not yet released when
  it is disposed, and preparing a batch releases what it allocated if it throws. Buffers are
  still released as soon as they are written.
- The readers released a message body only by handing it to a record batch. A truncated stream,
  a malformed message or a failed decompression leaked it, as did a non-empty body on a schema or
  unrecognized message. CreateArrowObjectFromMessage now owns the body and releases it unless the
  result refers to it, and the stream readers release it if reading it fails.
- BitmapBuilder copies a source that overlaps its own memory whenever it appends it: appending
  bit by bit could otherwise overwrite source bits before reading them.

Dictionary batch bodies are still never released; that needs a separate design.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A dictionary's arrays are views into its message body, but nothing owned the body: it was
reclaimed only by finalization, so with a caller-owned allocator every dictionary message
leaked its body permanently. It cannot simply be released when the dictionary is replaced or
the reader is disposed, because every record batch that uses the dictionary shares it and can
outlive both.

- Dictionary buffers now count references to the body (a MessageBody hands out sliced
  SharedMemoryHandles), including buffers stored uncompressed in a compressed stream.
- Each record batch retains its own reference to the dictionary it uses, so disposing a batch
  releases only that batch's references.
- The reader owns the dictionaries in its memo: it releases one when it is replaced, a delta and
  the dictionary it extends once they are concatenated, and all of them when it is disposed.
- Arrays already built are released if building a batch or dictionary fails.

With the default allocator the body is now released as soon as the reader and every batch using
it are disposed, rather than at finalization.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@CurtHagenlocher
CurtHagenlocher requested a lite review from Copilot November 16, 2026 01:07
@CurtHagenlocher

Copy link
Copy Markdown
Contributor Author

The latest review reported no open issues, but its summary mentioned "exception-safety leaks with caller-owned allocators" and "overlapping-memory corruption risk in bitmap appends" without locations. Following those up found real problems, all of which predate this PR: with the default allocator, finalizers reclaimed the memory, so the leaks only became permanent with caller-owned allocators. Two new commits fix them, each with tests that failed before the fix.

f8346b5: release IPC buffers when reading or writing fails

  • Writer: all of a batch's temporary buffers (rewritten offsets, compression output) are allocated before writing, and each was released only after it was written. A stream that threw partway through (for example on a broken connection) leaked the rest, as did an exception while preparing the batch. The batch builder's Dispose now releases anything not yet released, and buffers are still released as soon as they're written. The test fails the stream at every write the writer makes, sync and async.
  • Readers: a message body was released only by handing it to a record batch, so a truncated stream, a failed decompression or a malformed message leaked it, as did a non-empty body on a schema or unrecognized message. CreateArrowObjectFromMessage now owns the body and releases it unless the result refers to it, and the stream readers release it if reading it fails.
  • Bitmap appends: appending bit by bit could overwrite source bits before reading them when the source was the builder's own memory and extended past its length. BitmapBuilder now copies a source that overlaps its memory whenever it appends it.

35c2e41: reference-count dictionary batch bodies

This one also leaked when nothing went wrong. A dictionary's arrays are views into its message body, but nothing owned the body, so every dictionary message read through a caller-owned allocator leaked it. Releasing it when the dictionary is replaced or the reader is disposed would be a use-after-free, because every batch using the dictionary shares it and can outlive both. So it now uses the same reference counting as shared buffers:

  • Dictionary buffers hold counted references to slices of the body, including buffers stored uncompressed in a compressed stream.
  • Each record batch retains its own reference to its dictionary (ArrayData.Retain(), no data copied), so disposing a batch releases only that batch's references.
  • The reader owns the dictionaries it keeps: it releases one when it's replaced, a delta and the dictionary it extends once they're merged, and all of them when the reader is disposed.

Tests read consecutive dictionaries through a counting caller-owned allocator and check nothing is left outstanding, and check that both batches' dictionaries stay intact after the reader is disposed and the dictionary replaced, using an allocator that overwrites freed memory. The delta path isn't covered by a test, because the .NET writer can't produce delta dictionaries.

One behaviour change with the default allocator: a dictionary body is now freed once the reader and every batch using it are disposed, rather than at finalization. Code that keeps using a dictionary after disposing both its batch and the reader would now read freed memory, which is the rule that already applies to a batch's own columns.

All tests pass on net462, net472 and net8.0, including the pyarrow interop tests.

🤖 Generated with Claude Code

CurtHagenlocher added a commit that referenced this pull request Nov 16, 2026
## What's wrong

On macOS, `verify_rc.sh` can fail with dozens of spurious compile errors
in the test step: "type or namespace name 'FlightTicket' could not be
found", `K4os` not found in the compression tests, and so on, even
though the same projects built successfully moments earlier in the build
step.

The cause is the temporary directory. On macOS it's under `/var`, which
is a symlink to `/private/var`, and the script works under the symlinked
path. Tools can then see the same files under both `/var/folders/…` and
`/private/var/folders/…`. When that happens, MSBuild treats each project
as two different projects and builds them concurrently into the same
output folder, so builds read references that another build is still
writing.

## Evidence

From the `Verify (8.0.x, macos-latest)` job of the RC workflow on #504:

| Run | Runner image | Restores under `/var` | Restores under
`/private/var` | Result |
|---|---|---:|---:|---|
| [Failing
run](https://github.com/apache/arrow-dotnet/actions/runs/43776728836/job/133004801520)
| `20261103.629` | 25 | 20 | 127 compile errors |
| [Rerun of the same
commit](https://github.com/apache/arrow-dotnet/actions/runs/43776728836/attempts/2)
| `20261028.624` | 0 | 34 | Passed |
| Earlier runs, and the latest run on `main` | `20261028.624` | 0 | 34 |
Passed |

Every run reports its sandbox under `/var/folders/…`, but only the run
on the newer image ended up with both paths in use. So this is likely to
start failing on `main` too once that image becomes the default for
`macos-latest`.

## The fix

`setup_tmpdir` now resolves `VERIFY_TMPDIR` to its physical path with
`pwd -P` before anything works in it, so the shell's `$PWD` and the
operating system's view of the current directory agree. This applies
both when the script creates the directory with `mktemp` and when the
caller supplies `VERIFY_TMPDIR`. The cleanup trap removes the same
directory either way. On Linux and Windows, where the path normally has
no symlinks, nothing changes.

## Testing

- With a real symlinked directory, both branches of `setup_tmpdir`
(`mktemp` and a caller-supplied `VERIFY_TMPDIR`) resolve to the physical
path, `$PWD` is physical after the `cd`, and the cleanup trap still
removes the directory.
- `shellcheck` 0.10.0 (the version pinned in `.pre-commit-config.yaml`)
and `shfmt` report no issues.
- I couldn't run the script on macOS locally; this PR's own RC workflow
run exercises it on `macos-latest`.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Approval is blocked by active allocator-related issues involving unnecessary bitmap allocation and potential buffer leaks during primitive array construction and IPC field loading.

2 open issues

Review effort: Lite

If reading a field failed after some of its buffers or children were created (for example, a
decompression failure on a later buffer), the reader dropped the partially built field, leaking
its decompressed buffers and references to a shared message body. LoadField and GetChildren now
release what they built before rethrowing.

Also build the shredded variant's top-level validity bitmap with the supplied allocator, so the
result is handed over rather than copied, and dispose the builder.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because allocator-lifetime and exception-path ownership issues across multiple builders can leak buffers or retain backing allocations after disposal.

No open issues 🎉

2 resolved since last review

Review effort: Balanced

After handing its working memory to a built buffer, a builder kept its own reference to that
buffer so it could keep appending. With a caller-owned allocator, a builder that was never
disposed therefore leaked what it built permanently, even once the built array was disposed,
and the usual chained style (new Builder(allocator).Append(x).Build()) never disposes it.

A builder now holds a reference only for allocators whose lifetime is Finalized, where
collection reclaims it, so contents still survive a built array that is dropped without being
disposed. With other lifetimes it takes a reference only while reading the built buffer, which
it can do while the buffer is alive. Once the buffer has been disposed the builder's contents
are gone: using them throws ObjectDisposedException, and Clear makes the builder usable again.

Also release what the list, list-view, fixed-size list, map and binary-view builders have built
when a later part of Build fails, for example a child fixed-size list of the wrong length.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

Several allocator-backed array builders still leak caller-owned buffers or child arrays when later allocation or construction steps fail, leaving active moderate findings that require cleanup before approval.

6 open issues

Review effort: Lite

Comment thread src/Apache.Arrow/Arrays/BinaryArray.cs Resolved
Comment thread src/Apache.Arrow/Arrays/BooleanArray.cs Resolved
Comment thread src/Apache.Arrow/Arrays/VariantArray.cs Resolved
Builders that build several buffers or child arrays in sequence dropped the earlier ones if a
later step threw. With a pluggable allocator that is not only running out of memory: an
allocator that enforces a budget refuses allocations by throwing, and with a caller-owned
allocator the dropped buffers leaked permanently.

The primitive, Boolean, binary, large binary, fixed-size binary, run-end encoded, timestamp
with offset and variant builders now release what they built before rethrowing. A new test
builds each into an allocator that refuses its n-th allocation, for every n a successful build
makes, and checks that nothing is left outstanding in either allocator.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔵 Needs a closer look

Human assessment is needed because a critical lifetime issue in ArrowBuffer.Clone may allow copying from freed native memory, making the change too complex to approve automatically.

No open issues 🎉

6 resolved since last review

Review effort: Lite

Both read the buffer through a span, which doesn't root the handle that
owns the memory. A buffer that was otherwise unreachable could have its
handle finalized, freeing native memory, partway through the copy or
comparison. This predates the allocator work; only buffers whose
lifetime is Finalized are affected, since other lifetimes have no
finalizer.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@CurtHagenlocher

Copy link
Copy Markdown
Contributor Author

The review on 34dc888 reported no open issues, but its summary mentioned a lifetime issue in ArrowBuffer.Clone. It's real, though narrow and older than this PR: Clone (and Equals) read the buffer through a span, which doesn't root the handle that owns the memory. So a buffer that is otherwise unreachable, with the default Finalized lifetime, could be finalized and its native memory freed partway through the copy. 399ab94 keeps the handle alive until the read finishes. Other lifetimes weren't affected, because they have no finalizer.

🤖 Generated with Claude Code

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants