Skip to content

feat(🖼️): NativeBufferView, a display-only view for native buffers - #635

Merged
wcandillon merged 27 commits into
mainfrom
feat/native-buffer-view
Nov 21, 2026
Merged

wcandillon merged 27 commits into
mainfrom
feat/native-buffer-view

Conversation

@wcandillon

Copy link
Copy Markdown
Owner

Description

Agent-written content

Step 1 of the NativeBufferView plan: a display-only native view that shows a native buffer (AHardwareBuffer on Android, IOSurface on Apple) with no swap chain, no GPUCanvasContext and no Dawn device on the display side. It's meant for static or occasionally updated content; animated content keeps using <Canvas>. The experimental HardwareBufferView backing is untouched (its removal is step 2), and nothing about the existing <Canvas> backings changes.

The contract

  • Immutable while displayed: a buffer handed to the view is never written again while it's on screen. To update, render into a new buffer and present that one.
  • Consumer only: the view never allocates buffers. It reports its size in physical pixels, and the producer (WebGPU, Skia, a camera, a decoder) decides what to render.
  • Present on change: there is no frame loop. The OS shows the new content on the next vsync after its fence signals.

API

const ref = useRef<NativeBufferViewRef>(null);
const { size, onPixelSizeChange } = useNativeBufferSize(); // debounced resizes
<NativeBufferView ref={ref} resizeMode="stretch" onPixelSizeChange={onPixelSizeChange} />

const frame = createNativeBuffer(size.width, size.height); // { format?, colorSpace? }
const target = device.createNativeBufferTexture(frame, { usage: GPUTextureUsage.RENDER_ATTACHMENT });
target.beginAccess();
// render + submit
ref.current.present(frame, target.endAccess()); // or RNWebGPU.presentNativeBuffer(viewId, frame, fence)
target.destroy(); frame.release();
  • resizeMode: stretch (default), contain, cover or topLeft (no scaling, crop instead of blur). A buffer that lags the view's size is scaled, never blank.
  • createNativeBuffer formats: rgba8unorm (Android default), bgra8unorm (Apple default), and rgba16float on both. colorSpace is srgb or display-p3; Apple also tags kIOSurfaceColorSpace.

Design

  • Hand-off: Fabric props can't carry host objects, so JS hands buffers through a C++ NativeBufferViewRegistry keyed by a view id that JS allocates. Presents are numbered, so a buffer whose fence signals after a later present has landed is dropped. A registry entry is erased once both the JS component and the native view have let go, in either order, which keeps StrictMode safe.
  • Acquire fence: waited off the main thread. On Android, one shared FIFO thread polls the sync fd. On Apple, an MTLSharedEventListener is used, so no thread blocks. Without a fence, the buffer is presented immediately.
  • Android, API 29+: Bitmap.wrapHardwareBuffer(hb, colorSpace), drawn in onDraw (zero-copy).
  • Android, API 26 to 28: the pixels are read back with AHardwareBuffer_lock (stride-aware) into a regular bitmap. CPU_READ_RARELY is added at allocation only below 29.
  • Apple: layer.contents = ioSurface inside a CATransaction on the main thread, with contentsScale set to the screen scale.
  • Size: a native onPixelSizeChange event reports the layout's pixel size on both platforms.

Testing

  • iOS simulator (iPhone, iOS 26.5): the example screen renders 1206x1200 px at a sharp 1:1. An animated resize stretches the old buffer without blanking, then re-renders at 603x480 px once settled. stretch, contain, cover and topLeft all checked against a mismatched buffer.
  • Android: native and Java build for arm64 and the APK assembles, but it has not been run on a device yet.
  • TypeScript, eslint and clang-format checks pass on the new code.

Not verified yet

  • Android at runtime (API 29+), and the API 26 to 28 readback path.
  • rgba16float on Android 8 and 9: Bitmap.createBitmap may refuse EXTENDED_SRGB there; the view falls back with a warning.
  • macOS (it compiles the same sources; topLeft gravity is mapped separately there).

Displays an AHardwareBuffer (Android) or IOSurface (Apple) presented from JS, with no swap chain, canvas context or GPU device on the display side. Adds createNativeBuffer(), RNWebGPU.presentNativeBuffer(viewId, frame, fence?), the NativeBufferView component and useNativeBufferSize(), and an example screen.
NativeVideoFrame, NativeVideoFrameColorSpace and NativeVideoPixelFormat become NativeBuffer, NativeBufferColorSpace and NativeBufferPixelFormat; the old names stay as deprecated type aliases. The runtime brand is now "NativeBuffer". A buffer is no longer only a video frame: createNativeBuffer() allocates one to render into and NativeBufferView displays it, while shared texture memory stays the name of a device's import of it.
@wcandillon
wcandillon marked this pull request as ready for review November 20, 2026 16:36
@wcandillon

Copy link
Copy Markdown
Owner Author

@YevheniiKotyrlo you are a co-author on this :)

200 Skia Icons: Skia Canvas vs NativeBufferView

For 200 Skia icons on one screen, NativeBufferView beats 200 Skia <Canvas> on memory and on time-to-display. Frame rate while the screen moves is the same for both.

Setup

A 10×20 grid of 200 different icons (gradient rounded rect, white glyph, border), all on screen. Every variant draws the same SkPicture per icon:

  • Skia Canvas x200: one <Canvas> per icon, so 200 TextureViews on Android and 200 CAMetalLayers on iOS.
  • NativeBufferView x200: Skia draws each icon once into a native buffer on Skia's own device, then presents it. This works on both platforms. The draw path:
    1. importDevice(Skia.getNativeDevice())
    2. createNativeBufferTexture
    3. Skia.Surface.MakeFromGPUTexture
  • One Skia Canvas: a single canvas drawing all 200 icons, as a reference.

Memory

Measured in a fresh process per mode, three rounds each. Values are the extra memory over an empty screen.

Mode Samsung (PSS) iPhone 14 Pro (footprint)
Skia Canvas x200 +218 MB +46 MiB
NativeBufferView x200 +40 MB +17 MiB
One Skia Canvas +88 MB +21 MiB
  • Rounds varied by about 1% at most.
  • On Android, 200 TextureViews cost 5× more memory than 200 NativeBufferViews.
  • NativeBufferView x200 even uses less memory than one big canvas.

Time to display

Time until all 200 icons are on screen, from a Samsung screen recording. Counted from the frame where React's commit first appears.

Mode Samsung
Skia Canvas x200 253 to 286 ms (canvases appear in batches)
NativeBufferView x200 101 to 102 ms
One Skia Canvas 18 ms

…oid acquire fence

A present that landed after the component unmounted (or to an id nobody holds) recreated the registry entry and leaked its buffer; presents now go only to entries the JS component holds. The component retains its entry in a layout effect (retainNativeBufferView), so React StrictMode's discarded useState ids create nothing and its simulated unmount releases and retains again.

A JS reload never ran the components' cleanup, so every view's last buffer stayed allocated. RNWebGPUManager teardown now calls releaseAllFromJS, like SurfaceRegistry::clear.

On Android, presentNativeBuffer ignored the frame's own acquire fence; it now waits for it (a dup of the fd) alongside the Dawn fence.

Documents that pooled frames (VideoPlayer, camera, decoder) must be copied into a createNativeBuffer() buffer before presenting: the pool reuses the buffer once the frame is released, even while the view shows it.

Verified on a Samsung SM-S937B and an iOS 26.5 simulator under StrictMode with temporary diagnostics: late presents dropped, entries back to zero after each of three reloads, and the producer-fence path displays correctly.
…ers in the example

NativeBufferView shows values in [0, 1]: rgba16float gives more precision, not extended range. The registry header claimed otherwise; the header, the createNativeBuffer comment and CreateNativeBufferOptions.format now say so.

The NativeBufferView example started an async render on every size change with no cancellation, so an older render could present an outdated size after a newer one, or present after unmount. Each render now checks a flag cleared by the effect cleanup before presenting.
…tforms

A present whose GPU work never finished (a lost device) kept its buffer forever: the MTLSharedEvent listener had no timeout, unlike the Android waiter. A 10 s dispatch_after now races the notification on the same serial queue; whichever runs first releases the fence and the buffer, and a timeout is logged.

Metal cannot cancel a notification, so after a timeout the listener keeps its block, which now only holds a small struct. ANGLE's Metal sync (SyncMtl.mm) makes the same trade-off.

Verified on an iOS 26.5 simulator: a normal present is shown and the later timer is a no-op; a fence forced never to signal is dropped and its buffer freed after 10 s.
A getting-started guide that consolidates the native paths: what a NativeBuffer is, where buffers come from (pooled or yours), which import path to pick to read one, rendering into one (and why the wrapper rather than importSharedTextureMemory), drawing into one with Skia, displaying one with NativeBufferView, fences, lifetimes, device features and platform notes.

A Native Buffer API page for NativeBuffer, createNativeBuffer, createVideoFrameFromNativeBuffer, NativeBufferView, its ref, resize modes and useNativeBufferSize. The NativeBuffer section moves there from Video Player; Native APIs, Native Extensions, React Native Skia, Vision Camera and the API overview link to the guide and the new page.
…00 icons example

The guide's sources section is prose instead of a table: your own rendering with createNativeBuffer() (linking the 200 icons example), VisionCamera, and React Native Skia Video for video (its React Native Skia v3 PR goes through React Native WebGPU). createVideoPlayer() is no longer promoted there; the reading example takes a frame from any producer.

The reading table gets a Returns column, and importSharedTextureMemory gets its own row.

Adds the 200 icons example screen: Skia draws 200 icons into native buffers shown by NativeBufferViews, compared with one Skia Canvas per icon and with a single canvas, with memory and timing reports.
…al textures, without wrappers

The wrappers added in #540 (GPUNativeBufferTexture and GPUNativeBufferExternalTexture, after GPUHardwareBufferWrapper in Dawn's Android layer) bundled shared texture memory, a texture or an external texture, and fence handling under new names. Each gap they covered is now closed where it lives, so every type a reader meets is NativeBuffer, Dawn's, or the spec's:

- importSharedTextureMemory({ source: frame }) takes a NativeBuffer (handle still works). The memory keeps the buffer alive, and the textures it creates remember the buffer's color space.
- importExternalTexture({ source: texture }) takes a texture of a GPUSharedTextureMemory: the external texture is built once per buffer, color conversion included, and is sampled between the memory's beginAccess() / endAccess(). This was the one thing only the wrapper could do (a YUV buffer imported once).
- NativeBufferView.present(frame, done) takes what GPUSharedTextureMemory.endAccess() returns, and merges its fences into one (sync_merge on Android, the latest value of the shared event on Apple platforms).

createNativeBufferTexture / createNativeBufferExternalTexture and their types are removed. The NativeBufferWrapper e2e spec becomes SharedTextureMemoryExternalTexture (the color-texture and destroy cases are covered by the SharedTextureMemory specs). The examples and the docs (Native Buffers guide, Native Extensions, Native Buffer, Native APIs, React Native Skia) follow the new paths; the guide's read-path table returns standard types only.

Verified on a Samsung SM-S937B and an iPhone 14 Pro: SharedTextureMemory*, ExternalTexture*, ImportExternalTexture, CopyVideoFrameToTexture and SharedFence specs 18/18 on each (YUV AHardwareBuffer and NV12 IOSurface imported once), and the NativeBufferView and 200 icons screens render on both.
The vocabulary table becomes a flow: NativeBuffer, imported into a device as GPUSharedTextureMemory, a GPUTexture over its pixels, and the spec's GPUExternalTexture to sample it as RGB, with importExternalTexture({ source: buffer }) as the one-frame shortcut. Fences and NativeBufferView sit next to it. The read table drops its Returns column.
Replaces the text diagram with a Mermaid flowchart, which also shows the two shortcuts: importExternalTexture() for one frame, and present() to a NativeBufferView.
The flow diagram now shows two branches from NativeBuffer: shared texture memory and its GPUTexture (read and write), and importExternalTexture() (read only, YUV as RGB). The link from a texture to an external texture moves to the prose, as the import-once pattern.
…guide

The React Native Skia page now links to Rendering into a buffer and the 200 icons example instead.
Review of the second half of the guide:
- endAccess() returns once the work is submitted, not queued; the view waits for fences (plural).
- One access per texture, not per buffer.
- Lifetimes: the shared memory keeps the import (Dawn holds its own reference on the buffer); an external texture over a memory texture opens no access and is destroyed before the texture and the memory.
- Platform notes: one YUV bullet for both platforms.
- The importExternalTexture sample is introduced as the first row of the table, and the copy sentence follows it.
The section is about iOS and tvOS refusing GPU work in the background. Also fixes its dawnToggles link (/docs/api/ is not a valid path).
… aliases

BREAKING: createVideoFrameFromNativeBuffer (the export and the RNWebGPU global) is renamed createNativeBufferFromPointer, matching createNativeBuffer and the pointer it takes. No alias is kept.

BREAKING: the deprecated NativeVideoFrame, NativeVideoFrameColorSpace and NativeVideoPixelFormat aliases are removed; use NativeBuffer, NativeBufferColorSpace and NativeBufferPixelFormat.
…d the reader

A pass over the whole documentation, with what we settled on for the Native Buffers guide:

- Model first: the Canvas guide renders its first triangle (moved from the API page, which becomes a reference); "Native APIs" becomes the "Platform Guide" with an overview of each difference from the web; the Integrations introduction gains React Native Skia and a "Which one to pick" section.
- One explanation per topic: threading points to Worklets, native textures to the Native Buffers guide, and the Canvas topics live in the guide.
- The video player is a test tool: GPUDeviceProvider gets a neutral example, the React Native Skia video section and the Vision Camera page point to libraries and to importing a buffer once, and the Video Player page moves last with a "for tests and examples" note.
- Reader-facing: the pipeline cache, iOS background work and Android rendering options keep their contracts and lose internals; the Dawn mirror moves to Troubleshooting and Swift Package Manager becomes optional.
- Wording: no spaced hyphens as dashes, colon-style See also lists, product names in prose, clearer subtitles, the Android rendering options heading level, and createNativeBufferFromPointer.
- The Mermaid diagrams are monochrome: a white figure with an outline and light gray boxes, dark equivalents in dark mode.
…referenced

Each texture created by GPUSharedTextureMemory.createTexture() held a
SharedTextureOrigin with its own strong reference on Dawn's shared texture
memory, so destroy() on the memory no longer released the import: it lived
until the GPUTexture JS object was garbage collected, and the native buffer
with it. With a buffer imported per frame, a pool could not recycle them.

The origin is now one object per memory, shared with its textures, and
destroy() clears its reference. importExternalTexture({ source: texture })
on a texture of a destroyed memory is reported like the other uses of a
destroyed memory: a validation error, then the throw.

Measured on a Galaxy S25 Edge with 64 imports of a 512x512 buffer, texture
and memory destroyed, frame released, texture objects kept referenced:
before, 256 more dmabuf fds and about 100 MB more graphics memory per
round; after, none.
On Android 10 and above the view draws a hardware bitmap that wraps the
buffer, and Canvas.drawBitmap() throws for one on a software canvas
("Software rendering doesn't support hardware bitmaps"): capturing a screen
with view.draw(new Canvas(bitmap)), as view capture libraries do, crashed
the app. onDraw() now draws a software copy of the pixels on a canvas that
is not hardware accelerated.
@wcandillon
wcandillon merged commit c04329c into main Nov 21, 2026
9 checks passed
@github-actions

Copy link
Copy Markdown

🎉 This PR is included in version 0.23.0 🎉

The release is available on:

Your semantic-release bot 📦🚀

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant