Skip to content

feat: add lazy Arrow and database viewing v2 - #1876

Draft
Fred-Wu wants to merge 2 commits into
REditorSupport:mainfrom
Fred-Wu:lazy-dataview
Draft

Fred-Wu wants to merge 2 commits into
REditorSupport:mainfrom
Fred-Wu:lazy-dataview

Conversation

@Fred-Wu

@Fred-Wu Fred-Wu commented Oct 15, 2026

Copy link
Copy Markdown
Contributor

Closes #1785

This PR is a clean replacement for #1804. It includes the implementation and subsequent fixes, incorporates the latest master, and reorganises the changes into two commits: DBI support with shared viewer infrastructure, followed by Arrow support.

I am submitting both implementations for review and discussion; they do not necessarily need to be adopted together.

Summary

Add on-demand Data Viewer support for Arrow datasets, arrow_dplyr_query objects, and DBI-backed tbl_sql objects, currently targeting SQL Server.

Opening and scrolling do not require collecting the complete result into an R data frame. Paging, filtering, and sorting use different strategies for datasets, bare database tables, and transformed lazy queries.

Arrow datasets

  • Reuse a persistent reader for sequential scrolling and cached rows for backward access.
  • Use fragment-aware access for multi-file datasets and Parquet row groups for suitable distant or sparse requests.
  • Select requested rows and columns before converting Arrow batches into R data.
  • Preserve partition columns and support nested columns, with scanner fallback for nested Parquet schemas.
  • Compute filtering and sorting across the full dataset using the required key columns and source-row indices.
  • Preserve supported types, including signed integer64, dates, datetimes, and durations. Disable filtering and sorting for nested columns.

SQL Server tables

  • Support bare database tables represented by tbl_sql.
  • Execute viewer filtering and sorting in SQL Server.
  • Discover primary keys where available to provide stable paging order.
  • Reuse an open DBI result for sequential reads and use OFFSET when repositioning outside the cache.
  • Preserve temporal and signed 64-bit values during paging.

Transformed lazy queries

For both arrow_dplyr_query and transformed database queries:

  • Execute the original query progressively, retaining fetched batches and spilling older batches to temporary Arrow IPC files.
  • Execute database queries using their original top-level SQL, preserving their query semantics and ordering.
  • On the first viewer filter or sort, finish caching the active query execution, then apply the viewer operation to that result.
  • Apply subsequent control changes to the original cached result. Clearing the controls restores its original execution order.
  • Reuse cached results for backward access and column projection changes.
  • Reset failed or interrupted reads so retries cannot combine results from different executions.
  • Support unknown row counts until the result is exhausted.

Viewer filtering and sorting therefore act as additional operations over the original query result. They do not rewrite the user's query. Transformed database queries also require the optional arrow package for disk caching.

Shared viewer behaviour

  • Support column projection and unknown row counts.
  • Serialise page requests and discard queued requests when the viewer closes.
  • Allow long-running page requests without the normal request timeout.
  • Interrupt active reads on close where the owning terminal is available.
  • Clean up readers, database results, and temporary cache files.
  • Keep the session responsive to subsequent requests after an interrupted fetch.

Memory and current limitations

Block caches use a 20,000-row retention threshold, but this is not a strict total-memory limit. Individual Arrow batches can be larger, and filtering or sorting can require complete key columns and row indices in memory. Cached lazy-query results also consume temporary disk space.

Stable bare-table ordering depends on an available unique key. Source changes require reopening the viewer.

Large unsigned 64-bit integers and high-precision decimals remain a known precision limitation when converted to R numeric values, affecting display, filtering, and sorting.

Validation

Regression coverage includes sequential and random paging, fragments and row groups, projection, filtering and sorting, cache spilling, ordering, failure recovery, interruption, disposal, and supported column types. Arrow query tests account for platform-dependent batch arrival order.

Local Windows validation passed:

  • 2,570 R checks.
  • R and TypeScript linting.
  • TypeScript type checking.
  • Production build.

DBI tests use simulated connections and verify SQL generation and cursor behaviour; they do not replace live SQL Server/ODBC integration testing. macOS and Ubuntu CI results are not claimed by this local validation.

Fred Wu added 2 commits October 15, 2026 23:49
Add SQL Server table paging, primary-key discovery, and cached query execution with viewer filtering and sorting. Preserve temporal and integer64 values through Arrow IPC spill storage.

Add shared column projection, unknown row counts, cancellation, cleanup, recovery, and required CI dependencies. Include DBI and primary-key regression tests.
Add progressive paging, projected random access, and cached query filtering and sorting. Preserve nested and temporal values, recover failed reads, and handle unknown row counts.

Cover fragment and row-group access, spill caches, recovery, and platform-independent Arrow stream ordering in the regression tests.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

(feat) Additional Arrow support

1 participant