feat: add lazy Arrow and database viewing

[Fred-Wu]

Closes #1785

This might be a big change to support both Arrow's FileSystemDataset and SQL Server in the Data Viewer. I submit it here for review and discussion. It is not necessarily intended to adopt both of them. The implementations are more complicated than I initially thought, especially when performance is one part of the goals. Nevertheless, the following decisions are mine, with GPT used to generate and iterate on the implementation here.

Summary

Add lazy/on-demand Data Viewer support for large Arrow and DBI-backed data sources without loading the full dataset into memory.

Arrow and DBI now share the same general paging model:

• fetch data on demand while scrolling;
• use 1,000-row blocks for normal access and 5,000-row blocks for sorted results;
• retain up to 20,000 fetched rows in the cache;
• reuse cached rows for backward scrolling;
• invalidate the relevant cache when filtering, sorting, or column projection changes;
• fetch only columns required by the current page;
• apply filtering and sorting to the complete data source rather than only loaded rows.
• Preserve source ordering where applicable, while allowing Data Viewer sorting to take precedence over the same columns (e.g. dplyr::arrange()).
• clean up backend resources when the viewer closes.

The existing in-memory Data Viewer behaviour remains unchanged.

Changes

Arrow

• Reuse a persistent Arrow reader for sequential forward scrolling.
• Keep filtered/sorted source-row indices so arbitrary display positions can be mapped back to source rows.
• Maintain separate source-row and filtered/sorted display caches.
• Use fragment-aware access for multi-file datasets.
• Use Parquet row-group access for distant or sparse requests.
• Preserve partition columns when reading individual files or row groups.
• Preserve Arrow-specific column types during paging, including integer64, dates, datetimes, durations, and nested columns.

DBI

• Add lazy viewing for tbl_sql objects, currently targeting SQL Server.
• Push filtering and sorting to SQL Server.
• Reuse an open DBI result with dbFetch() for continuous forward scrolling.
• Use OFFSET to reposition the result for jumps or backward requests outside the cache.
• Avoid placing source ORDER BY clauses inside SQL Server subqueries where they are invalid.

Other changes

• sess/R/handlers.R

  • Route lazy Arrow and DBI objects through the Data Viewer.
  • Add shared paging, filtering, sorting, column-projection, and row-index handling used by the lazy viewers.
  • Dispose backend resources when a Data Viewer is closed.

• sess/R/server.R

  • Add Data Viewer init, page, and dispose RPC handling.
  • Allow long-running page requests to be interrupted.
  • Report active Data Viewer reads so the extension can interrupt the correct request.
  • Ensure sess continues processing subsequent requests after an interrupted Data Viewer fetch.

• src/session.ts

  • Add the dynamic Data Viewer request bridge.
  • Send selected column fields with page requests.
  • Serialise page requests and discard queued requests after the viewer closes.
  • Interrupt an active Data Viewer read when its panel is closed.
  • Remove the normal request timeout for page and dispose operations while retaining the initialisation timeout.

• sess/DESCRIPTION

  • Add the optional dependencies required by lazy Arrow and DBI Data Viewer support.

• Data Viewer tests

  • Extend the existing Data Viewer tests and add coverage for lazy paging, column projection, caching, filtering, sorting, interruption, disposal, Arrow fragment/Parquet access, and DBI behaviour.

Arrow and DBI use backend-specific strategies under the same Data Viewer behaviour: Arrow keeps source-row mappings for efficient random access, while DBI lets SQL Server perform filtering and sorting and streams the resulting rows through an open database result.

Validation

Added regression tests covering:

• Arrow sequential and random paging;
• multi-file fragment access and Parquet row groups;
• filtering, sorting, caching, and column projection;
• DBI SQL generation and source ordering;
• DBI cursor reuse and repositioning;
• viewer interruption and disposal;
• date, datetime, logical, formatted, and integer64 columns.

[eitsupi]

I wonder if we should reconsider how lazy objects are handled before adding backend-driven paging/filtering/sorting here.

Existing tools generally seem to treat lazy/remote data as a bounded preview rather than a full query client. For example, dbplyr prints only a small head without querying the total row count, DuckDB UI previews the first 100 rows, and DBeaver fetches a limited batch by default and makes “Fetch All” explicit.

Given that precedent, could we instead evaluate a lazy object once for the first N rows (possibly configurable, with N+1 used to detect truncation) and materialize that as a local snapshot (Ideally, it would be an .arrows / Arrow IPC stream file)?
The viewer could then operate entirely on that snapshot, rather than repeatedly querying the backend and transferring paged results as JSON whenever the user scrolls, filters, or sorts.

This would also remove a substantial amount of backend-specific paging/cache/query logic.
Full-data browsing could remain a separate explicit feature if we decide we need it later.

[Fred-Wu]

I wonder if we should reconsider how lazy objects are handled before adding backend-driven paging/filtering/sorting here.

Existing tools generally seem to treat lazy/remote data as a bounded preview rather than a full query client. For example, dbplyr prints only a small head without querying the total row count, DuckDB UI previews the first 100 rows, and DBeaver fetches a limited batch by default and makes “Fetch All” explicit.

Given that precedent, could we instead evaluate a lazy object once for the first N rows (possibly configurable, with N+1 used to detect truncation) and materialize that as a local snapshot (Ideally, it would be an .arrows / Arrow IPC stream file)? The viewer could then operate entirely on that snapshot, rather than repeatedly querying the backend and transferring paged results as JSON whenever the user scrolls, filters, or sorts.

This would also remove a substantial amount of backend-specific paging/cache/query logic. Full-data browsing could remain a separate explicit feature if we decide we need it later.

I have already implemented it so that each paged fetch is written to Arrow IPC during scrolling. If the user scrolls backward and the requested data is already cached in IPC, the cached data is reused. Requests beyond the cached range continue fetching from the existing result stream rather than re-evaluating the query. This has been done for arrow_dplyr_query, but I haven't pushed yet given some PRs tapping into sess. Just waiting for those to be cleared first. Also, I have disabled filtering and sorting for lazy query for now, and will review this later.

[Fred-Wu]

Superseded by #1876