Repository navigation
Conversation
added 2 commits
October 15, 2026 23:49
Add SQL Server table paging, primary-key discovery, and cached query execution with viewer filtering and sorting. Preserve temporal and integer64 values through Arrow IPC spill storage. Add shared column projection, unknown row counts, cancellation, cleanup, recovery, and required CI dependencies. Include DBI and primary-key regression tests.
Add progressive paging, projected random access, and cached query filtering and sorting. Preserve nested and temporal values, recover failed reads, and handle unknown row counts. Cover fragment and row-group access, spill caches, recovery, and platform-independent Arrow stream ordering in the regression tests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #1785
This PR is a clean replacement for #1804. It includes the implementation and subsequent fixes, incorporates the latest master, and reorganises the changes into two commits: DBI support with shared viewer infrastructure, followed by Arrow support.
I am submitting both implementations for review and discussion; they do not necessarily need to be adopted together.
Summary
Add on-demand Data Viewer support for Arrow datasets,
arrow_dplyr_queryobjects, and DBI-backedtbl_sqlobjects, currently targeting SQL Server.Opening and scrolling do not require collecting the complete result into an R data frame. Paging, filtering, and sorting use different strategies for datasets, bare database tables, and transformed lazy queries.
Arrow datasets
integer64, dates, datetimes, and durations. Disable filtering and sorting for nested columns.SQL Server tables
tbl_sql.OFFSETwhen repositioning outside the cache.Transformed lazy queries
For both
arrow_dplyr_queryand transformed database queries:Viewer filtering and sorting therefore act as additional operations over the original query result. They do not rewrite the user's query. Transformed database queries also require the optional
arrowpackage for disk caching.Shared viewer behaviour
Memory and current limitations
Block caches use a 20,000-row retention threshold, but this is not a strict total-memory limit. Individual Arrow batches can be larger, and filtering or sorting can require complete key columns and row indices in memory. Cached lazy-query results also consume temporary disk space.
Stable bare-table ordering depends on an available unique key. Source changes require reopening the viewer.
Large unsigned 64-bit integers and high-precision decimals remain a known precision limitation when converted to R numeric values, affecting display, filtering, and sorting.
Validation
Regression coverage includes sequential and random paging, fragments and row groups, projection, filtering and sorting, cache spilling, ordering, failure recovery, interruption, disposal, and supported column types. Arrow query tests account for platform-dependent batch arrival order.
Local Windows validation passed:
DBI tests use simulated connections and verify SQL generation and cursor behaviour; they do not replace live SQL Server/ODBC integration testing. macOS and Ubuntu CI results are not claimed by this local validation.