Skip to content

perf(database): integrate CompactResultDataset into SQL drivers to eliminate 3x memory copies and string allocations #1355

Closed

Description

@ZhuchkaTriplesix

Summary

lib/core/database/compact_result_dataset.dart implements a columnar dataset CompactResultDataset with typed arrays (Int64List, Float64List), bit-packed null bitmaps, string interning, and lazy on-demand string generation via asLazyRowList(). It is fully tested in test/core/database/compact_result_dataset_test.dart, but is currently completely unused in production code.

Instead, PostgreSQL, MySQL, and SQLite drivers eagerly convert all query results into a full List<List<String>> matrix:

  1. Driver raw records (Result / taken.items / results).
  2. Intermediate rawRows (List<List<Object?>>).
  3. Eagerly stringified outRows (List<List<String>>).

For a 10,000–50,000 row result set across 30 columns, this creates up to 1,500,000 individual String objects on the Dart VM heap, taking 150–300 MB of RAM and causing large GC pauses, instead of 10–20 MB when packed in columnar typed data.

Proposed Solution

  1. Integrate CompactResultDataset into SqlExecutionResult, or construct CompactResultDataset.fromRawRows directly in PostgresSqlExecutionDelegate, MySqlSqlExecutionDelegate, and SqliteSqlExecutionDelegate.
  2. Provide rows to VirtualResultGrid via dataset.asLazyRowList(), so only visible cells in the viewport are stringified.

Acceptance Criteria

  • Memory footprint for queries returning $\ge 10,000$ rows is reduced by $\ge 60%$.
  • Intermediate List<List<Object?>> and List<List<String>> allocations are eliminated in favor of columnar typed storage.

Activity

  1. added
    performanceTheme parser epic label: performance
    coreCore library logic and services
    backendBackend database driver execution and queries
    P2Medium priority / Parity & Refactoring
    on Oct 10, 2026
  2. ZhuchkaTriplesix commented on Oct 11, 2026

    @ZhuchkaTriplesix
    MemberAuthor

    Resolved in #1464. CompactResultDataset integrated into SQLite, PostgreSQL, and MySQL execution delegates; SqlExecutionResult.fromDataset provides lazy viewport row access and maintains backwards compatibility.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Medium priority / Parity & RefactoringbackendBackend database driver execution and queriescoreCore library logic and servicesperformanceTheme parser epic label: performance

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions