diff --git a/CHANGELOG.md b/CHANGELOG.md index b9a562f9..15c3ef5e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,3 +1,183 @@ +## 1.35.0 + +API/server layer clean-up (behavior-preserving): + +- `APIServerWorker`: both branches of `_processAPIResponse` end in one + `_finishResponse`; helpers renamed to what they do (`_buildShelfResponse`, + `_disposeAndCompressResponse`, `_startServersAndInitializeAPIRoot`, + `_errorResponse`). The `CORS` headers are set once (they were set again + when building the headers). +- Request payload: one `_readPayloadBytes` for text and binary payloads, one + `maxPayloadLength` check, done before copying the blocks (a single + `Uint8List` block is still used as is). +- `APIServer.toAPIRequest`: headers and query parameters flattened by one + helper. New constants: `APIServer.sessionIDCookieName` (`SESSIONID`), + `headerServerTiming`, `headerKeepAlive`, `headerContentLength`, + `headerXAPIServerWorker`; `APIRoot.apiInfoRoute` (`API-INFO`) and + `APIRoot.defaultAuthenticationRoute` (`authenticate`). +- `APISecurity`: one `resolveMapped` chain resolving a token's credential; + the `disposeAuthentication*` methods share one helper. +- `APIRoot`: the default-module fallback shared by `acceptsRequest` and the + call. +- `APIServerResponseCache`: an evicted entry is removed once (not per + alternative); one `GET` check in `cacheResponse`. +- `APICronScheduler._execute` split (`_runCalls`, `_storeRunResult`). +- `FieldsFromMap`: one lookup helper for the lower-case and simple keys. +- Private cache fields named `*Cache`, documented, declared before their + users. + +Conditions and SQL generation clean-up (behavior-preserving, the same SQL): + +- `Condition`: the ordering comparisons (`>`, `>=`, `<`, `<=`) written once + (a private enum and a private base of the `KeyCondition{GreaterThan,...}` + classes; public API unchanged); `KeyCondition.encodeValue` in the base + class; `ConditionParameter.matchesIn` resolves its parameters once. +- `EncodingContext.parameterValue`; `isReservedWord` by a set; + `ConditionSQLEncoder.qualifiedColumnSQL` and shared ID/empty-set helpers. +- `SQLDialect.qualifiedColumnSQL`; `SQLGenerator.uniqueConstraintName`, + `indexName`, `foreignKeyName` and `relationshipTableName` (the names of + the generated indexes, constraints and relationship tables, in one place). +- `DBSQLAdapter`: the table check (`_checkDBTableScheme`), the relationship + SQL generators and the `JOIN`s split into documented helpers. +- `DBSQLMemoryAdapter`: one select matcher; shared delete helpers. +- `APIDBModule.select`: directives, table and pagination helpers. +- Cache fields named `*Cache` and documented. + +Entity model clean-up (behavior-preserving, except the fixes below): + +- Entity rules: `EntityResolutionRules.merge`/`fetchTypes` and + `EntityAccessRules` (group flattening, condition check, reversed scan) + through shared helpers. +- `EntityHandler`: one scalar equality chain for `equalsValues` and + `equalsValuesBasic`; the comparisons through one numeric helper; a shared + cached field-type lookup for `GenericEntityHandler` and + `ClassReflectionEntityHandler`; private caches named `*Cache` and + documented. +- `Transaction`: identity sets beside the operation lists, for O(1) + membership checks (a bulk `storeAll` was O(n²)), created once a list + passes 16 operations (a smaller transaction scans its list, cheaper than + allocating the sets). Memory `remove` in the benchmark: 4.9× faster. +- Cascade delete, `storeAllFromJson` (named block-size constants), the seed + JSON helpers, the provider lookups, `IterableEntityRepository` and + `TransactionOperation.toString`: duplicated blocks moved into helpers. +- `EntityReference`/`EntityReferenceList`, `TypeInfo` and `Map` getter + extensions, `Time`, `EntityPagination`: duplicated code merged. +- `isEqualsIterableDeep`/`intersectsIterableDeep`: iterate instead of + `elementAt(i)` (O(n²) on a non-`List` `Iterable`). + +Adapters clean-up (behavior-preserving): + +- `DBObjectAdapter`: the implementation copy-pasted in the memory, + directory and GCS object adapters (table schemes, transactions, + order/pagination, ID/table names and entity JSON normalization, operation + execution), with the hooks `createAdapterException` (each adapter keeps + its exception type), `encodeBytesAsDataURL` and `adapterLogger`. +- `DBSQLAdapter`: the schema introspection of the PostgreSQL, MySQL and + SQLite adapters (tables names and fields references caches, relationship + tables) over `listTablesNamesImpl`/`findFieldsReferencedTablesImpl`; the + 30 s fields references cache TTL named; `openTransactionWithPool`, the + shared transaction defaults and `autoInsertAfterNoopUpdate`. +- `DBEntityRepository`: one `ensureStored` implementation (hooks + `storeWithParentOperation`, `onEnsureStoredExisting`) and + `newSelectOperation`; `getEntityRepositoriesTables` (the typo + `getEntityRepositoresTables` kept, `@Deprecated`). +- Cache fields named `*Cache` and documented. +- New API (for subclasses of the adapters): + - `DBSQLAdapter`: `@protected` `openTransactionWithPool`, + `listTablesNamesImpl`, `findFieldsReferencedTablesImpl` (default: throw + `UnsupportedError`, for an adapter that doesn't introspect a database), + `listTablesNamesCached`, `findFieldsReferencedTablesCached`, + `findRelationshipTables`, `autoInsertAfterNoopUpdate`; the static + `sqlStringLiteral`. + - `DBSQLAdapter` defaults for what was abstract: `cancelTransaction` + (`true`), `closeTransaction` (nothing), `callCloseTransactionRequired` + (`false`), `cancelTransactionResultWithError` (`false`), + `throwTransactionResultWithError` (`true`), the behavior of a pooled + driver whose transaction callback commits or rolls back (as the + PostgreSQL, MySQL and SQLite adapters): a subclass that relied on them + being abstract now inherits these. + - `DBObjectAdapter`: `@protected` `executeOperation`, + `normalizeEntityJSON`, `normalizeObjectID`, `normalizeTableName`, + `getTableIDFieldName`, `applyTableOrderAndPagination`, + `createAdapterException`, `encodeBytesAsDataURL`, `adapterLogger`, + `onTransactionContextCanceled`, `onTransactionContextClosed`. + - The object adapters' `openTransaction` stays typed as their context + (synchronous); `cancelTransaction` of the MySQL/SQLite adapters stays + `bool`. +- `DBSQLMemoryAdapter` keeps its own relationship-tables resolution (it + doesn't introspect a database). + +Deprecated (still working): + +- `EntityHandler.graterThanValue`/`graterThanOrEqualValue`: use + `greaterThanValue`/`greaterThanOrEqualValue`. +- `EntityRulesResolver.cotextProviders`: use `contextProviders`. +- `FieldsFromMap.getFieldsKeysInMap`'s `fieldLC`/`fieldSimple` (ignored). +- `DBAdapter.getEntityRepositoresTables`: use + `getEntityRepositoriesTables`. + +Fixes: + +- `APIServerResponseCache`: the eviction (`maxMemorySize` reached) sorted + the cached values, which hold a `List` for an entry with compressed + alternatives: a `TypeError`. The entries are now sorted with all their + alternatives (by total usage and memory size). +- `APIServerResponseCache`: the eviction removed the most used entries + first (usage sorted descending): now the least used first, and among + equally used, the largest first. +- `APIServerResponseCache`: an expired entry (file modified) was removed + without subtracting its memory size from `totalMemorySize`. +- `APIServerConfig`: the user directory (Let's Encrypt paths) fell back to + `Directory('~/')`, a literal `~` directory, not the home: it's resolved + from the environment only (`resolveUserDirectory`): `USERPROFILE` then + `HOME` on Windows, `HOME` then `USERPROFILE` elsewhere (other platforms, + e.g. Android, used to have none). +- `APIServerConfig.normalizeHeaderValue`: the `k3` path key was ignored (a + value at a 4-key config path returned the default): now used. +- `APIServer`: the JSON-decode error log printed the stack trace in place + of the invalid payload (a shadowed `catch` variable). +- `IdenticalSet`: grouped elements by `hashCode`, missing an element whose + `hashCode` changed: now by `identityHashCode`. +- `FieldsFromMap.getFieldsKeysInMap`: the given `fieldLC`/`fieldSimple` + (of one field) were used for every field: now computed per field (the + parameters are ignored and `@Deprecated`). +- `Condition.parse`: an escaped quote in a single-quoted string + (`name == 'a\'b'`) failed to parse. +- `SQLDialect.toString`: lists all its fields (and a missing space). +- `APIDBModule.select`: the entities' values are HTML-escaped in its page + (they were written as is: a `<` or `&` broke the page, and could inject + markup). +- `GenericEntityHandler.getFieldType` with `resolveFiledName`: the entity is + asked for the resolved field name (it was asked for the key again). +- `EntityRepositoryProvider.populateFromSource`: `workingPath` is used to + resolve the source (and the variables) file. +- `EntityRepositoryProvider.storeAllFromJson`/`storeAllFromJsonEncoded`: + `transaction` is used, and the `%name%` `variables` are resolved (they + were ignored). +- `EntityReference.from`/`EntityReferenceList.from`: `entityHandlerProvider` + is forwarded. +- `IterableEntityRepository.deleteEntity`: `transaction` is forwarded. +- `EntityAccessRules` `==`/`hashCode`: `entityFields` compared element-wise + (it was identity). +- `EntityResolutionRules` with `allEager` and `allLazy` both `true`: invalid + (`validate` throws `ValidateEntityRulesError`); + `isEagerEntityType`/`isLazyEntityType` recursed forever on it. A `merge` + of one side's `allEager` with the other's `allLazy` drops the conflict as + a same-flag one: both sides tolerant, both flags dropped; one tolerant, it + yields its flag; neither, it throws. +- `DBPostgreSQLAdapter.doCountSQL`: a result without a row threw a + `StateError`: now `0`, as the SQLite and MySQL adapters. +- `DBPostgreSQLAdapter`: the ID sequence fix after an update's auto-insert + used `max(id)`, failing for a table whose ID column isn't `id`: now the + table's (quoted) ID column. +- `DBPostgreSQLAdapter`, `DBMySQLAdapter`: the schema introspection + queries interpolated the table (and database) names into SQL string + literals unescaped: a `'` in a name broke (or changed) the query. Now + escaped by `DBSQLAdapter.sqlStringLiteral`: the `'` doubled, and for + MySQL (`DBMySQLAdapter.mysqlStringLiteral`; its default `sql_mode` makes + `\` an escape in literals) also the `\`. The queries are otherwise the + same (MySQL's foreign keys query lost two trailing spaces). + ## 1.34.0 `null` and `[]` mean the same in a condition: an empty set for `=~`/`!~`, a diff --git a/benchmark/db_users/HISTORY.md b/benchmark/db_users/HISTORY.md index 4d6f6f7e..9e9267b1 100644 --- a/benchmark/db_users/HISTORY.md +++ b/benchmark/db_users/HISTORY.md @@ -18,52 +18,100 @@ ops/sec per operation (higher is better), one column per date, oldest first, from the entries below; in parentheses, the change from the previous date. A date with more than one run shows its last one (2026-10-03: the optimized runs at `98cc744 (modified)`; the -first runs of that day are in their entries). When adding an entry, add -its date's column here too. +first runs of that day are in their entries; 2026-11-08: 1.35.0 at +`00e258c`, the 1.34.0 runs of that day are in their entries). When +adding an entry, add its date's column here too. ### `memory` | Operation | 2026-10-03 | 2026-10-13 | 2026-10-14 | 2026-11-08 | |---|--:|--:|--:|--:| -| `user/byId` | 25303 | 30312 (+19.8%) | 30937 (+2.1%) | 36725 (+18.7%) | -| `user/byEmail` | 1407 | 1449 (+3.0%) | 1437 (-0.8%) | 1428 (-0.6%) | -| `user/byState (join, limit 20)` | 1409 | 1594 (+13.1%) | 1601 (+0.4%) | 1790 (+11.8%) | -| `user/list (page of 20)` | 682 | 720 (+5.6%) | 719 (-0.1%) | 766 (+6.5%) | -| `user/count` | 193802 | 320214 (+65.2%) | 320629 (+0.1%) | 244367 (-23.8%) | -| `authenticate (login)` | 1393 | 1459 (+4.7%) | 1491 (+2.2%) | 1435 (-3.8%) | -| `user/register` | 2623 | 3514 † | 3525 (+0.3%) | 3553 (+0.8%) | -| `user/update` | 1881 | 5215 † | 5190 (-0.5%) | 5444 (+4.9%) | -| `user/remove` | 19969 | 12506 † | 12476 (-0.2%) | 2909 (-76.7%) ‡ | +| `user/byId` | 25303 | 30312 (+19.8%) | 30937 (+2.1%) | 37645 (+21.7%) | +| `user/byEmail` | 1407 | 1449 (+3.0%) | 1437 (-0.8%) | 1399 (-2.6%) | +| `user/byState (join, limit 20)` | 1409 | 1594 (+13.1%) | 1601 (+0.4%) | 1817 (+13.5%) | +| `user/list (page of 20)` | 682 | 720 (+5.6%) | 719 (-0.1%) | 748 (+4.0%) | +| `user/count` | 193802 | 320214 (+65.2%) | 320629 (+0.1%) | 243874 (-23.9%) | +| `authenticate (login)` | 1393 | 1459 (+4.7%) | 1491 (+2.2%) | 1476 (-1.0%) | +| `user/register` | 2623 | 3514 † | 3525 (+0.3%) | 3503 (-0.6%) | +| `user/update` | 1881 | 5215 † | 5190 (-0.5%) | 5389 (+3.8%) | +| `user/remove` | 19969 | 12506 † | 12476 (-0.2%) | 13924 (+11.6%) ‡ | ### `sqlite` | Operation | 2026-10-03 | 2026-10-13 | 2026-10-14 | 2026-11-08 | |---|--:|--:|--:|--:| -| `user/byId` | 17138 | 21220 (+23.8%) | 24721 (+16.5%) | 29556 (+19.6%) | -| `user/byEmail` | 16580 | 20528 (+23.8%) | 24067 (+17.2%) | 28649 (+19.0%) | -| `user/byState (join, limit 20)` | 2326 | 3184 (+36.9%) | 3283 (+3.1%) | 4382 (+33.5%) | -| `user/list (page of 20)` | 2405 | 3352 (+39.4%) | 3433 (+2.4%) | 4731 (+37.8%) | -| `user/count` | 121903 | 162445 (+33.3%) | 195377 (+20.3%) | 190389 (-2.6%) | -| `authenticate (login)` | 14764 | 18140 (+22.9%) | 20985 (+15.7%) | 24460 (+16.6%) | -| `user/register` | 7727 | 8284 (+7.2%) | 9719 (+17.3%) | 9520 (-2.0%) | -| `user/update` | 10302 | 11665 (+13.2%) | 13774 (+18.1%) | 15050 (+9.3%) | -| `user/remove` | 13036 | 14247 (+9.3%) | 16953 (+19.0%) | 17013 (+0.4%) | +| `user/byId` | 17138 | 21220 (+23.8%) | 24721 (+16.5%) | 29362 (+18.8%) | +| `user/byEmail` | 16580 | 20528 (+23.8%) | 24067 (+17.2%) | 28337 (+17.7%) | +| `user/byState (join, limit 20)` | 2326 | 3184 (+36.9%) | 3283 (+3.1%) | 4394 (+33.8%) | +| `user/list (page of 20)` | 2405 | 3352 (+39.4%) | 3433 (+2.4%) | 4734 (+37.9%) | +| `user/count` | 121903 | 162445 (+33.3%) | 195377 (+20.3%) | 185330 (-5.1%) | +| `authenticate (login)` | 14764 | 18140 (+22.9%) | 20985 (+15.7%) | 24323 (+15.9%) | +| `user/register` | 7727 | 8284 (+7.2%) | 9719 (+17.3%) | 8839 (-9.1%) | +| `user/update` | 10302 | 11665 (+13.2%) | 13774 (+18.1%) | 14734 (+7.0%) | +| `user/remove` | 13036 | 14247 (+9.3%) | 16953 (+19.0%) | 16198 (-4.5%) | ### `postgres` | Operation | 2026-10-03 | 2026-10-13 | 2026-10-14 | 2026-11-08 | |---|--:|--:|--:|--:| -| `user/byId` | 1453 | 1436 (-1.2%) | 1795 (+25.0%) | 1629 (-9.2%) | -| `user/byEmail` | 1518 | 1414 (-6.9%) | 1823 (+28.9%) | 1669 (-8.4%) | -| `user/byState (join, limit 20)` | 537 | 733 (+36.5%) | 887 (+21.0%) | 1069 (+20.5%) | -| `user/list (page of 20)` | 490 | 810 (+65.3%) | 888 (+9.6%) | 1148 (+29.3%) | -| `user/count` | 6167 | 5424 (-12.0%) | 6693 (+23.4%) | 6033 (-9.9%) | -| `authenticate (login)` | 1474 | 1350 (-8.4%) | 1724 (+27.7%) | 1668 (-3.2%) | -| `user/register` | 611 | 629 (+2.9%) | 718 (+14.1%) | 927 (+29.1%) | -| `user/update` | 713 | 684 (-4.1%) | 850 (+24.3%) | 875 (+2.9%) | -| `user/remove` | 803 | 762 (-5.1%) | 890 (+16.8%) | 1187 (+33.4%) | - -2026-11-08: 1.34.0 at `024551e` (PR #205, the condition operators' fixes +| `user/byId` | 1453 | 1436 (-1.2%) | 1795 (+25.0%) | 1774 (-1.2%) | +| `user/byEmail` | 1518 | 1414 (-6.9%) | 1823 (+28.9%) | 1764 (-3.2%) | +| `user/byState (join, limit 20)` | 537 | 733 (+36.5%) | 887 (+21.0%) | 1122 (+26.5%) | +| `user/list (page of 20)` | 490 | 810 (+65.3%) | 888 (+9.6%) | 1174 (+32.2%) | +| `user/count` | 6167 | 5424 (-12.0%) | 6693 (+23.4%) | 6439 (-3.8%) | +| `authenticate (login)` | 1474 | 1350 (-8.4%) | 1724 (+27.7%) | 1555 (-9.8%) | +| `user/register` | 611 | 629 (+2.9%) | 718 (+14.1%) | 820 (+14.2%) | +| `user/update` | 713 | 684 (-4.1%) | 850 (+24.3%) | 748 (-12.0%) | +| `user/remove` | 803 | 762 (-5.1%) | 890 (+16.8%) | 1032 (+16.0%) | + +2026-11-08 shows 1.35.0 at `00e258c` (PR #209: the clean-up of the API, +conditions/SQL, entity model and adapters, behavior-preserving, and its +fixes), the median of 5 rounds (`--repeat 5`). 1.34.0 (`32e1dbf`) was run +in the same session, alternating with it on each DB (PostgreSQL: one +container, `--pg-host`, as below). + +1.35.0 vs 1.34.0, same session (ops/sec; memory and SQLite from the +alternated `--repeat 5` runs; PostgreSQL the medians of an A/B +alternating the two 3 times, `--repeat 3`, on the operations shown): + +| Operation | memory 1.34.0 | 1.35.0 | | sqlite 1.34.0 | 1.35.0 | | postgres 1.34.0 | 1.35.0 | | +|---|--:|--:|--:|--:|--:|--:|--:|--:|--:| +| `user/byId` | 37053 | 37645 | +1.6% | 28918 | 29362 | +1.5% | 1766 | 1770 | +0.2% | +| `user/byEmail` | 1426 | 1399 | -1.9% | 28143 | 28337 | +0.7% | | | | +| `user/byState (join, limit 20)` | 1836 | 1817 | -1.0% | 4344 | 4394 | +1.2% | 1116 | 1121 | +0.4% | +| `user/list (page of 20)` | 780 | 748 | -4.1% | 4681 | 4734 | +1.1% | 1179 | 1189 | +0.8% | +| `user/count` | 244480 | 243874 | -0.2% | 164870 | 185330 | +12.4% | 6545 | 6440 | -1.6% | +| `authenticate (login)` | 1485 | 1476 | -0.6% | 24296 | 24323 | +0.1% | | | | +| `user/register` | 3579 | 3503 | -2.1% | 8710 | 8839 | +1.5% | | | | +| `user/update` | 5434 | 5389 | -0.8% | 14723 | 14734 | +0.1% | | | | +| `user/remove` | 2859 | 13924 | +387% | 16091 | 16198 | +0.7% | | | | + +- Memory `remove` ‡: 4.9× faster, back above 1.22.0's 12476. Most likely + the `Transaction` identity sets: it checked its operations with + `List.contains` (O(n) per operation added or executed), and a delete + adds many operations (not profiled; it's the change on that path). +- Those sets were first allocated with every `Transaction`, which cost + the cheapest operations a little (an A/B alternating 1.34.0 and the + eager sets: SQLite `count` -4.0%, `byId` -1.3%). They're now created + only past 16 operations: the same A/B with the lazy sets measured + SQLite `count` 193219 vs 189273 (-2.0%, runs 190907–193334 vs + 189246–191221) and `byId` -0.9%, PostgreSQL within ±1.6%. SQLite `count` + (~5 us a call) varies between runs: +12.4% in the table's runs. +- Everything else is within ±2% except memory `list` (-4.1%, rounds + 736–789 vs 777–789, overlapping). +- PostgreSQL in Docker Desktop drifted (and stalled) during this session: + two 1.35.0 runs had stalled rounds (`list` 399–1123 and 362–560 + ops/sec), and in the entry's run the writes (the last operations) and + the 1.34.0 run after it caught a drop (1.34.0 `list` 848). A commit A/B + (`4a867d4`, before the lazy sets, vs `00e258c`, alternated twice on one + container) measured them the same: `list` 1113 vs 1093, `count` 5563 vs + 5577. The entry's reads match the A/B; its writes (`update` 748, + `login` 1555) are lower from the drift, not the code. +- The PR's runs needed `sqlite3`'s native library, which GitHub's release + downloads failed to serve for a while (HTTP 504; reported in + simolus3/sqlite3.dart#435): those runs were repeated once it served. + +2026-11-08 (earlier): 1.34.0 at `024551e` (PR #205, the condition operators' fixes of 1.32.1–1.34.0), the median of 5 rounds (`--repeat 5`). The change from 2026-10-14 spans 1.22.0 → 1.34.0, not only these fixes: to measure them, 1.32.1, 1.33.0 and this PR were run one after another in the same session @@ -101,9 +149,10 @@ These fixes, same machine and session (ops/sec; PR vs 1.32.1): `count` 5712–7006). The table's drop is the container's drift during the session (see the 2026-10-13 notes), not these fixes. -‡ Memory `remove` is ~2.7k ops/sec on all three versions (1.32.1 too): the +‡ Memory `remove` was ~2.7k ops/sec on all three versions (1.32.1 too): the drop from 12476 (2026-10-14, 1.22.0) is older than these fixes (between -1.22.0 and 1.32.1), as `count`'s -24%. Not investigated here. +1.22.0 and 1.32.1), as `count`'s -24%. Not investigated here. 1.35.0 +recovered it (13924; see its notes above); `count`'s -24% remains. The machine's label here is the one detected (`Apple M5`; the earlier entries were labeled `Mac17,3 (Apple M5)`): the same machine. Two idle @@ -140,6 +189,83 @@ scans tables, so its numbers depend the most on the table size. ### 2026-11-08 — `postgres` — Apple M5 +- **bones_api:** 1.35.0 (`00e258c`) +- **Machine:** Apple M5, 10 cores, 16 GB RAM +- **OS:** macos (Version 26.6.2 (Build 25G83)) +- **Dart:** 3.13.5 +- **DB:** postgres (PostgreSQL 18.6 (Debian 18.6-1.pgdg13+2), localhost:55432) +- **Users:** 1000; **duration:** 3s (warm-up 1s) × 5 rounds, median shown per operation + +| Operation | ops/sec | mean (us) | p50 (us) | p95 (us) | p99 (us) | rounds (ops/sec) | +|---|--:|--:|--:|--:|--:|--:| +| `user/byId` | 1774 | 563.7 | 558 | 605 | 667 | 1630–1783 | +| `user/byEmail` | 1764 | 567.0 | 562 | 610 | 658 | 1760–1774 | +| `user/byState (join, limit 20)` | 1122 | 891.2 | 882 | 957 | 1008 | 1101–1140 | +| `user/list (page of 20)` | 1174 | 851.6 | 845 | 916 | 954 | 1172–1197 | +| `user/count` | 6439 | 155.3 | 154 | 168 | 187 | 6225–6521 | +| `authenticate (login)` | 1555 | 643.0 | 639 | 694 | 761 | 1525–1590 | +| `user/register` | 820 | 1220.2 | 1197 | 1532 | 1618 | 809–835 | +| `user/update` | 748 | 1337.0 | 1320 | 1448 | 1531 | 713–755 | +| `user/remove` | 1032 | 969.2 | 963 | 1008 | 1071 | 1026–1058 | + +1.35.0, PR #209. On a fresh `postgres:latest` container in Docker Desktop +(`--pg-host`), run first, then 1.34.0. The container drifted down from +the writes (the last operations) on, into the 1.34.0 run after it: the +reads here match the A/Bs of the session (see the progression's notes), +the writes are lower from that drift. + +### 2026-11-08 — `sqlite` — Apple M5 + +- **bones_api:** 1.35.0 (`00e258c`) +- **Machine:** Apple M5, 10 cores, 16 GB RAM +- **OS:** macos (Version 26.6.2 (Build 25G83)) +- **Dart:** 3.13.5 +- **DB:** sqlite (temporary file) +- **Users:** 1000; **duration:** 3s (warm-up 1s) × 5 rounds, median shown per operation + +| Operation | ops/sec | mean (us) | p50 (us) | p95 (us) | p99 (us) | rounds (ops/sec) | +|---|--:|--:|--:|--:|--:|--:| +| `user/byId` | 29362 | 34.1 | 33 | 38 | 77 | 29265–29419 | +| `user/byEmail` | 28337 | 35.3 | 34 | 39 | 79 | 28243–28350 | +| `user/byState (join, limit 20)` | 4394 | 227.6 | 221 | 278 | 298 | 4361–4396 | +| `user/list (page of 20)` | 4734 | 211.2 | 206 | 263 | 280 | 4726–4738 | +| `user/count` | 185330 | 5.4 | 5 | 6 | 7 | 183238–186039 | +| `authenticate (login)` | 24323 | 41.1 | 39 | 45 | 105 | 24300–24427 | +| `user/register` | 8839 | 113.1 | 86 | 109 | 1868 | 7979–8975 | +| `user/update` | 14734 | 67.9 | 64 | 72 | 86 | 14445–14746 | +| `user/remove` | 16198 | 61.7 | 48 | 53 | 82 | 15917–16441 | + +1.35.0, PR #209, alternated with 1.34.0 in the same session. + +### 2026-11-08 — `memory` — Apple M5 + +- **bones_api:** 1.35.0 (`00e258c`) +- **Machine:** Apple M5, 10 cores, 16 GB RAM +- **OS:** macos (Version 26.6.2 (Build 25G83)) +- **Dart:** 3.13.5 +- **DB:** memory (DBSQLMemoryAdapter) +- **Users:** 1000; **duration:** 3s (warm-up 1s) × 5 rounds, median shown per operation + +| Operation | ops/sec | mean (us) | p50 (us) | p95 (us) | p99 (us) | rounds (ops/sec) | +|---|--:|--:|--:|--:|--:|--:| +| `user/byId` | 37645 | 26.6 | 25 | 29 | 41 | 37510–37914 | +| `user/byEmail` | 1399 | 714.7 | 713 | 1330 | 1399 | 1375–1489 | +| `user/byState (join, limit 20)` | 1817 | 550.4 | 540 | 615 | 631 | 1795–1821 | +| `user/list (page of 20)` | 748 | 1337.2 | 1330 | 1409 | 1423 | 736–789 | +| `user/count` | 243874 | 4.1 | 4 | 4 | 4 | 243519–244245 | +| `authenticate (login)` | 1476 | 677.7 | 679 | 1244 | 1323 | 1420–1547 | +| `user/register` | 3503 | 285.5 | 282 | 415 | 452 | 3433–3648 | +| `user/update` | 5389 | 185.6 | 184 | 219 | 236 | 5386–5414 | +| `user/remove` | 13924 | 71.8 | 70 | 89 | 94 | 8075–14047 | + +1.35.0, PR #209, alternated with 1.34.0 in the same session. `remove`: +4.9× 1.34.0's (see the progression's notes); one round was much lower +(8075, the others 13.8k–14.0k). + +--- + +### 2026-11-08 — `postgres` — Apple M5 + - **bones_api:** 1.34.0 (`024551e`) - **Machine:** Apple M5, 10 cores, 16 GB RAM - **OS:** macos (Version 26.6.2 (Build 25G83)) diff --git a/lib/src/bones_api_base.dart b/lib/src/bones_api_base.dart index 92c6ec26..d9aaa8f0 100644 --- a/lib/src/bones_api_base.dart +++ b/lib/src/bones_api_base.dart @@ -52,7 +52,7 @@ typedef APILogger = void Function( /// Bones API Library class. class BonesAPI { // ignore: constant_identifier_names - static const String VERSION = '1.34.0'; + static const String VERSION = '1.35.0'; static bool _boot = false; @@ -1033,7 +1033,7 @@ abstract class APIRoot with Initializable, Closable { // first part. var pathPartRoot = apiRequest.pathPartFirst; - if (pathPartRoot == 'API-INFO') { + if (pathPartRoot == apiInfoRoute) { var info = apiInfo(apiRequest); return APIResponse.ok(info as T, mimeType: _mimeTypeJson); } @@ -1068,13 +1068,7 @@ abstract class APIRoot with Initializable, Closable { /// Returns `true` if [apiRequest] is an accepted route/call. bool acceptsRequest(APIRequest apiRequest) { - var module = getModuleByRequest(apiRequest); - if (module == null) { - var def = defaultModuleName; - if (def != null) { - module = _modules![def]; - } - } + var module = _resolveModuleOrDefault(getModuleByRequest(apiRequest)); if (module == null) { if (apiRequest.pathPartAt(0) == authenticationRoute || @@ -1096,12 +1090,7 @@ abstract class APIRoot with Initializable, Closable { APIModule? module, APIRequest request, ) { - if (module == null) { - var def = defaultModuleName; - if (def != null) { - module = _modules![def]; - } - } + module = _resolveModuleOrDefault(module); if (module == null) { return onNoRouteForPath(request); @@ -1110,6 +1099,19 @@ abstract class APIRoot with Initializable, Closable { return module.call(request); } + /// Returns [module] (the module matched by the request path), or the + /// [defaultModuleName] module when no module matched. + /// Shared by [acceptsRequest] and [_callModule], so both resolve a request + /// to the same module. It takes the already matched module (not the + /// request) because `_callImpl` checks the authentication route between + /// matching and this fallback. + APIModule? _resolveModuleOrDefault(APIModule? module) { + if (module != null) return module; + + var def = defaultModuleName; + return def != null ? _modules![def] : null; + } + FutureOr> onNoRouteForPath(APIRequest request) { var posResponse = callHandlers( posApiRequestHandlers, @@ -1126,22 +1128,31 @@ abstract class APIRoot with Initializable, Closable { return _responseNotFoundNoRouteForPath(request); } - String get authenticationRoute => 'authenticate'; + /// The route that returns the [APIRootInfo]/[APIModuleInfo] of an API. + static const String apiInfoRoute = 'API-INFO'; + + /// The default [authenticationRoute] (of [APIRoot] and [APIModule]). + static const String defaultAuthenticationRoute = 'authenticate'; + + String get authenticationRoute => defaultAuthenticationRoute; String? get securityModuleName => null; APIModule? get securityModule => _securityModuleImpl(); - APIModule? _securityModule; + /// Cache of the module named [securityModuleName], resolved by + /// [_securityModuleImpl] (a lookup in the modules map), so that each + /// [securityModule] read doesn't repeat the lookup. + APIModule? _securityModuleCache; APIModule? _securityModuleImpl() { - if (_securityModule != null) return _securityModule; + if (_securityModuleCache != null) return _securityModuleCache; var securityModuleName = this.securityModuleName; if (securityModuleName != null) { var module = getModule(securityModuleName); if (module != null) { - return _securityModule = module; + return _securityModuleCache = module; } else { throw StateError( "Can't find security module with name: $securityModuleName", @@ -1154,15 +1165,20 @@ abstract class APIRoot with Initializable, Closable { APISecurity? get security => _securityImpl(); - APISecurity? _security; - bool _securityResolved = false; + /// Cache of the [APISecurity] of the [securityModule] (possibly `null`), + /// filled by [_securityImpl]: [security] is read on every call. + APISecurity? _securityCache; + + /// Whether [_securityCache] was resolved: needed because `null` is a valid + /// (cached) [security]. + bool _securityCacheResolved = false; APISecurity? _securityImpl() { - if (_securityResolved) return _security; - _securityResolved = true; + if (_securityCacheResolved) return _securityCache; + _securityCacheResolved = true; var securityModule = this.securityModule; - return _security = securityModule?.security; + return _securityCache = securityModule?.security; } @override @@ -2269,10 +2285,12 @@ class APIRequest extends APIMetricSet with APIPayload { return idx >= 0 && idx < length ? _pathParts[idx] : ''; } - String? _lastPathPart; + /// A cache of [lastPathPart]: the request's path doesn't change, and the + /// last path part is read on every routing of the request. + String? _lastPathPartCache; /// Returns the last [pathParts]. Sames as [pathPartReversed] for index `0`. - String get lastPathPart => _lastPathPart ??= pathPartReversed(0); + String get lastPathPart => _lastPathPartCache ??= pathPartReversed(0); /// Returns from [parameters] a value for [name1]. Defaults to [def] if defined. V? getParameter(String name1, [V? def]) { @@ -2501,9 +2519,11 @@ class APIRequest extends APIMetricSet with APIPayload { } } - String? _hostname; + /// Cache of [hostname], parsed from the `Host` header by [_hostnameImpl] + /// on first read (the headers don't change during the request). + String? _hostnameCache; - String get hostname => _hostname ??= _hostnameImpl(); + String get hostname => _hostnameCache ??= _hostnameImpl(); String _hostnameImpl() { var host = getHeader('host'); @@ -2525,9 +2545,11 @@ class APIRequest extends APIMetricSet with APIPayload { return host; } - int? _port; + /// Cache of [port], parsed from the `Port`/`Host` headers by [_portImpl] + /// on first read. + int? _portCache; - int get port => _port ??= _portImpl(); + int get port => _portCache ??= _portImpl(); int _portImpl() { var port = getHeader('port'); @@ -2553,10 +2575,13 @@ class APIRequest extends APIMetricSet with APIPayload { String get hostnameAndPort => '$hostname:$port'; - String? _requesterAddressResolved; + /// Cache of [requesterAddress], resolved by [_requesterAddressImpl] + /// (the given address, or the `X-Forwarded-For`/`Remote-Address` + /// headers). A `null` result isn't cached (it's re-resolved on each read). + String? _requesterAddressCache; String? get requesterAddress => - _requesterAddressResolved ??= _requesterAddressImpl(); + _requesterAddressCache ??= _requesterAddressImpl(); String? _requesterAddressImpl() { if (_requesterAddress != null) return _requesterAddress; @@ -2579,9 +2604,11 @@ class APIRequest extends APIMetricSet with APIPayload { return null; } - String? _origin; + /// Cache of [origin], resolved by [_originImpl] (the `Origin` header, or + /// built from [scheme] and [hostnameAndPort]) on first read. + String? _originCache; - String get origin => _origin ??= _originImpl(); + String get origin => _originCache ??= _originImpl(); String _originImpl() { var origin = getHeader('origin'); diff --git a/lib/src/bones_api_condition.dart b/lib/src/bones_api_condition.dart index b012697c..21c62cda 100644 --- a/lib/src/bones_api_condition.dart +++ b/lib/src/bones_api_condition.dart @@ -270,27 +270,14 @@ class ConditionParameter extends ConditionElement { List? positionalParameters, Map? namedParameters, EntityHandler? entityHandler, - }) { - var myValue = getValue( - parameters: parameters, - positionalParameters: positionalParameters, - namedParameters: namedParameters, - ); - - var otherValue = value is ConditionParameter - ? value.getValue( - parameters: parameters, - positionalParameters: positionalParameters, - namedParameters: namedParameters, - ) - : value; - - return EntityHandler.graterThanValue( - myValue, - otherValue, - entityHandler: entityHandler, - ); - } + }) => _compareOrdering( + _Ordering.gt, + value, + parameters, + positionalParameters, + namedParameters, + entityHandler, + ); bool greaterThanOrEqual( Object? value, { @@ -298,27 +285,14 @@ class ConditionParameter extends ConditionElement { List? positionalParameters, Map? namedParameters, EntityHandler? entityHandler, - }) { - var myValue = getValue( - parameters: parameters, - positionalParameters: positionalParameters, - namedParameters: namedParameters, - ); - - var otherValue = value is ConditionParameter - ? value.getValue( - parameters: parameters, - positionalParameters: positionalParameters, - namedParameters: namedParameters, - ) - : value; - - return EntityHandler.graterThanOrEqualValue( - myValue, - otherValue, - entityHandler: entityHandler, - ); - } + }) => _compareOrdering( + _Ordering.gte, + value, + parameters, + positionalParameters, + namedParameters, + entityHandler, + ); bool lessThan( Object? value, { @@ -326,27 +300,14 @@ class ConditionParameter extends ConditionElement { List? positionalParameters, Map? namedParameters, EntityHandler? entityHandler, - }) { - var myValue = getValue( - parameters: parameters, - positionalParameters: positionalParameters, - namedParameters: namedParameters, - ); - - var otherValue = value is ConditionParameter - ? value.getValue( - parameters: parameters, - positionalParameters: positionalParameters, - namedParameters: namedParameters, - ) - : value; - - return EntityHandler.lessThanValue( - myValue, - otherValue, - entityHandler: entityHandler, - ); - } + }) => _compareOrdering( + _Ordering.lt, + value, + parameters, + positionalParameters, + namedParameters, + entityHandler, + ); bool lessThanOrEqual( Object? value, { @@ -354,7 +315,28 @@ class ConditionParameter extends ConditionElement { List? positionalParameters, Map? namedParameters, EntityHandler? entityHandler, - }) { + }) => _compareOrdering( + _Ordering.lte, + value, + parameters, + positionalParameters, + namedParameters, + entityHandler, + ); + + /// Compares this parameter's value (the left side) with [value] (the right + /// side, itself resolved if it's a [ConditionParameter]) by [ordering]. + /// + /// The shared body of [greaterThan], [greaterThanOrEqual], [lessThan] and + /// [lessThanOrEqual], which only differ by the [ordering]. + bool _compareOrdering( + _Ordering ordering, + Object? value, + Object? parameters, + List? positionalParameters, + Map? namedParameters, + EntityHandler? entityHandler, + ) { var myValue = getValue( parameters: parameters, positionalParameters: positionalParameters, @@ -369,11 +351,7 @@ class ConditionParameter extends ConditionElement { ) : value; - return EntityHandler.lessThanOrEqualValue( - myValue, - otherValue, - entityHandler: entityHandler, - ); + return ordering.test(myValue, otherValue, entityHandler); } bool matchesIn( @@ -391,18 +369,25 @@ class ConditionParameter extends ConditionElement { checkNotRegExpValue(myValue); - if (myValue is List) { - for (var v1 in myValue) { - checkNotRegExpValue(v1); - for (var v2 in values) { - var otherValue = v2 is ConditionParameter + // Resolve the parameters among [values] once, not once per element of + // `myValue` (`getValue` is pure, but it may convert the `parameters` + // object with `Json.toJson` on every call): + var otherValues = values + .map( + (v2) => v2 is ConditionParameter ? v2.getValue( parameters: parameters, positionalParameters: positionalParameters, namedParameters: namedParameters, ) - : v2; + : v2, + ) + .toList(); + if (myValue is List) { + for (var v1 in myValue) { + checkNotRegExpValue(v1); + for (var otherValue in otherValues) { var match = EntityHandler.equalsValuesBasic( v1, otherValue, @@ -414,15 +399,7 @@ class ConditionParameter extends ConditionElement { return false; } else { - for (var v2 in values) { - var otherValue = v2 is ConditionParameter - ? v2.getValue( - parameters: parameters, - positionalParameters: positionalParameters, - namedParameters: namedParameters, - ) - : v2; - + for (var otherValue in otherValues) { var match = EntityHandler.equalsValuesBasic( myValue, otherValue, @@ -472,6 +449,54 @@ class ConditionParameter extends ConditionElement { (contextPosition?.hashCode ?? 0); } +/// The four ordering comparisons of a condition (`>`, `>=`, `<`, `<=`). +/// +/// Written once here, instead of once per operator in [ConditionParameter], +/// [Condition] and the `KeyCondition` subclasses: used by +/// [ConditionParameter._compareOrdering], [Condition._orderingConditionValue] +/// and [_KeyConditionOrdering]. +enum _Ordering { + gt('>'), + gte('>='), + lt('<'), + lte('<='); + + /// The operator symbol, as encoded in a condition (see [Condition.encode]). + final String symbol; + + const _Ordering(this.symbol); + + /// The same comparison with its sides swapped: `a > b` is `b < a`. Used + /// when the [ConditionParameter] to resolve is the right side. + _Ordering get mirrored => switch (this) { + gt => lt, + gte => lte, + lt => gt, + lte => gte, + }; + + /// Compares [a] with [b] by the matching [EntityHandler] function. + bool test(Object? a, Object? b, EntityHandler? entityHandler) => + switch (this) { + gt => EntityHandler.greaterThanValue( + a, + b, + entityHandler: entityHandler, + ), + gte => EntityHandler.greaterThanOrEqualValue( + a, + b, + entityHandler: entityHandler, + ), + lt => EntityHandler.lessThanValue(a, b, entityHandler: entityHandler), + lte => EntityHandler.lessThanOrEqualValue( + a, + b, + entityHandler: entityHandler, + ), + }; +} + abstract class EntityMatcher { bool matchesEntity( O o, { @@ -558,6 +583,10 @@ abstract class Condition extends ConditionElement static bool _isToManyValues(Object? value1, Object? value2) => value1 is Iterable && value2 is! Iterable; + /// If [value1] `>` [value2] (see [_orderingConditionValue]). + /// + /// [keyType] is unused: kept for the API (and symmetry with + /// [equalsConditionValue]). bool greaterThanConditionValue( Object? value1, Object? value2, { @@ -566,47 +595,20 @@ abstract class Condition extends ConditionElement Map? namedParameters, TypeInfo? keyType, EntityHandler? keyEntityHandler, - }) { - if (_isToManyValues(value1, value2)) { - return (value1 as Iterable).any( - (v) => greaterThanConditionValue( - v, - value2, - parameters: parameters, - positionalParameters: positionalParameters, - namedParameters: namedParameters, - keyType: keyType, - keyEntityHandler: keyEntityHandler, - ), - ); - } - - if (value1 is ConditionParameter) { - return value1.greaterThan( - value2, - parameters: parameters, - positionalParameters: positionalParameters, - namedParameters: namedParameters, - entityHandler: keyEntityHandler, - ); - } else if (value2 is ConditionParameter) { - return value2.lessThan( - value1, - parameters: parameters, - positionalParameters: positionalParameters, - namedParameters: namedParameters, - entityHandler: keyEntityHandler, - ); - } else { - var match = EntityHandler.graterThanValue( - value1, - value2, - entityHandler: keyEntityHandler, - ); - return match; - } - } + }) => _orderingConditionValue( + _Ordering.gt, + value1, + value2, + parameters, + positionalParameters, + namedParameters, + keyEntityHandler, + ); + /// If [value1] `>=` [value2] (see [_orderingConditionValue]). + /// + /// [keyType] is unused: kept for the API (and symmetry with + /// [equalsConditionValue]). bool greaterThanOrEqualConditionValue( Object? value1, Object? value2, { @@ -615,47 +617,20 @@ abstract class Condition extends ConditionElement Map? namedParameters, TypeInfo? keyType, EntityHandler? keyEntityHandler, - }) { - if (_isToManyValues(value1, value2)) { - return (value1 as Iterable).any( - (v) => greaterThanOrEqualConditionValue( - v, - value2, - parameters: parameters, - positionalParameters: positionalParameters, - namedParameters: namedParameters, - keyType: keyType, - keyEntityHandler: keyEntityHandler, - ), - ); - } - - if (value1 is ConditionParameter) { - return value1.greaterThanOrEqual( - value2, - parameters: parameters, - positionalParameters: positionalParameters, - namedParameters: namedParameters, - entityHandler: keyEntityHandler, - ); - } else if (value2 is ConditionParameter) { - return value2.lessThanOrEqual( - value1, - parameters: parameters, - positionalParameters: positionalParameters, - namedParameters: namedParameters, - entityHandler: keyEntityHandler, - ); - } else { - var match = EntityHandler.graterThanOrEqualValue( - value1, - value2, - entityHandler: keyEntityHandler, - ); - return match; - } - } + }) => _orderingConditionValue( + _Ordering.gte, + value1, + value2, + parameters, + positionalParameters, + namedParameters, + keyEntityHandler, + ); + /// If [value1] `<` [value2] (see [_orderingConditionValue]). + /// + /// [keyType] is unused: kept for the API (and symmetry with + /// [equalsConditionValue]). bool lessThanConditionValue( Object? value1, Object? value2, { @@ -664,47 +639,20 @@ abstract class Condition extends ConditionElement Map? namedParameters, TypeInfo? keyType, EntityHandler? keyEntityHandler, - }) { - if (_isToManyValues(value1, value2)) { - return (value1 as Iterable).any( - (v) => lessThanConditionValue( - v, - value2, - parameters: parameters, - positionalParameters: positionalParameters, - namedParameters: namedParameters, - keyType: keyType, - keyEntityHandler: keyEntityHandler, - ), - ); - } - - if (value1 is ConditionParameter) { - return value1.lessThan( - value2, - parameters: parameters, - positionalParameters: positionalParameters, - namedParameters: namedParameters, - entityHandler: keyEntityHandler, - ); - } else if (value2 is ConditionParameter) { - return value2.greaterThan( - value1, - parameters: parameters, - positionalParameters: positionalParameters, - namedParameters: namedParameters, - entityHandler: keyEntityHandler, - ); - } else { - var match = EntityHandler.lessThanValue( - value1, - value2, - entityHandler: keyEntityHandler, - ); - return match; - } - } + }) => _orderingConditionValue( + _Ordering.lt, + value1, + value2, + parameters, + positionalParameters, + namedParameters, + keyEntityHandler, + ); + /// If [value1] `<=` [value2] (see [_orderingConditionValue]). + /// + /// [keyType] is unused: kept for the API (and symmetry with + /// [equalsConditionValue]). bool lessThanOrEqualConditionValue( Object? value1, Object? value2, { @@ -713,44 +661,69 @@ abstract class Condition extends ConditionElement Map? namedParameters, TypeInfo? keyType, EntityHandler? keyEntityHandler, - }) { + }) => _orderingConditionValue( + _Ordering.lte, + value1, + value2, + parameters, + positionalParameters, + namedParameters, + keyEntityHandler, + ); + + /// Compares [value1] (an entity's value) with [value2] by [ordering]: the + /// shared body of [greaterThanConditionValue], + /// [greaterThanOrEqualConditionValue], [lessThanConditionValue] and + /// [lessThanOrEqualConditionValue]. + /// + /// - The values of a to-many relationship: any of them (as a SQL join of + /// its rows). + /// - A [ConditionParameter] on the left is resolved and compared by + /// [ordering]; on the right, it's the left side of the [_Ordering.mirrored] + /// comparison (`a > ?` is `? < a`). + bool _orderingConditionValue( + _Ordering ordering, + Object? value1, + Object? value2, + Object? parameters, + List? positionalParameters, + Map? namedParameters, + EntityHandler? keyEntityHandler, + ) { if (_isToManyValues(value1, value2)) { return (value1 as Iterable).any( - (v) => lessThanOrEqualConditionValue( + (v) => _orderingConditionValue( + ordering, v, value2, - parameters: parameters, - positionalParameters: positionalParameters, - namedParameters: namedParameters, - keyType: keyType, - keyEntityHandler: keyEntityHandler, + parameters, + positionalParameters, + namedParameters, + keyEntityHandler, ), ); } if (value1 is ConditionParameter) { - return value1.lessThanOrEqual( + return value1._compareOrdering( + ordering, value2, - parameters: parameters, - positionalParameters: positionalParameters, - namedParameters: namedParameters, - entityHandler: keyEntityHandler, + parameters, + positionalParameters, + namedParameters, + keyEntityHandler, ); } else if (value2 is ConditionParameter) { - return value2.greaterThanOrEqual( + return value2._compareOrdering( + ordering.mirrored, value1, - parameters: parameters, - positionalParameters: positionalParameters, - namedParameters: namedParameters, - entityHandler: keyEntityHandler, + parameters, + positionalParameters, + namedParameters, + keyEntityHandler, ); } else { - var match = EntityHandler.lessThanOrEqualValue( - value1, - value2, - entityHandler: keyEntityHandler, - ); - return match; + return ordering.test(value1, value2, keyEntityHandler); } } @@ -1488,6 +1461,9 @@ abstract class KeyCondition extends Condition { } } + /// The [value] encoded as in a condition string (see [encode]). + String get encodeValue => Condition.encodeConditionValue(value); + /// If `true` this is a negated operator (`!=`, `!~`), that also matches a /// `null` field. bool get isNegated => false; @@ -2149,8 +2125,6 @@ class KeyConditionEQ extends KeyCondition { ); } - String get encodeValue => Condition.encodeConditionValue(value); - @override String encode() => '$encodeKey == $encodeValue'; } @@ -2262,21 +2236,20 @@ class KeyConditionNotEQ extends KeyCondition { return notEquals(keyValue); } - String get encodeValue => Condition.encodeConditionValue(value); - @override String encode() => '$encodeKey != $encodeValue'; } -class KeyConditionGreaterThan extends KeyCondition { - KeyConditionGreaterThan(super.keys, dynamic super.value); +/// The base of the ordering key conditions ([KeyConditionGreaterThan], +/// [KeyConditionGreaterThanOrEqual], [KeyConditionLessThan] and +/// [KeyConditionLessThanOrEqual]), which only differ by their [_ordering]: +/// matching and encoding are written once here. Each public subclass keeps +/// its own constructor and [cast]. +abstract class _KeyConditionOrdering extends KeyCondition { + _KeyConditionOrdering(super.keys, super.value); - @override - KeyConditionGreaterThan cast() => - this is KeyConditionGreaterThan - ? this as KeyConditionGreaterThan - : KeyConditionGreaterThan(keys, value) - .._markResolved(resolved, _parent); + /// The comparison of this condition. + _Ordering get _ordering; @override bool matchesEntity( @@ -2288,12 +2261,14 @@ class KeyConditionGreaterThan extends KeyCondition { }) { var keyValue = getEntityKeyValue(o, entityHandler); - return greaterThanConditionValue( + return _orderingConditionValue( + _ordering, keyValue, value, - parameters: parameters, - positionalParameters: positionalParameters, - namedParameters: namedParameters, + parameters, + positionalParameters, + namedParameters, + null, ); } @@ -2307,189 +2282,75 @@ class KeyConditionGreaterThan extends KeyCondition { }) { var keyValue = getEntityMapKeyValue(o, entityHandler); - return greaterThanConditionValue( + return _orderingConditionValue( + _ordering, keyValue?.value, value, - parameters: parameters, - positionalParameters: positionalParameters, - namedParameters: namedParameters, - keyType: keyValue?.type, - keyEntityHandler: keyValue?.entityHandler, + parameters, + positionalParameters, + namedParameters, + keyValue?.entityHandler, ); } - String get encodeValue => Condition.encodeConditionValue(value); - @override - String encode() => '$encodeKey > $encodeValue'; + String encode() => '$encodeKey ${_ordering.symbol} $encodeValue'; } -class KeyConditionGreaterThanOrEqual extends KeyCondition { - KeyConditionGreaterThanOrEqual(super.keys, dynamic super.value); +class KeyConditionGreaterThan extends _KeyConditionOrdering { + KeyConditionGreaterThan(super.keys, dynamic super.value); @override - KeyConditionGreaterThanOrEqual cast() => - this is KeyConditionGreaterThanOrEqual - ? this as KeyConditionGreaterThanOrEqual - : KeyConditionGreaterThanOrEqual(keys, value) - .._markResolved(resolved, _parent); + _Ordering get _ordering => _Ordering.gt; @override - bool matchesEntity( - O o, { - EntityHandler? entityHandler, - Object? parameters, - List? positionalParameters, - Map? namedParameters, - }) { - var keyValue = getEntityKeyValue(o, entityHandler); + KeyConditionGreaterThan cast() => + this is KeyConditionGreaterThan + ? this as KeyConditionGreaterThan + : KeyConditionGreaterThan(keys, value) + .._markResolved(resolved, _parent); +} - return greaterThanOrEqualConditionValue( - keyValue, - value, - parameters: parameters, - positionalParameters: positionalParameters, - namedParameters: namedParameters, - ); - } +class KeyConditionGreaterThanOrEqual extends _KeyConditionOrdering { + KeyConditionGreaterThanOrEqual(super.keys, dynamic super.value); @override - bool matchesEntityMap( - Map o, { - EntityHandler? entityHandler, - Object? parameters, - List? positionalParameters, - Map? namedParameters, - }) { - var keyValue = getEntityMapKeyValue(o, entityHandler); - - return greaterThanOrEqualConditionValue( - keyValue?.value, - value, - parameters: parameters, - positionalParameters: positionalParameters, - namedParameters: namedParameters, - keyType: keyValue?.type, - keyEntityHandler: keyValue?.entityHandler, - ); - } - - String get encodeValue => Condition.encodeConditionValue(value); + _Ordering get _ordering => _Ordering.gte; @override - String encode() => '$encodeKey >= $encodeValue'; + KeyConditionGreaterThanOrEqual cast() => + this is KeyConditionGreaterThanOrEqual + ? this as KeyConditionGreaterThanOrEqual + : KeyConditionGreaterThanOrEqual(keys, value) + .._markResolved(resolved, _parent); } -class KeyConditionLessThan extends KeyCondition { +class KeyConditionLessThan extends _KeyConditionOrdering { KeyConditionLessThan(super.keys, dynamic super.value); + @override + _Ordering get _ordering => _Ordering.lt; + @override KeyConditionLessThan cast() => this is KeyConditionLessThan ? this as KeyConditionLessThan : KeyConditionLessThan(keys, value) .._markResolved(resolved, _parent); - - @override - bool matchesEntity( - O o, { - EntityHandler? entityHandler, - Object? parameters, - List? positionalParameters, - Map? namedParameters, - }) { - var keyValue = getEntityKeyValue(o, entityHandler); - - return lessThanConditionValue( - keyValue, - value, - parameters: parameters, - positionalParameters: positionalParameters, - namedParameters: namedParameters, - ); - } - - @override - bool matchesEntityMap( - Map o, { - EntityHandler? entityHandler, - Object? parameters, - List? positionalParameters, - Map? namedParameters, - }) { - var keyValue = getEntityMapKeyValue(o, entityHandler); - - return lessThanConditionValue( - keyValue?.value, - value, - parameters: parameters, - positionalParameters: positionalParameters, - namedParameters: namedParameters, - keyType: keyValue?.type, - keyEntityHandler: keyValue?.entityHandler, - ); - } - - String get encodeValue => Condition.encodeConditionValue(value); - - @override - String encode() => '$encodeKey < $encodeValue'; } -class KeyConditionLessThanOrEqual extends KeyCondition { +class KeyConditionLessThanOrEqual extends _KeyConditionOrdering { KeyConditionLessThanOrEqual(super.keys, dynamic super.value); + @override + _Ordering get _ordering => _Ordering.lte; + @override KeyConditionLessThanOrEqual cast() => this is KeyConditionLessThanOrEqual ? this as KeyConditionLessThanOrEqual : KeyConditionLessThanOrEqual(keys, value) .._markResolved(resolved, _parent); - - @override - bool matchesEntity( - O o, { - EntityHandler? entityHandler, - Object? parameters, - List? positionalParameters, - Map? namedParameters, - }) { - var keyValue = getEntityKeyValue(o, entityHandler); - - return lessThanOrEqualConditionValue( - keyValue, - value, - parameters: parameters, - positionalParameters: positionalParameters, - namedParameters: namedParameters, - ); - } - - @override - bool matchesEntityMap( - Map o, { - EntityHandler? entityHandler, - Object? parameters, - List? positionalParameters, - Map? namedParameters, - }) { - var keyValue = getEntityMapKeyValue(o, entityHandler); - - return lessThanOrEqualConditionValue( - keyValue?.value, - value, - parameters: parameters, - positionalParameters: positionalParameters, - namedParameters: namedParameters, - keyType: keyValue?.type, - keyEntityHandler: keyValue?.entityHandler, - ); - } - - String get encodeValue => Condition.encodeConditionValue(value); - - @override - String encode() => '$encodeKey <= $encodeValue'; } abstract class KeyConditionINBase extends KeyCondition> { @@ -2721,8 +2582,6 @@ abstract class KeyConditionINBase extends KeyCondition> { return inIDs(keyValue) != not; } - String get encodeValue => Condition.encodeConditionValue(value); - @override String encode() => '$encodeKey ${not ? '!' : '='}~ $encodeValue'; } @@ -2906,8 +2765,12 @@ class KeyConditionRegExp extends KeyCondition { } } - /// The [RegExp] of a literal pattern (without a parameter). - RegExp? _literalRegExp; + /// A cache of the [RegExp] of a literal [pattern] (without a parameter), + /// filled by [_resolveRegExp]. + /// + /// Why: a literal pattern never changes, and a condition is matched once + /// per entity (an in-memory select): compiled once, not once per entity. + RegExp? _literalRegExpCache; /// The [RegExp] of this condition, with its parameter's value, if any. /// @@ -2921,7 +2784,7 @@ class KeyConditionRegExp extends KeyCondition { ) { var pattern = this.pattern; if (pattern != null) { - return _literalRegExp ??= toDartRegExp( + return _literalRegExpCache ??= toDartRegExp( pattern, caseInsensitive: caseInsensitive, ); @@ -3066,6 +2929,8 @@ class KeyConditionRegExp extends KeyCondition { return s.toString(); } + /// A regexp is encoded as `/pattern/flags`, not as a JSON value. + @override String get encodeValue { var value = this.value; var body = value is String ? encodePatternLiteral(value) : value.toString(); diff --git a/lib/src/bones_api_condition_encoder.dart b/lib/src/bones_api_condition_encoder.dart index 89030052..6db422a9 100644 --- a/lib/src/bones_api_condition_encoder.dart +++ b/lib/src/bones_api_condition_encoder.dart @@ -1267,38 +1267,36 @@ class EncodingContext { return null; } - bool isValidAlias(String alias) { - if (isReservedWord(alias)) { - return false; - } - - return true; - } - - bool isReservedWord(String alias) { - return alias == 'as' || - alias == 'eq' || - alias == 'by' || - alias == 'null' || - alias == 'not' || - alias == 'def' || - alias == 'sum' || - alias == 'add' || - alias == 'from' || - alias == 'default' || - alias == 'alias' || - alias == 'equals' || - alias == 'count' || - alias == 'order' || - alias == 'table' || - alias == 'column' || - alias == 'select' || - alias == 'alter' || - alias == 'drop' || - alias == 'delete' || - alias == 'sql' || - alias == 'empty'; - } + bool isValidAlias(String alias) => !isReservedWord(alias); + + /// The words that can't be a generated table alias (see [isReservedWord]): + /// SQL keywords and function names an alias could collide with. + static const Set _reservedAliases = { + 'as', + 'eq', + 'by', + 'null', + 'not', + 'def', + 'sum', + 'add', + 'from', + 'default', + 'alias', + 'equals', + 'count', + 'order', + 'table', + 'column', + 'select', + 'alter', + 'drop', + 'delete', + 'sql', + 'empty', + }; + + bool isReservedWord(String alias) => _reservedAliases.contains(alias); String addEncodingParameter( String suggestedKey, @@ -1326,6 +1324,16 @@ class EncodingContext { throw StateError("Can't create encoding parameter: $suggestedKey = $value"); } + + /// The value of [parameter], resolved with all the parameters of this + /// context ([parameters], [positionalParameters], [namedParameters] and + /// [encodingParameters]). + Object? parameterValue(ConditionParameter parameter) => parameter.getValue( + parameters: parameters, + positionalParameters: positionalParameters, + namedParameters: namedParameters, + encodingParameters: encodingParameters, + ); } /// Base class to encode [Condition]. @@ -1732,12 +1740,7 @@ abstract class ConditionEncoder { Type? valueType, bool valueAsList, ) { - var paramValue = value.getValue( - parameters: context.parameters, - positionalParameters: context.positionalParameters, - namedParameters: context.namedParameters, - encodingParameters: context.encodingParameters, - ); + var paramValue = context.parameterValue(value); // Not a regexp pattern parameter (`/?/`), which `encodeKeyConditionRegExp` // resolves: a `RegExp` here would be compared as its text. @@ -1805,13 +1808,15 @@ abstract class ConditionEncoder { } FutureOr _resolveValueToTypeImpl(Object? value, Type valueType) { + // `late`: computed once, and only if needed (the early returns below + // don't need it). + late final valueTypeInfo = TypeInfo.from(valueType); + // A list as a single value (`== [x]` is `== x`), unless the field is a // list itself (a `List`, a `Uint8List`: compared as a whole): // first, so its element is resolved as a value (an enum by its name, an // entity by its ID), not the list as text. - if (value is Iterable && - value is! TypedData && - !TypeInfo.from(valueType).isIterable) { + if (value is Iterable && value is! TypedData && !valueTypeInfo.isIterable) { var list = value.toList(); if (list.isEmpty) return null; if (list.length == 1) value = list.first; @@ -1872,8 +1877,6 @@ abstract class ConditionEncoder { if (entityID != null) value = entityID; } - var valueTypeInfo = TypeInfo.from(valueType); - var valueParser = TypeParser.parserForTypeInfo(valueTypeInfo); if (valueParser != null) { if (value is Iterable && valueTypeInfo.isPrimitiveType) { diff --git a/lib/src/bones_api_condition_parser.dart b/lib/src/bones_api_condition_parser.dart index 27be55ff..2e33f91b 100644 --- a/lib/src/bones_api_condition_parser.dart +++ b/lib/src/bones_api_condition_parser.dart @@ -96,11 +96,18 @@ abstract class JsonGrammarLexer extends GrammarDefinition { Parser characterEscape() => char('\\') & pattern(jsonEscapeCharsString); + /// The escapes of a single-quoted string: also `\'` + /// ([jsonEscapeCharsString2]). + Parser characterEscape2() => + char('\\') & pattern(jsonEscapeCharsString2); + Parser characterEscapeValue() => characterEscape().map((each) => jsonEscapeChars[each[1]]!); + // With [characterEscape2]: [characterEscape] doesn't accept `\'`, so a + // single-quoted string with an escaped quote (`'a\'b'`) didn't parse. Parser characterEscapeValue2() => - characterEscape().map((each) => jsonEscapeChars2[each[1]]!); + characterEscape2().map((each) => jsonEscapeChars2[each[1]]!); Parser characterUnicode() => string('\\u') & pattern('0-9A-Fa-f').times(4); diff --git a/lib/src/bones_api_condition_sql.dart b/lib/src/bones_api_condition_sql.dart index 1bbc614a..4cc96917 100644 --- a/lib/src/bones_api_condition_sql.dart +++ b/lib/src/bones_api_condition_sql.dart @@ -137,40 +137,42 @@ class ConditionSQLEncoder extends ConditionEncoder { FutureOr encodeIDCondition( ConditionID c, EncodingContext context, - ) { - var tableName = context.tableNameOrEntityName; - var tableAlias = context.resolveEntityAlias(tableName); + ) => _encodeIDComparison(context, c.idValue, isList: false); - // Never `null`: required by the constructor. - var tableSchemeRet = schemeProvider!.getTableScheme(tableName); + @override + FutureOr encodeIDConditionIN( + ConditionIdIN c, + EncodingContext context, + ) => _encodeIDComparison(context, c.idsValues, isList: true); - return tableSchemeRet.resolveMapped((tableScheme) { - if (tableScheme == null) { - var errorMsg = "Can't find `TableScheme` for entity/table: $tableName"; - _log.severe(errorMsg); - throw StateError(errorMsg); - } + /// The shared body of [encodeIDCondition] (`#ID == x`) and + /// [encodeIDConditionIN] (`#ID =~ [x, y]`, [isList]): resolves the ID + /// column of the [context]'s table and writes its comparison with + /// [idValue] (see [encodeIDValues]). + FutureOr _encodeIDComparison( + EncodingContext context, + Object? idValue, { + required bool isList, + }) { + var tableName = context.tableNameOrEntityName; + var tableAlias = context.resolveEntityAlias(tableName); + return _requireTableScheme(tableName).resolveMapped((tableScheme) { var idFieldName = tableScheme.idFieldName ?? 'id'; var idType = tableScheme.fieldsTypes[idFieldName] ?? int; - var idKey = context.addEncodingParameter(idFieldName, c.idValue); - var q = sqlElementQuote; + var idKey = context.addEncodingParameter(idFieldName, idValue); // The column, not `idKey` (renamed if a query parameter has its name): - var tableKey = '$q$tableAlias$q.$q$idFieldName$q'; + var tableKey = qualifiedColumnSQL(tableAlias, idFieldName); - return encodeIDValues(context, idType, idKey, tableKey, c.idValue, false); + return encodeIDValues(context, idType, idKey, tableKey, idValue, isList); }); } - @override - FutureOr encodeIDConditionIN( - ConditionIdIN c, - EncodingContext context, - ) { - var tableName = context.tableNameOrEntityName; - var tableAlias = context.resolveEntityAlias(tableName); - + /// The [TableScheme] of [tableName], or a [StateError] (logged) if there's + /// none. Used by the encoders that need the main table's scheme + /// ([_encodeIDComparison] and [keyFieldToSQL]). + FutureOr _requireTableScheme(String tableName) { // Never `null`: required by the constructor. var tableSchemeRet = schemeProvider!.getTableScheme(tableName); @@ -180,26 +182,18 @@ class ConditionSQLEncoder extends ConditionEncoder { _log.severe(errorMsg); throw StateError(errorMsg); } - - var idFieldName = tableScheme.idFieldName ?? 'id'; - var idType = tableScheme.fieldsTypes[idFieldName] ?? int; - - var idKey = context.addEncodingParameter(idFieldName, c.idsValues); - var q = sqlElementQuote; - // The column, not `idKey` (renamed if a query parameter has its name): - var tableKey = '$q$tableAlias$q.$q$idFieldName$q'; - - return encodeIDValues( - context, - idType, - idKey, - tableKey, - c.idsValues, - true, - ); + return tableScheme; }); } + /// A column qualified by its table alias, quoted with [sqlElementQuote]: + /// `"alias"."column"`. As `SQLDialect.qualifiedColumnSQL` (with the + /// dialect's quote). + String qualifiedColumnSQL(String tableAlias, String column) { + var q = sqlElementQuote; + return '$q$tableAlias$q.$q$column$q'; + } + /// Writes the comparison of the ID column ([tableKey]) with [idValue] (`=`) /// or with the [idValue] list (`IN`), as parameters (placeholders) rather /// than values written into the SQL: the statement text is the same for @@ -385,12 +379,7 @@ class ConditionSQLEncoder extends ConditionEncoder { var patternParameter = c.patternParameter; if (patternParameter != null) { - var paramValue = patternParameter.getValue( - parameters: context.parameters, - positionalParameters: context.positionalParameters, - namedParameters: context.namedParameters, - encodingParameters: context.encodingParameters, - ); + var paramValue = context.parameterValue(patternParameter); (pattern, caseInsensitive) = KeyConditionRegExp.resolvePatternValue( paramValue, @@ -486,14 +475,19 @@ class ConditionSQLEncoder extends ConditionEncoder { var values = c.value; if (values is! List || values.length != 1) return false; var p = values.first; - return p is ConditionParameter && - p.getValue( - parameters: context.parameters, - positionalParameters: context.positionalParameters, - namedParameters: context.namedParameters, - encodingParameters: context.encodingParameters, - ) == - null; + return p is ConditionParameter && context.parameterValue(p) == null; + } + + /// Writes the match of an empty set of values (`x =~ []`): `IN` matches + /// nothing (`FALSE`) and `NOT IN` everything (`TRUE`, also a `NULL`). + /// Used by [encodeConditionValuesWithOperator] for an empty list, and for a + /// relationship's list with only `null`s. + EncodingContext _writeEmptySetMatch( + EncodingContext context, + String operator, + ) { + context.write(operator == 'IN' ? ' FALSE ' : ' TRUE '); + return context; } /// If [c]'s key is a single relationship field of the main entity: its `IN` @@ -601,8 +595,7 @@ class ConditionSQLEncoder extends ConditionEncoder { // as a list with a `null`), except for a relationship // ([nullsAsAbsent], below). if (elements.isEmpty) { - context.write(operator == 'IN' ? ' FALSE ' : ' TRUE '); - return context; + return _writeEmptySetMatch(context, operator); } var nonNulls = elements @@ -620,8 +613,7 @@ class ConditionSQLEncoder extends ConditionEncoder { // A relationship's set of IDs ([_isRelationshipKey]): its `null`s // aren't IDs. Without any other, an empty set (as `[]`). if (nonNulls.isEmpty) { - context.write(operator == 'IN' ? ' FALSE ' : ' TRUE '); - return context; + return _writeEmptySetMatch(context, operator); } value = EncodingValueList( value.key, @@ -801,17 +793,7 @@ class ConditionSQLEncoder extends ConditionEncoder { var keys = c.keys; var key0 = keys.first as ConditionKeyField; - var tableSchemeRet = schemeProvider.getTableScheme(tableName); - - return tableSchemeRet.resolveMapped((tableScheme) { - if (tableScheme == null) { - var errorMsg = "Can't find `TableScheme` for entity/table: $tableName"; - _log.severe(errorMsg); - throw StateError(errorMsg); - } - - var q = sqlElementQuote; - + return _requireTableScheme(tableName).resolveMapped((tableScheme) { var fieldName = key0.name; var tableFieldName = tableScheme.resolveTableFieldName(fieldName); var tableFieldType = tableFieldName != null @@ -819,7 +801,10 @@ class ConditionSQLEncoder extends ConditionEncoder { : null; if (tableFieldType != null) { - return MapEntry(tableFieldType, '$q$tableAlias$q.$q$tableFieldName$q'); + return MapEntry( + tableFieldType, + qualifiedColumnSQL(tableAlias, tableFieldName!), + ); } var retFieldType = schemeProvider.getFieldType( @@ -874,7 +859,7 @@ class ConditionSQLEncoder extends ConditionEncoder { return MapEntry( relationshipTargetFieldType, - '$q$relationshipAlias$q.$q$relationshipTargetField$q', + qualifiedColumnSQL(relationshipAlias, relationshipTargetField), ); }); }); @@ -952,9 +937,8 @@ class ConditionSQLEncoder extends ConditionEncoder { final alias = context.resolveEntityAlias(targetTable); final type = scheme.fieldsTypes[fieldName]!; - final q = sqlElementQuote; - return MapEntry(type, '$q$alias$q.$q$fieldName$q'); + return MapEntry(type, qualifiedColumnSQL(alias, fieldName)); }); } diff --git a/lib/src/bones_api_cron_scheduler.dart b/lib/src/bones_api_cron_scheduler.dart index ccbac8f3..501b4779 100644 --- a/lib/src/bones_api_cron_scheduler.dart +++ b/lib/src/bones_api_cron_scheduler.dart @@ -458,20 +458,90 @@ class APICronScheduler { run.status = APICronRunStatus.running; if (!await store.updateRun(run, previousOwner: previousOwner)) { - _log.warning( - 'Run `${job.name}` (${run.scheduledAt.toIso8601String()}): ' - 'taken by another instance', - ); + _log.warning('${_runLabel(job, run)}: taken by another instance'); return run; } _log.info( - 'Run `${job.name}` (${run.scheduledAt.toIso8601String()}): started, ' + '${_runLabel(job, run)}: started, ' 'owner=$owner${run.calls > 0 ? ', resuming after ${run.calls} call(s)' : ''}', ); renewer.start(); + var (:succeeded, :error, interrupted: interruptedByStore) = await _runCalls( + job, + run, + renewer, + ); + + var interrupted = + interruptedByStore ?? + (renewer.lost ? 'lock lost' : (_stopping ? 'scheduler stopped' : null)); + + if (interrupted != null) { + // Left `running`: resumed by the new owner (or another instance). + _log.warning( + '${_runLabel(job, run)}: ' + '$interrupted, left running after ${run.calls} call(s)', + ); + return run; + } + + var finished = _now; + run.finishedAt = finished; + run.durationMs = finished + .difference(run.startedAt ?? started) + .inMilliseconds; + run.status = succeeded + ? APICronRunStatus.succeeded + : APICronRunStatus.failed; + run.lastError = succeeded ? null : error; + + if (!await _storeRunResult(job, run)) { + // Rejected (taken by another instance) or not stored: as stored. + _log.warning( + '${_runLabel(job, run)}: result ' + '(${run.status.name}) not stored: interrupted', + ); + return run..status = APICronRunStatus.running; + } + + var msg = + '${_runLabel(job, run)}: ' + '${run.status.name}, calls=${run.calls}, attempts=${run.attempts}, ' + 'duration=${run.durationMs}ms'; + + if (succeeded) { + _log.info(msg); + } else { + _log.severe('$msg, error: $error'); + } + + return run; + } + + /// The log label of a [run] of [job]: `` Run `name` (scheduledAt) ``. + /// Used by [_execute] for every message about a run's lifecycle. + static String _runLabel(APICronJob job, APICronJobRun run) => + 'Run `${job.name}` (${run.scheduledAt.toIso8601String()})'; + + /// Calls the route of [job] (with retries) until there's no continuation, + /// storing the progress ([APICronJobRun.calls] and continuation) after + /// each call. Used by [_execute]. + /// + /// Returns whether the run [succeeded], its [error] (when it failed) and, + /// when the progress store was rejected (taken by another instance), why + /// it was [interrupted]. An interruption by the [renewer] (lock lost) or + /// by [stop] only stops the loop: [_execute] checks it afterwards. + /// + /// An error never escapes: it fails the run (a run left `running` by an + /// error would be resumed again and again). + Future<({bool succeeded, String? error, String? interrupted})> _runCalls( + APICronJob job, + APICronJobRun run, + _LockRenewer renewer, + ) async { String? error; var succeeded = false; String? interrupted; @@ -526,58 +596,19 @@ class APICronScheduler { _log.severe('Run `${job.name}`: error', e, s); } - interrupted ??= renewer.lost - ? 'lock lost' - : (_stopping ? 'scheduler stopped' : null); - - if (interrupted != null) { - // Left `running`: resumed by the new owner (or another instance). - _log.warning( - 'Run `${job.name}` (${run.scheduledAt.toIso8601String()}): ' - '$interrupted, left running after ${run.calls} call(s)', - ); - return run; - } - - var finished = _now; - run.finishedAt = finished; - run.durationMs = finished - .difference(run.startedAt ?? started) - .inMilliseconds; - run.status = succeeded - ? APICronRunStatus.succeeded - : APICronRunStatus.failed; - run.lastError = succeeded ? null : error; + return (succeeded: succeeded, error: error, interrupted: interrupted); + } - bool stored; + /// Stores the final result of [run] (already set by [_execute]). Returns + /// `false` if the store rejected it (taken by another instance) or failed + /// (the error is logged, not rethrown). + Future _storeRunResult(APICronJob job, APICronJobRun run) async { try { - stored = await store.updateRun(run); + return await store.updateRun(run); } catch (e, s) { _log.severe('Run `${job.name}`: error storing its result', e, s); - stored = false; - } - - if (!stored) { - // Rejected (taken by another instance) or not stored: as stored. - _log.warning( - 'Run `${job.name}` (${run.scheduledAt.toIso8601String()}): result ' - '(${run.status.name}) not stored: interrupted', - ); - return run..status = APICronRunStatus.running; - } - - var msg = - 'Run `${job.name}` (${run.scheduledAt.toIso8601String()}): ' - '${run.status.name}, calls=${run.calls}, attempts=${run.attempts}, ' - 'duration=${run.durationMs}ms'; - - if (succeeded) { - _log.info(msg); - } else { - _log.severe('$msg, error: $error'); + return false; } - - return run; } /// Calls the route of [job], with the [APICronJob.retry] attempts. diff --git a/lib/src/bones_api_db_module.dart b/lib/src/bones_api_db_module.dart index b72c66cd..dda0f8ec 100644 --- a/lib/src/bones_api_db_module.dart +++ b/lib/src/bones_api_db_module.dart @@ -1,3 +1,4 @@ +import 'dart:convert' show HtmlEscape, HtmlEscapeMode; import 'dart:typed_data'; import 'package:archive/archive.dart'; @@ -345,43 +346,21 @@ class APIDBModule extends APIModule { return APIResponse.error(error: "Can't find table: $table"); } - var query = Uri.decodeQueryComponent(requestedUri.query); - - // See [selectQueryDirectives]. The arguments passed to this method take - // precedence over the query `String` directives. - { - var extracted = _extractQueryDirective(query, 'EAGER'); - query = extracted.query; - if (extracted.value == 'true') eager = true; - } - - { - var extracted = _extractQueryDirective(query, 'LIMIT'); - query = extracted.query; - limit ??= int.tryParse(extracted.value?.trim() ?? ''); - - // A page of `defaultSelectLimit` by default (`LIMIT=0`: no limit). - limit ??= defaultSelectLimit; - if (limit <= 0) limit = null; - } - - { - var extracted = _extractQueryDirective(query, 'OFFSET'); - query = extracted.query; - offset ??= int.tryParse(extracted.value?.trim() ?? ''); - } - - { - var extracted = _extractQueryDirective(query, 'PAGE'); - query = extracted.query; - page ??= int.tryParse(extracted.value?.trim() ?? ''); - } + var directives = _parseSelectDirectives( + Uri.decodeQueryComponent(requestedUri.query), + eager: eager, + limit: limit, + offset: offset, + page: page, + orderDirection: orderDirection, + ); - { - var extracted = _extractQueryDirective(query, 'ORDER'); - query = extracted.query; - orderDirection ??= OrderDirection.parse(extracted.value); - } + var query = directives.query; + eager = directives.eager; + limit = directives.limit; + offset = directives.offset; + page = directives.page; + orderDirection = directives.orderDirection; // Resolved here (and not by the repository) so an invalid `PAGE` becomes an // error response instead of an uncaught `ArgumentError`: @@ -453,7 +432,123 @@ class APIDBModule extends APIModule { footer: _buildFooter(), ); - var content = []; + var repoName = entityRepository.name; + + var content = [ + ..._buildSelectTableHTML( + entityRepository, + entityRepositoryProviders, + list, + ), + ..._buildPaginationHTML( + repoName, + list, + query: query, + eager: eager, + limit: limit, + offset: offset, + orderDirection: orderDirection, + ), + ]; + + htmlDoc.content = content; + + var html = htmlDoc.build(); + + return APIResponse.ok(html, mimeType: 'html'); + } + + /// Parses the directives of a select's [query] `String` (see + /// [selectQueryDirectives]), returning the [query] without them. The + /// arguments given (of [select]) take precedence over the directives. + /// + /// Without an explicit `LIMIT`, a page of [defaultSelectLimit] (`LIMIT=0`: + /// no limit, `null`). Used by [select]. + ({ + String query, + bool? eager, + int? limit, + int? offset, + int? page, + OrderDirection? orderDirection, + }) + _parseSelectDirectives( + String query, { + bool? eager, + int? limit, + int? offset, + int? page, + OrderDirection? orderDirection, + }) { + { + var extracted = _extractQueryDirective(query, 'EAGER'); + query = extracted.query; + if (extracted.value == 'true') eager = true; + } + + { + var extracted = _extractIntDirective(query, 'LIMIT'); + query = extracted.query; + limit ??= extracted.value; + + // A page of `defaultSelectLimit` by default (`LIMIT=0`: no limit). + limit ??= defaultSelectLimit; + if (limit <= 0) limit = null; + } + + { + var extracted = _extractIntDirective(query, 'OFFSET'); + query = extracted.query; + offset ??= extracted.value; + } + + { + var extracted = _extractIntDirective(query, 'PAGE'); + query = extracted.query; + page ??= extracted.value; + } + + { + var extracted = _extractQueryDirective(query, 'ORDER'); + query = extracted.query; + orderDirection ??= OrderDirection.parse(extracted.value); + } + + return ( + query: query, + eager: eager, + limit: limit, + offset: offset, + page: page, + orderDirection: orderDirection, + ); + } + + /// As [_extractQueryDirective], with the value parsed as an `int` (`null` + /// if absent or not an `int`). Used for the `LIMIT`, `OFFSET` and `PAGE` + /// directives by [_parseSelectDirectives]. + static ({String query, int? value}) _extractIntDirective( + String query, + String key, + ) { + var extracted = _extractQueryDirective(query, key); + return ( + query: extracted.query, + value: int.tryParse(extracted.value?.trim() ?? ''), + ); + } + + /// The HTML `` of the selected entities ([list]) of + /// [entityRepository], with links to the related entities, and to edit, + /// delete and insert. Used by [select]. + List _buildSelectTableHTML( + EntityRepository entityRepository, + List entityRepositoryProviders, + List list, + ) { + var content = []; + + var entityType = entityRepository.type; var entityHandler = entityRepository.entityHandler; @@ -497,29 +592,36 @@ class APIDBModule extends APIModule { content.add(''); } else { - content.add(''); + content.add(''); } } var id = entityHandler.getID(o); if (id != null) { + var idAttr = _escapeHTMLAttribute(id); + content.add(''); } @@ -534,10 +636,41 @@ class APIDBModule extends APIModule { content.add('
'); + // A related entity's ID (also a `String` one) is escaped as any + // value, in the link and its text: + String idLink(Object? id) => + '' + '${_escapeHTML(id)}'; + if (v2 is List) { - var ids = v2 - .map((e) => '$e') - .join(' , '); - content.add(ids); + content.add(v2.map(idLink).join(' , ')); } else { - content.add('$v2'); + content.add(idLink(v2)); } content.add('$v2${_escapeHTML(v2)}   [ '); - content.add('edit'); + content.add('edit'); - content.add('  |  del'); + content.add( + '  |  del', + ); content.add(' ]  
'); + return content; + } + + /// [value] (its `toString`) escaped as HTML text (`&`, `<`, `>`): an + /// entity's value (e.g. a name with `<`) would otherwise be read as HTML + /// (and could inject markup or scripts in the page). Used by + /// [_buildSelectTableHTML]. + static String _escapeHTML(Object? value) => + const HtmlEscape(HtmlEscapeMode.element).convert('$value'); + + /// [value] (its `toString`) escaped for an HTML attribute value between + /// `"` (as [_escapeHTML], also `"`): an entity's ID in a link. Used by + /// [_buildSelectTableHTML]. + static String _escapeHTMLAttribute(Object? value) => + const HtmlEscape(HtmlEscapeMode.attribute).convert('$value'); + + /// The HTML of the pagination of a select of [repoName] (when it has a + /// [limit]): the range of the selected entities ([list]) and the links to + /// the previous and next pages; and the link to this page as JSON. The + /// links keep the select's [query] and directives. Used by [select]. + List _buildPaginationHTML( + String repoName, + List list, { + required String query, + required bool eager, + required int? limit, + required int? offset, + required OrderDirection? orderDirection, + }) { + var content = []; + // The query `String` of a page (`OFFSET`) of this select: String pageQuery(int? pageOffset) => [ if (query.isNotEmpty) Uri.encodeQueryComponent(query), - if (eager!) 'EAGER=true', + if (eager) 'EAGER=true', if (orderDirection != null) 'ORDER=${orderDirection.name}', 'LIMIT=${limit ?? 0}', if (pageOffset != null && pageOffset > 0) 'OFFSET=$pageOffset', @@ -568,11 +701,7 @@ class APIDBModule extends APIModule { '
', ); - htmlDoc.content = content; - - var html = htmlDoc.build(); - - return APIResponse.ok(html, mimeType: 'html'); + return content; } static const _fieldsNamesOrder = [ diff --git a/lib/src/bones_api_entity.dart b/lib/src/bones_api_entity.dart index 0e684d8f..3a62d206 100644 --- a/lib/src/bones_api_entity.dart +++ b/lib/src/bones_api_entity.dart @@ -13,8 +13,7 @@ import 'package:statistics/statistics.dart' DynamicInt, DynamicNumber, IterableMapEntryExtension, - NumericTypeExtension, - IterableExtension; + NumericTypeExtension; import 'package:swiss_knife/swiss_knife.dart' show EventStream, DataURLBase64; import 'bones_api_base.dart'; @@ -567,9 +566,11 @@ abstract class EntityHandler with FieldsFromMap, EntityRulesResolver { entities.map((o) => MapEntry(getIDFromMap(o), o)), ); - Type? _idType; + /// Cache of [idType]: the [Type] of the ID field, fixed per entity type + /// and looked up on every ID parse/compare. + Type? _idTypeCache; - Type idType([O? o]) => _idType ??= getFieldType( + Type idType([O? o]) => _idTypeCache ??= getFieldType( o, idFieldName(o), resolveFiledName: false, @@ -581,15 +582,24 @@ abstract class EntityHandler with FieldsFromMap, EntityRulesResolver { Map fieldsTypes([O? o]); - Map? _fieldsWithTypeList; + // The `fieldsWith*` caches below hold filtered views of [fieldsTypes]. + // The fields of an entity type are fixed, so each filter runs once. - Map fieldsWithTypeList([O? o]) => _fieldsWithTypeList ??= - fieldsWithType((_, fieldType) => fieldType.isList, o); + /// Cache of [fieldsWithTypeList]: the fields of a `List` type. + Map? _fieldsWithTypeListCache; - Map? _fieldsWithTypeEntity; + Map fieldsWithTypeList([O? o]) => + _fieldsWithTypeListCache ??= fieldsWithType( + (_, fieldType) => fieldType.isList, + o, + ); + + /// Cache of [fieldsWithTypeEntity]: the fields holding an entity + /// (not an [EntityReference]). + Map? _fieldsWithTypeEntityCache; Map fieldsWithTypeEntity([O? o]) => - _fieldsWithTypeEntity ??= fieldsWithType( + _fieldsWithTypeEntityCache ??= fieldsWithType( (_, fieldType) => !fieldType.isBasicType && !fieldType.isEntityReferenceListType && @@ -598,10 +608,12 @@ abstract class EntityHandler with FieldsFromMap, EntityRulesResolver { o, ); - Map? _fieldsWithTypeEntityOrReference; + /// Cache of [fieldsWithTypeEntityOrReference]: the fields holding an + /// entity or an [EntityReference]. + Map? _fieldsWithTypeEntityOrReferenceCache; Map fieldsWithTypeEntityOrReference([O? o]) => - _fieldsWithTypeEntityOrReference ??= fieldsWithType( + _fieldsWithTypeEntityOrReferenceCache ??= fieldsWithType( (_, fieldType) => (!fieldType.isBasicType && !fieldType.isEntityReferenceListType && @@ -611,56 +623,68 @@ abstract class EntityHandler with FieldsFromMap, EntityRulesResolver { o, ); - Map? _fieldsWithEntityReference; + /// Cache of [fieldsWithEntityReference]: the fields of an + /// [EntityReference] type. + Map? _fieldsWithEntityReferenceCache; Map fieldsWithEntityReference([O? o]) => - _fieldsWithEntityReference ??= Map.unmodifiable( + _fieldsWithEntityReferenceCache ??= Map.unmodifiable( fieldsWithTypeEntityOrReference(o).entries .where((e) => e.value.isEntityReferenceType) .toMapFromEntries(), ); - Map? _fieldsWithTypeListEntity; + /// Cache of [fieldsWithTypeListEntity]: the fields of a list of entities. + Map? _fieldsWithTypeListEntityCache; Map fieldsWithTypeListEntity([O? o]) => - _fieldsWithTypeListEntity ??= fieldsWithType( + _fieldsWithTypeListEntityCache ??= fieldsWithType( (_, fieldType) => fieldType.isValidListEntityType, o, ); - Map? _fieldsWithTypeListEntityReference; + /// Cache of [fieldsWithTypeListEntityOrReference]: the (non-hidden) fields + /// of a list of entities or an [EntityReferenceList]. + Map? _fieldsWithTypeListEntityOrReferenceCache; Map fieldsWithTypeListEntityOrReference([O? o]) => - _fieldsWithTypeListEntityReference ??= fieldsWithTypeAndAnnotations(( - fieldName, - fieldType, - fieldAnnotations, - ) { - if (!fieldType.isValidListEntityOrReferenceType) { - return false; - } + _fieldsWithTypeListEntityOrReferenceCache ??= + fieldsWithTypeAndAnnotations(( + fieldName, + fieldType, + fieldAnnotations, + ) { + if (!fieldType.isValidListEntityOrReferenceType) { + return false; + } - if (fieldAnnotations != null && fieldAnnotations.isNotEmpty) { - var hidden = fieldAnnotations.hasHidden; - return !hidden; - } + if (fieldAnnotations != null && fieldAnnotations.isNotEmpty) { + var hidden = fieldAnnotations.hasHidden; + return !hidden; + } - return true; - }, o); + return true; + }, o); - Map? _fieldsWithEntityReferenceList; + /// Cache of [fieldsWithEntityReferenceList]: the (non-hidden) fields of an + /// [EntityReferenceList] type. + Map? _fieldsWithEntityReferenceListCache; Map fieldsWithEntityReferenceList([O? o]) => - _fieldsWithEntityReferenceList ??= Map.unmodifiable( - fieldsWithTypeListEntityOrReference(o).entries - .where((e) => e.value.isEntityReferenceListType) - .toMapFromEntries(), - ); + _fieldsWithEntityReferenceListCache ??= + Map.unmodifiable( + fieldsWithTypeListEntityOrReference(o).entries + .where((e) => e.value.isEntityReferenceListType) + .toMapFromEntries(), + ); - Map>? _fieldsEntityAnnotations; + /// Cache of [fieldsEntityAnnotations] (also filled by subclasses): + /// field name -> its unmodifiable [EntityAnnotation]s. Built once (fixed + /// fields; a reflection lookup per field). + Map>? _fieldsEntityAnnotationsCache; Map> fieldsEntityAnnotations() => - _fieldsEntityAnnotations ??= + _fieldsEntityAnnotationsCache ??= Map>.unmodifiable( (getAllFieldsEntityAnnotations() ?? {}) .map>( @@ -1031,33 +1055,26 @@ abstract class EntityHandler with FieldsFromMap, EntityRulesResolver { JsonEntityCache entityCache, EntityResolutionRulesResolved resolutionRulesResolved, ) { - if (type.isList && type.hasArguments) { - var elementType = type.arguments.first; - var valEntityHandler = _resolveEntityHandler( - elementType, - entityProvider, - entityHandlerProvider, - entityRepositoryProvider, - ); + final isTypedList = type.isList && type.hasArguments; - return _resolveListValueByType( - valEntityHandler, - value, - elementType, - entityProvider, - entityCache, - resolutionRulesResolved, - ); - } else if (type.isEntityReferenceListType) { - var elementType = type.arguments.first; + if (!isTypedList && !type.isEntityReferenceListType) { + return type.parseEntity(value); + } - var valEntityHandler = _resolveEntityHandler( - elementType, - entityProvider, - entityHandlerProvider, - entityRepositoryProvider, - ); + // A typed `List` or an `EntityReferenceList`: resolve the elements + // through `E`'s handler. + var elementType = type.arguments.first; + + var valEntityHandler = _resolveEntityHandler( + elementType, + entityProvider, + entityHandlerProvider, + entityRepositoryProvider, + ); + if (!isTypedList) { + // An `EntityReferenceList` of a lazy (not eager) type keeps the + // references, without loading the entities: var eagerEntityType = resolutionRulesResolved.isEagerEntityType( elementType.type, ); @@ -1069,18 +1086,16 @@ abstract class EntityHandler with FieldsFromMap, EntityRulesResolver { entityCache: entityCache, ) as T; } - - return _resolveListValueByType( - valEntityHandler, - value, - elementType, - entityProvider, - entityCache, - resolutionRulesResolved, - ); - } else { - return type.parseEntity(value); } + + return _resolveListValueByType( + valEntityHandler, + value, + elementType, + entityProvider, + entityCache, + resolutionRulesResolved, + ); } FutureOr _resolveMapByType( @@ -1267,36 +1282,28 @@ abstract class EntityHandler with FieldsFromMap, EntityRulesResolver { } } - if (valEntityHandler != null) { - var listFutures = TypeParser.parseList( - value, - elementParser: (e) => valEntityHandler.resolveValueByType( - elementType, - e, - entityProvider: entityProvider, - entityCache: entityCache, - resolutionRules: resolutionRulesResolved, - ), - ); + // Without the element type's handler, this handler resolves the + // elements (and the list isn't cast to the element type): + final EntityHandler resolver = valEntityHandler ?? this; + + var listFutures = TypeParser.parseList( + value, + elementParser: (e) => resolver.resolveValueByType( + elementType, + e, + entityProvider: entityProvider, + entityCache: entityCache, + resolutionRules: resolutionRulesResolved, + ), + ); - if (listFutures == null) return null; + if (listFutures == null) return null; + if (valEntityHandler != null) { return listFutures.resolveAll().resolveMapped((l) { return valEntityHandler.castList(l, elementType.type)! as T; }); } else { - var listFutures = TypeParser.parseList( - value, - elementParser: (e) => resolveValueByType( - elementType, - e, - entityProvider: entityProvider, - entityCache: entityCache, - resolutionRules: resolutionRulesResolved, - ), - ); - - if (listFutures == null) return null; return listFutures.resolveAll().resolveMapped((l) => l as T); } } @@ -1575,10 +1582,13 @@ abstract class EntityHandler with FieldsFromMap, EntityRulesResolver { } } - EntityHandler? valEntityHandler = type == entityType ? this : null; - if (valEntityHandler != null) return valEntityHandler; + if (type == entityType) return this; - if (valEntityHandler == null && entityRepositoryProvider != null) { + // Each step below returns as soon as it resolves a handler, so the next + // step is reached only while the handler is still unresolved. + EntityHandler? valEntityHandler; + + if (entityRepositoryProvider != null) { var entityRepository = entityRepositoryProvider.getEntityRepositoryByType( entityType, ); @@ -1586,7 +1596,7 @@ abstract class EntityHandler with FieldsFromMap, EntityRulesResolver { if (valEntityHandler != null) return valEntityHandler; } - if (valEntityHandler == null && entityHandlerProvider != null) { + if (entityHandlerProvider != null) { valEntityHandler = entityHandlerProvider.getEntityHandler( type: entityType, ); @@ -1602,42 +1612,60 @@ abstract class EntityHandler with FieldsFromMap, EntityRulesResolver { return valEntityHandler; } - Map? _fieldsNamesIndexes; + /// Cache of [fieldsNamesIndexes]: field name -> index in [fieldsNames]. + /// The fields of an entity type are fixed, so it's built once. + Map? _fieldsNamesIndexesCache; - Map fieldsNamesIndexes([O? o]) { - if (_fieldsNamesIndexes == null) { - var fieldsNames = this.fieldsNames(o); - _fieldsNamesIndexes = buildFieldsNamesIndexes(fieldsNames); - } - return _fieldsNamesIndexes!; - } + Map fieldsNamesIndexes([O? o]) => + _fieldsNamesIndexesCache ??= buildFieldsNamesIndexes(fieldsNames(o)); - List? _fieldsNamesLC; + /// Cache of [fieldsNamesLC]: the [fieldsNames] in lower case, used for + /// case-insensitive field lookups. Built once (fixed fields). + List? _fieldsNamesLCCache; - List fieldsNamesLC([O? o]) { - if (_fieldsNamesLC == null) { - var fieldsNames = this.fieldsNames(o); - _fieldsNamesLC = buildFieldsNamesLC(fieldsNames); - } - return _fieldsNamesLC!; - } + List fieldsNamesLC([O? o]) => + _fieldsNamesLCCache ??= buildFieldsNamesLC(fieldsNames(o)); - List? _fieldsNamesSimple; + /// Cache of [fieldsNamesSimple]: the [fieldsNames] as simple keys + /// (`fieldToSimpleKey`), used for lenient field lookups. Built once (fixed + /// fields). + List? _fieldsNamesSimpleCache; - List fieldsNamesSimple([O? o]) { - if (_fieldsNamesSimple == null) { - var fieldsNames = this.fieldsNames(o); - _fieldsNamesSimple = buildFieldsNamesSimple(fieldsNames); - } - return _fieldsNamesSimple!; - } + List fieldsNamesSimple([O? o]) => + _fieldsNamesSimpleCache ??= buildFieldsNamesSimple(fieldsNames(o)); TypeInfo? getFieldType(O? o, String key, {bool resolveFiledName = false}); - Map? _fieldsTypes; + /// Returns the [TypeInfo] of field [key] from [_fieldsTypesCache] (`null` + /// if not cached yet or not found). With [resolveFiledName], a [key] not + /// found as is gets resolved against [fieldsNames] (case/format lenient). + /// + /// The first lookup step of [getFieldType] in [GenericEntityHandler] and + /// [ClassReflectionEntityHandler]; each one then falls back to its own + /// source (the entity instance, or the reflection). + TypeInfo? _getCachedFieldType(String key, bool resolveFiledName) { + final fieldsTypes = _fieldsTypesCache; + if (fieldsTypes == null) return null; + + var fieldType = fieldsTypes[key]; + + if (fieldType == null && resolveFiledName) { + var resolvedFieldName = this.resolveFiledName(fieldsNames(), key); + if (resolvedFieldName != null) { + fieldType = fieldsTypes[resolvedFieldName]; + } + } + + return fieldType; + } + + /// Cache of [getFieldsTypes] (also filled by subclasses' [fieldsTypes]): + /// field name -> [TypeInfo]. Resolving each field type can be expensive + /// (reflection), and the fields of an entity type are fixed. + Map? _fieldsTypesCache; Map getFieldsTypes([O? o]) { - return _fieldsTypes ??= Map.unmodifiable( + return _fieldsTypesCache ??= Map.unmodifiable( Map.fromEntries( fieldsNames(o).map( (key) => MapEntry( @@ -1649,10 +1677,12 @@ abstract class EntityHandler with FieldsFromMap, EntityRulesResolver { ); } - Map? _fieldsEnumTypes; + /// Cache of [getFieldsEnumTypes]: the fields whose type is a registered + /// enum. Built once (fixed fields; a reflection lookup per field). + Map? _fieldsEnumTypesCache; Map getFieldsEnumTypes([O? o]) { - var enumFields = _fieldsEnumTypes; + var enumFields = _fieldsEnumTypesCache; if (enumFields != null) return enumFields; final reflectionFactory = ReflectionFactory(); @@ -1664,13 +1694,17 @@ abstract class EntityHandler with FieldsFromMap, EntityRulesResolver { ) .toMapFromEntries(); - return _fieldsEnumTypes = Map.unmodifiable(mapEnumFields); + return _fieldsEnumTypesCache = Map.unmodifiable( + mapEnumFields, + ); } - Map? _fieldsEntityTypes; + /// Cache of [getFieldsEntityTypes]: the (non-enum) fields that hold a + /// single entity (or reference). Built once (fixed fields). + Map? _fieldsEntityTypesCache; Map getFieldsEntityTypes([O? o]) { - var entityFields = _fieldsEntityTypes; + var entityFields = _fieldsEntityTypesCache; if (entityFields != null) return entityFields; var enumFields = getFieldsEnumTypes(o); @@ -1692,15 +1726,17 @@ abstract class EntityHandler with FieldsFromMap, EntityRulesResolver { .nonNulls .toMapFromEntries(); - return _fieldsEntityTypes = Map.unmodifiable( + return _fieldsEntityTypesCache = Map.unmodifiable( mapEntityFields, ); } - Map? _fieldsListEntityTypes; + /// Cache of [getFieldsListEntityTypes]: the (non-enum) fields that hold a + /// list of entities (or references). Built once (fixed fields). + Map? _fieldsListEntityTypesCache; Map getFieldsListEntityTypes([O? o]) { - var listEntityFields = _fieldsListEntityTypes; + var listEntityFields = _fieldsListEntityTypesCache; if (listEntityFields != null) return listEntityFields; var enumFields = getFieldsEnumTypes(o); @@ -1722,7 +1758,7 @@ abstract class EntityHandler with FieldsFromMap, EntityRulesResolver { .nonNulls .toMapFromEntries(); - return _fieldsListEntityTypes = Map.unmodifiable( + return _fieldsListEntityTypesCache = Map.unmodifiable( mapListEntityFields, ); } @@ -2089,31 +2125,11 @@ abstract class EntityHandler with FieldsFromMap, EntityRulesResolver { if (value1 == value2) return true; - equals = equalsValuesDateTime(value1, value2); + equals = _equalsValuesScalar(value1, value2); if (equals != null) return equals; - equals = equalsValuesTime(value1, value2); - if (equals != null) return equals; - - equals = equalsValuesDynamicNumber(value1, value2); - if (equals != null) return equals; - - equals = equalsValuesPrimitive(value1, value2); - if (equals != null) return equals; - - equals = equalsValuesEnum(value1, value2); - if (equals != null) return equals; - - var collection1 = - TypeParser.isCollectionValue(value1) || - value1.isEntityReferenceList || - value1 is Uint8List || - value1 is Int8List; - var collection2 = - TypeParser.isCollectionValue(value2) || - value2.isEntityReferenceList || - value2 is Uint8List || - value2 is Int8List; + var collection1 = _isCollectionLike(value1); + var collection2 = _isCollectionLike(value2); if (collection1 && collection2) { return isEqualsDeep(value1, value2, valueEquality: equalsValues); @@ -2152,6 +2168,27 @@ abstract class EntityHandler with FieldsFromMap, EntityRulesResolver { return false; } + /// Compares [value1] and [value2] as scalar values, trying in order: + /// [DateTime], [Time], [DynamicNumber], primitive and [Enum] equality. + /// Returns the first decided result, or `null` if none applies. + /// + /// The shared chain of [equalsValues] and [equalsValuesBasic] (which run + /// their own checks before and after it). + static bool? _equalsValuesScalar(Object value1, Object value2) => + equalsValuesDateTime(value1, value2) ?? + equalsValuesTime(value1, value2) ?? + equalsValuesDynamicNumber(value1, value2) ?? + equalsValuesPrimitive(value1, value2) ?? + equalsValuesEnum(value1, value2); + + /// Returns `true` if [value] is compared element-wise by [equalsValues]: + /// a collection, an [EntityReferenceList] or a byte list. + static bool _isCollectionLike(Object value) => + TypeParser.isCollectionValue(value) || + value.isEntityReferenceList || + value is Uint8List || + value is Int8List; + static DynamicNumber valueToDynamicNumber( Object? value, { DynamicNumber? def, @@ -2182,48 +2219,59 @@ abstract class EntityHandler with FieldsFromMap, EntityRulesResolver { // A `null` in a comparison (`>`, `>=`, `<`, `<=`) doesn't match (as SQL's // unknown, `NULL`): not an error. - static bool graterThanValue( + static bool greaterThanValue( Object? value1, Object? value2, { EntityHandler? entityHandler, - }) { - if (value1 == null || value2 == null) return false; - var n1 = valueToDynamicNumber(value1); - var n2 = valueToDynamicNumber(value2); - return n1 > n2; - } + }) => _compareAsNumbers(value1, value2, (n1, n2) => n1 > n2); - static bool graterThanOrEqualValue( + static bool greaterThanOrEqualValue( Object? value1, Object? value2, { EntityHandler? entityHandler, - }) { - if (value1 == null || value2 == null) return false; - var n1 = valueToDynamicNumber(value1); - var n2 = valueToDynamicNumber(value2); - return n1 >= n2; - } + }) => _compareAsNumbers(value1, value2, (n1, n2) => n1 >= n2); static bool lessThanValue( Object? value1, Object? value2, { EntityHandler? entityHandler, - }) { - if (value1 == null || value2 == null) return false; - var n1 = valueToDynamicNumber(value1); - var n2 = valueToDynamicNumber(value2); - return n1 < n2; - } + }) => _compareAsNumbers(value1, value2, (n1, n2) => n1 < n2); static bool lessThanOrEqualValue( Object? value1, Object? value2, { EntityHandler? entityHandler, - }) { + }) => _compareAsNumbers(value1, value2, (n1, n2) => n1 <= n2); + + /// Deprecated misspelling of [greaterThanValue]. + @Deprecated('Use `greaterThanValue`') + static bool graterThanValue( + Object? value1, + Object? value2, { + EntityHandler? entityHandler, + }) => greaterThanValue(value1, value2, entityHandler: entityHandler); + + /// Deprecated misspelling of [greaterThanOrEqualValue]. + @Deprecated('Use `greaterThanOrEqualValue`') + static bool graterThanOrEqualValue( + Object? value1, + Object? value2, { + EntityHandler? entityHandler, + }) => greaterThanOrEqualValue(value1, value2, entityHandler: entityHandler); + + /// Applies the comparison [op] to [value1] and [value2] converted to + /// [DynamicNumber] (see [valueToDynamicNumber]). A `null` side is `false` + /// (see above). Shared by [greaterThanValue], [greaterThanOrEqualValue], + /// [lessThanValue] and [lessThanOrEqualValue]. + static bool _compareAsNumbers( + Object? value1, + Object? value2, + bool Function(DynamicNumber n1, DynamicNumber n2) op, + ) { if (value1 == null || value2 == null) return false; var n1 = valueToDynamicNumber(value1); var n2 = valueToDynamicNumber(value2); - return n1 <= n2; + return op(n1, n2); } static bool equalsValuesBasic( @@ -2235,19 +2283,7 @@ abstract class EntityHandler with FieldsFromMap, EntityRulesResolver { if (value2 == null) return false; if (identical(value1, value2) || value1 == value2) return true; - var equals = equalsValuesDateTime(value1, value2); - if (equals != null) return equals; - - equals = equalsValuesTime(value1, value2); - if (equals != null) return equals; - - equals = equalsValuesDynamicNumber(value1, value2); - if (equals != null) return equals; - - equals = equalsValuesPrimitive(value1, value2); - if (equals != null) return equals; - - equals = equalsValuesEnum(value1, value2); + var equals = _equalsValuesScalar(value1, value2); if (equals != null) return equals; equals = equalsValuesCollection( @@ -2274,25 +2310,8 @@ abstract class EntityHandler with FieldsFromMap, EntityRulesResolver { return value2.equalsEntityID(value1); } - var reflectionFactory = ReflectionFactory(); - - EntityHandler? entityHandler1 = value1.isPrimitiveValue - ? null - : reflectionFactory - .getRegisterClassReflection(value1.runtimeType) - ?.entityHandler ?? - EntityHandlerProvider.globalProvider.getEntityHandler( - obj: value1, - ); - - EntityHandler? entityHandler2 = value2.isPrimitiveValue - ? null - : reflectionFactory - .getRegisterClassReflection(value2.runtimeType) - ?.entityHandler ?? - EntityHandlerProvider.globalProvider.getEntityHandler( - obj: value2, - ); + var entityHandler1 = _entityHandlerForValue(value1); + var entityHandler2 = _entityHandlerForValue(value2); if (entityHandler1 != null) { var id1 = entityHandler1.getID(value1); @@ -2318,6 +2337,18 @@ abstract class EntityHandler with FieldsFromMap, EntityRulesResolver { return equalsValuesEntityMap(value1, value2, entityHandler: entityHandler); } + /// The [EntityHandler] of [value]'s type (its registered reflection's + /// handler, or the global provider's one), or `null` for a primitive value + /// (never an entity). Used by [equalsValuesEntity] for each side. + static EntityHandler? _entityHandlerForValue(Object value) { + if (value.isPrimitiveValue) return null; + + return ReflectionFactory() + .getRegisterClassReflection(value.runtimeType) + ?.entityHandler ?? + EntityHandlerProvider.globalProvider.getEntityHandler(obj: value); + } + /// Equals by ID: a `null` ID (no ID, or not resolved) is never equal (the /// values were not `==` already). See [equalsValuesEntity]. static bool _equalsIDs(Object? id1, Object? id2) => id1 != null && id1 == id2; @@ -2679,7 +2710,7 @@ abstract class EntityHandler with FieldsFromMap, EntityRulesResolver { entityCache: entityCache, resolutionRules: resolutionRules, ).resolveMapped((o) { - entityCache.cacheEntity(o, getID); + entityCache.cacheEntity(o, _getIDFunction); return o; }); }); @@ -2867,63 +2898,64 @@ class GenericEntityHandler extends EntityHandler { } } - String? _idFieldsName; - - @override - String idFieldName([O? o]) { - var idFieldsName = _idFieldsName; - - if (idFieldsName == null) { - inspectObject(o); - idFieldsName = _idFieldsName; - - if (idFieldsName == null) { - throw StateError( - "`idFieldsName` Not populated yet for type `$type`! " - "No Entity instance presented to this EntityHandler yet.", - ); - } - } + // A [GenericEntityHandler] can't reflect on [O]: the caches below are + // filled by [inspectObject] from the first entity instance seen (or one + // created by [createDefault]). - return idFieldsName; - } + /// Cache of [idFieldName] (the ID field's name), filled by + /// [inspectObject]. + String? _idFieldNameCache; - List? _fieldsNames; + /// Cache of [fieldsNames], filled by [inspectObject]. + List? _fieldsNamesCache; @override - List fieldsNames([O? o]) { - var fieldsNames = _fieldsNames; - - if (fieldsNames == null) { - inspectObject(o); - fieldsNames = _fieldsNames; - } + String idFieldName([O? o]) => _inspectedOrThrow( + o, + () => _idFieldNameCache, + () => + "`idFieldsName` Not populated yet for type `$type`! " + "No Entity instance presented to this EntityHandler yet.", + ); - if (fieldsNames == null) { - throw StateError( + @override + List fieldsNames([O? o]) => _inspectedOrThrow( + o, + () => _fieldsNamesCache, + () => "`fieldsNames` Not populated yet! No Entity instance presented to this EntityHandler yet. Type: $type", - ); - } - - return fieldsNames; - } + ); @override - Map fieldsTypes([O? o]) { - var fieldsTypes = _fieldsTypes; + Map fieldsTypes([O? o]) => _inspectedOrThrow( + o, + () => _fieldsTypesCache, + () => + "`fieldsTypes` Not populated yet! No Entity instance presented to this EntityHandler yet. Type: $type", + ); - if (fieldsTypes == null) { - inspectObject(o); - fieldsTypes = _fieldsTypes; - } + /// Returns the cached value read by [readCache], calling [inspectObject] + /// with [o] first when it's not cached yet. Throws a [StateError] with + /// [errorMessage] if it's still not available (no entity instance seen). + /// + /// Used by [idFieldName], [fieldsNames] and [fieldsTypes], whose values + /// only exist after an entity was inspected (see [inspectObject]). + T _inspectedOrThrow( + O? o, + T? Function() readCache, + String Function() errorMessage, + ) { + var value = readCache(); + if (value != null) return value; - if (fieldsTypes == null) { - throw StateError( - "`fieldsTypes` Not populated yet! No Entity instance presented to this EntityHandler yet. Type: $type", - ); + inspectObject(o); + + value = readCache(); + if (value == null) { + throw StateError(errorMessage()); } - return fieldsTypes; + return value; } @override @@ -2936,20 +2968,20 @@ class GenericEntityHandler extends EntityHandler { } } - if (o != null && _idFieldsName == null) { - _idFieldsName = o.idFieldName; + if (o != null && _idFieldNameCache == null) { + _idFieldNameCache = o.idFieldName; - final fieldsNames = _fieldsNames ??= List.unmodifiable( + final fieldsNames = _fieldsNamesCache ??= List.unmodifiable( o.fieldsNames, ); - _fieldsTypes ??= Map.unmodifiable( + _fieldsTypesCache ??= Map.unmodifiable( Map.fromEntries( fieldsNames.map((f) => MapEntry(f, o!.getFieldType(f)!)), ), ); - _fieldsEntityAnnotations ??= + _fieldsEntityAnnotationsCache ??= Map>.unmodifiable( Map>.fromEntries( fieldsNames.map((f) { @@ -3015,19 +3047,7 @@ class GenericEntityHandler extends EntityHandler { TypeInfo? getFieldType(O? o, String key, {bool resolveFiledName = false}) { inspectObject(o); - TypeInfo? fieldType; - - final fieldsTypes = _fieldsTypes; - if (fieldsTypes != null) { - fieldType = fieldsTypes[key]; - - if (fieldType == null && resolveFiledName) { - var resolvedFieldName = this.resolveFiledName(fieldsNames(), key); - if (resolvedFieldName != null) { - fieldType = fieldsTypes[resolvedFieldName]; - } - } - } + var fieldType = _getCachedFieldType(key, resolveFiledName); if (fieldType == null && o != null) { fieldType = o.getFieldType(key); @@ -3035,7 +3055,7 @@ class GenericEntityHandler extends EntityHandler { if (fieldType == null && resolveFiledName) { var resolvedFieldName = this.resolveFiledName(fieldsNames(), key); if (resolvedFieldName != null) { - fieldType = o.getFieldType(key); + fieldType = o.getFieldType(resolvedFieldName); } } } @@ -3069,7 +3089,7 @@ class GenericEntityHandler extends EntityHandler { if (o != null) { return o.getFieldEntityAnnotations(key); } else { - return _fieldsEntityAnnotations?[key]; + return _fieldsEntityAnnotationsCache?[key]; } } @@ -3119,10 +3139,12 @@ class ClassReflectionEntityHandler extends EntityHandler { ClassReflection reflectionWithObject([O? o]) => o == null ? reflection : reflection.withObject(o); - List? _siblingsEntityHandlers; + /// Cache of [siblingsEntityHandlers]: the handlers of the sibling classes + /// (same reflection bundle), fixed after the reflection is registered. + List? _siblingsEntityHandlersCache; List siblingsEntityHandlers() => - _siblingsEntityHandlers ??= + _siblingsEntityHandlersCache ??= List.unmodifiable( reflection.siblingsClassReflection().map((c) => c.entityHandler), ); @@ -3172,15 +3194,16 @@ class ClassReflectionEntityHandler extends EntityHandler { @override V? getField(O o, String key) => reflection.getField(key, o); - /// The ID field (see [idFieldName]), without an object. - FieldReflection? _idField; + /// Cache of the ID field reflection (see [idFieldName]), without an + /// object: avoids a field lookup by name on every [getID]. + FieldReflection? _idFieldCache; /// The same value as `getField(o, idFieldName(o))`, read through the ID - /// field kept in [_idField]: an entity is cached by its ID each time one is - /// decoded or loaded. + /// field kept in [_idFieldCache]: an entity is cached by its ID each time + /// one is decoded or loaded. @override V? getID(O o) { - var idField = _idField ??= reflection.getStaticInstance().field( + var idField = _idFieldCache ??= reflection.getStaticInstance().field( idFieldName(o), ); if (idField == null) return super.getID(o); @@ -3193,19 +3216,7 @@ class ClassReflectionEntityHandler extends EntityHandler { @override TypeInfo? getFieldType(O? o, String key, {bool resolveFiledName = false}) { - TypeInfo? fieldType; - - final fieldsTypes = _fieldsTypes; - if (fieldsTypes != null) { - fieldType = fieldsTypes[key]; - - if (fieldType == null && resolveFiledName) { - var resolvedFieldName = this.resolveFiledName(fieldsNames(), key); - if (resolvedFieldName != null) { - fieldType = fieldsTypes[resolvedFieldName]; - } - } - } + var fieldType = _getCachedFieldType(key, resolveFiledName); if (fieldType == null) { var field = reflection.field(key, o); @@ -3236,12 +3247,13 @@ class ClassReflectionEntityHandler extends EntityHandler { List? getFieldEntityAnnotations(O? o, String key) => getFieldAnnotations(o, key)?.whereType().toList(); - List? _entityIndexes; + /// Cache of [getEntityIndexes]: the class annotations are fixed. + List? _entityIndexesCache; /// The [EntityIndex] annotations of the entity's class. @override List getEntityIndexes() => - _entityIndexes ??= List.unmodifiable( + _entityIndexesCache ??= List.unmodifiable( reflection.classAnnotations.whereType(), ); @@ -3321,20 +3333,21 @@ class ClassReflectionEntityHandler extends EntityHandler { return reflection.fromJson(json); } - String? _idFieldsName; + /// Cache of [idFieldName]: found once by [findIdFieldName] (a scan of the + /// class fields), defaulting to `id`. + String? _idFieldNameCache; @override String idFieldName([O? o]) { - if (_idFieldsName == null) { - // Just to populate: - if (_fieldsNames == null) { - fieldsNames(o); - } + var idFieldName = _idFieldNameCache; + if (idFieldName != null) return idFieldName; - var fieldName = findIdFieldName(o); - _idFieldsName = fieldName ?? 'id'; + // Just to populate: + if (_fieldsNamesCache == null) { + fieldsNames(o); } - return _idFieldsName!; + + return _idFieldNameCache = findIdFieldName(o) ?? 'id'; } String? findIdFieldName([O? o]) { @@ -3389,15 +3402,17 @@ class ClassReflectionEntityHandler extends EntityHandler { return null; } - List? _fieldsNames; + /// Cache of [fieldsNames]: the keys of [fieldsTypes] (fixed per class). + List? _fieldsNamesCache; @override List fieldsNames([O? o]) => - _fieldsNames ??= List.unmodifiable(fieldsTypes(o).keys); + _fieldsNamesCache ??= List.unmodifiable(fieldsTypes(o).keys); @override Map fieldsTypes([O? o]) { - return _fieldsTypes ??= Map.unmodifiable( + // Shares the base `_fieldsTypesCache` (also used by `getFieldsTypes`). + return _fieldsTypesCache ??= Map.unmodifiable( Map.fromEntries( reflectionWithObject(o).fieldsNames.map((f) { var field = reflection.field(f, o); @@ -3409,19 +3424,25 @@ class ClassReflectionEntityHandler extends EntityHandler { ); } - Map>? _constructors; + /// Cache of [constructors]: constructor name -> its parameters' types. + /// Fixed per class, and a full reflection scan to build. + Map>? _constructorsCache; @override Map>? constructors([O? o]) { - return _constructors ??= Map>.unmodifiable( - reflectionWithObject(o).allConstructors().map((c) { - var name = c.name; - var args = c.allParameters - .map((p) => MapEntry(p.jsonName, p.type.typeInfo)) - .toMapFromEntries(); - return MapEntry(name, Map.unmodifiable(args)); - }).toMapFromEntries(), - ); + return _constructorsCache ??= + Map>.unmodifiable( + reflectionWithObject(o).allConstructors().map((c) { + var name = c.name; + var args = c.allParameters + .map( + (p) => + MapEntry(p.jsonName, p.type.typeInfo), + ) + .toMapFromEntries(); + return MapEntry(name, Map.unmodifiable(args)); + }).toMapFromEntries(), + ); } @override @@ -4209,7 +4230,10 @@ abstract class EntityStorage extends EntityAccessor { repositoryProvider: EntityRepositoryProvider.globalProvider, ); - static EntityRepository? _resolveRepositoryProvider( + /// Resolves the [EntityRepository] of [O] (or [obj]/[type]) asking, in + /// order: [entityRepository]'s provider, [repositoryProvider] and + /// [entityHandler]. Used by the cascade delete. + static EntityRepository? _resolveEntityRepository( EntityHandler? entityHandler, EntityRepository? entityRepository, EntityRepositoryProvider? repositoryProvider, { @@ -4227,7 +4251,7 @@ abstract class EntityStorage extends EntityAccessor { EntityRepository? entityRepository, EntityRepositoryProvider? repositoryProvider, }) async { - entityRepository ??= _resolveRepositoryProvider( + entityRepository ??= _resolveEntityRepository( entityHandler, entityRepository, repositoryProvider, @@ -4277,7 +4301,7 @@ abstract class EntityStorage extends EntityAccessor { entityHandler?.getEntityHandler(obj: obj, type: type); if (eh != null && eh.type == obj.genericType) return eh; - var er = _resolveRepositoryProvider( + var er = _resolveEntityRepository( entityHandler, entityRepository, repositoryProvider, @@ -4364,37 +4388,16 @@ abstract class EntityStorage extends EntityAccessor { if (entities == null || entities.isEmpty) continue; - EntityRepository? tEntityRepository; - EntityHandler? tEntityHandler; - - for (var e in entities) { - tEntityRepository ??= _resolveRepositoryProvider( - entityHandler, - entityRepository, - repositoryProvider, - obj: e, - type: fieldType, - ); - - tEntityHandler ??= _resolveEntityHandler( - entityHandler, - tEntityRepository, - repositoryProvider, - obj: e, - type: fieldType, - ); - - preDeleteCalls.add( - () => _deleteCascadeGenericImpl( - e, - transaction, - tEntityHandler, - tEntityRepository, - repositoryProvider, - deleted, - ), - ); - } + _addSubEntitiesDeleteCalls( + entities, + fieldType, + entityHandler, + entityRepository, + repositoryProvider, + transaction, + deleted, + preDeleteCalls, + ); fieldValue.set(null); @@ -4415,37 +4418,17 @@ abstract class EntityStorage extends EntityAccessor { continue; } - EntityRepository? tEntityRepository; - EntityHandler? tEntityHandler; - - for (var e in fieldValues.nonNulls) { - tEntityRepository ??= _resolveRepositoryProvider( - entityHandler, - entityRepository, - repositoryProvider, - obj: e, - type: tType, - ); - - tEntityHandler ??= _resolveEntityHandler( - entityHandler, - tEntityRepository, - repositoryProvider, - obj: e, - type: tType, - ); - - preDeleteCalls.add( - () => _deleteCascadeGenericImpl( - e, - transaction, - tEntityHandler, - tEntityRepository, - repositoryProvider, - deleted, - ), - ); - } + _addSubEntitiesDeleteCalls( + // `whereType()`: the non-null values, typed. + fieldValues.whereType(), + tType, + entityHandler, + entityRepository, + repositoryProvider, + transaction, + deleted, + preDeleteCalls, + ); var fieldValuesEmpty = fieldValues.toList()..clear(); entityHandler.setField( @@ -4506,6 +4489,57 @@ abstract class EntityStorage extends EntityAccessor { return delOk; } + /// Adds to [preDeleteCalls] a cascade delete of each one of [entities] + /// (the elements of a list field of [entityType]), used by + /// [_deleteCascadeGenericImpl] for an [EntityReferenceList] field and for + /// a `List` of entities field. + /// + /// The sub-entities' repository and handler are resolved once (with the + /// first entity) and shared by all the calls: the locals below are + /// captured by the closures, in one scope. + static void _addSubEntitiesDeleteCalls( + Iterable entities, + Type entityType, + EntityHandler entityHandler, + EntityRepository entityRepository, + EntityRepositoryProvider? repositoryProvider, + Transaction transaction, + List deleted, + List Function()> preDeleteCalls, + ) { + EntityRepository? tEntityRepository; + EntityHandler? tEntityHandler; + + for (var e in entities) { + tEntityRepository ??= _resolveEntityRepository( + entityHandler, + entityRepository, + repositoryProvider, + obj: e, + type: entityType, + ); + + tEntityHandler ??= _resolveEntityHandler( + entityHandler, + tEntityRepository, + repositoryProvider, + obj: e, + type: entityType, + ); + + preDeleteCalls.add( + () => _deleteCascadeGenericImpl( + e, + transaction, + tEntityHandler, + tEntityRepository, + repositoryProvider, + deleted, + ), + ); + } + } + static bool _deleteSubEntityImpl( EntityHandler entityHandler, EntityRepository entityRepository, @@ -4519,7 +4553,7 @@ abstract class EntityStorage extends EntityAccessor { List Function()> preDeleteCalls, List Function()> posDeleteCalls, ) { - var tEntityRepository = _resolveRepositoryProvider( + var tEntityRepository = _resolveEntityRepository( entityHandler, entityRepository, repositoryProvider, @@ -4699,47 +4733,50 @@ class EntityRepositoryProvider Type? type, String? name, }) { - if (!isClosed) { - var entityRepository = _entityRepositories[O]; - if (entityRepository != null && !entityRepository.isClosed) { - return entityRepository as EntityRepository; - } + if (isClosed) return null; - Type? objType; + var entityRepository = _openRepositoryOrNull(_entityRepositories[O]); + if (entityRepository != null) return entityRepository; - if (obj != null) { - objType = obj.runtimeType; - if (objType != O) { - entityRepository = _entityRepositories[objType]; - if (entityRepository != null && !entityRepository.isClosed) { - return entityRepository as EntityRepository; - } - } - } + Type? objType; - if (type != null && type != O && type != objType) { - entityRepository = _entityRepositories[type]; - if (entityRepository != null && !entityRepository.isClosed) { - return entityRepository as EntityRepository; - } + if (obj != null) { + objType = obj.runtimeType; + if (objType != O) { + entityRepository = _openRepositoryOrNull( + _entityRepositories[objType], + ); + if (entityRepository != null) return entityRepository; } + } - if (name != null) { - var nameSimplified = EntityAccessor.simplifiedName(name); + if (type != null && type != O && type != objType) { + entityRepository = _openRepositoryOrNull(_entityRepositories[type]); + if (entityRepository != null) return entityRepository; + } - entityRepository = _entityRepositories.values - .where((e) => e.name == name || e.nameSimplified == nameSimplified) - .firstOrNull; + if (name != null) { + var nameSimplified = EntityAccessor.simplifiedName(name); - if (entityRepository != null && !entityRepository.isClosed) { - return entityRepository as EntityRepository; - } - } + return _openRepositoryOrNull( + _entityRepositories.values + .where((e) => e.name == name || e.nameSimplified == nameSimplified) + .firstOrNull, + ); } return null; } + /// Returns [entityRepository] as an [EntityRepository] of [O], or `null` + /// if it's `null` or closed. Used by the local lookups + /// ([_getLocalEntityRepository], [_getLocalEntityRepositoryByType]). + static EntityRepository? _openRepositoryOrNull( + EntityRepository? entityRepository, + ) => entityRepository != null && !entityRepository.isClosed + ? entityRepository as EntityRepository + : null; + EntityRepository? getEntityRepositoryByTypeInfo( TypeInfo typeInfo, ) { @@ -4773,12 +4810,7 @@ class EntityRepositoryProvider ) { if (isClosed) return null; - var entityRepository = _entityRepositories[type]; - if (entityRepository != null && !entityRepository.isClosed) { - return entityRepository as EntityRepository; - } - - return null; + return _openRepositoryOrNull(_entityRepositories[type]); } final Set _knownEntityRepositoryProviders = @@ -4825,11 +4857,8 @@ class EntityRepositoryProvider allRepositories ??= {}; traversedProviders ??= {}; - if (traversedProviders.contains(this)) { - return allRepositories; - } - - traversedProviders.add(this); + // Already traversed (a cycle among known providers): nothing to add. + if (!traversedProviders.add(this)) return allRepositories; for (var e in _entityRepositories.entries) { allRepositories.putIfAbsent(e.key, () => e.value); @@ -4852,35 +4881,27 @@ class EntityRepositoryProvider } /// The lookups across providers: each one asked (resolving several -/// candidates). [traversed] (internal): the providers already asked in the -/// current lookup (see [EntityRepositoryProvider.lookupEntityRepository]). -extension IterableEntityRepositoryProviderExtension - on Iterable { - EntityRepository? getEntityRepository({ - T? obj, - Type? type, - String? name, - EntityRepositoryProvider? entityRepositoryProvider, - EntityHandlerProvider? entityHandlerProvider, - bool removeClosedProviders = false, - Set? traversed, - }) { - var self = this; - if (self is! List && - self is! Set) { - self = self.toList(growable: false); - } - - return self._getEntityRepositoryImpl( - obj: obj, - type: type, - name: name, - entityRepositoryProvider: entityRepositoryProvider, - entityHandlerProvider: entityHandlerProvider, - removeClosedProviders: removeClosedProviders, - traversed: traversed, - ); - } +/// candidates). [traversed] (internal): the providers already asked in the +/// current lookup (see [EntityRepositoryProvider.lookupEntityRepository]). +extension IterableEntityRepositoryProviderExtension + on Iterable { + EntityRepository? getEntityRepository({ + T? obj, + Type? type, + String? name, + EntityRepositoryProvider? entityRepositoryProvider, + EntityHandlerProvider? entityHandlerProvider, + bool removeClosedProviders = false, + Set? traversed, + }) => _asReiterable()._getEntityRepositoryImpl( + obj: obj, + type: type, + name: name, + entityRepositoryProvider: entityRepositoryProvider, + entityHandlerProvider: entityHandlerProvider, + removeClosedProviders: removeClosedProviders, + traversed: traversed, + ); EntityRepository? getEntityRepositoryByTypeInfo( TypeInfo typeInfo, { @@ -4888,18 +4909,13 @@ extension IterableEntityRepositoryProviderExtension EntityHandlerProvider? entityHandlerProvider, bool removeClosedProviders = false, Set? traversed, - }) { - var entityType = typeInfo.entityType; - if (entityType == null) return null; - - return getEntityRepositoryByType( - entityType, - entityRepositoryProvider: entityRepositoryProvider, - entityHandlerProvider: entityHandlerProvider, - removeClosedProviders: removeClosedProviders, - traversed: traversed, - ); - } + }) => _asReiterable()._getEntityRepositoryByTypeInfoImpl( + typeInfo, + entityRepositoryProvider: entityRepositoryProvider, + entityHandlerProvider: entityHandlerProvider, + removeClosedProviders: removeClosedProviders, + traversed: traversed, + ); EntityRepository? getEntityRepositoryByType( Type type, { @@ -4907,36 +4923,36 @@ extension IterableEntityRepositoryProviderExtension EntityHandlerProvider? entityHandlerProvider, bool removeClosedProviders = false, Set? traversed, - }) { - var self = this; - if (self is! List && - self is! Set) { - self = self.toList(growable: false); - } - - return self._getEntityRepositoryByTypeImpl( - type, - entityRepositoryProvider: entityRepositoryProvider, - entityHandlerProvider: entityHandlerProvider, - removeClosedProviders: removeClosedProviders, - traversed: traversed, - ); - } + }) => _asReiterable()._getEntityRepositoryByTypeImpl( + type, + entityRepositoryProvider: entityRepositoryProvider, + entityHandlerProvider: entityHandlerProvider, + removeClosedProviders: removeClosedProviders, + traversed: traversed, + ); Map allRepositories({ Map? allRepositories, Set? traversedProviders, - }) { + }) => _asReiterable()._allRepositoriesImpl( + allRepositories: allRepositories, + traversedProviders: traversedProviders, + ); + + /// Returns this [Iterable] if it's a [List] or a [Set], otherwise a + /// (fixed-length) [List] copy of it. + /// + /// The lookups (`_lookupEntityRepository`, `_allRepositoriesImpl`) read + /// [length] and [first] and then iterate again: a lazy [Iterable] (e.g. a + /// `where`/`map` view) would be re-evaluated on each pass. A [List]/[Set] + /// is also kept as is so `removeClosedProviders` can remove from it. + Iterable _asReiterable() { var self = this; - if (self is! List && - self is! Set) { - self = self.toList(growable: false); + if (self is List || + self is Set) { + return self; } - - return self._allRepositoriesImpl( - allRepositories: allRepositories, - traversedProviders: traversedProviders, - ); + return self.toList(growable: false); } } @@ -5071,7 +5087,7 @@ extension _IterableEntityRepositoryProviderExtension return ask(first); } - var notClosed = _whereNotClosed(this, length, removeClosedProviders); + var notClosed = _whereNotClosed(length, removeClosedProviders); var entityRepositories = notClosed .map(ask) @@ -5107,12 +5123,14 @@ extension _IterableEntityRepositoryProviderExtension removeClosedProviders: removeClosedProviders, ); + /// The providers not closed. With [removeClosedProviders], the closed ones + /// are also removed from this [List]/[Set] (when any, comparing with the + /// original [length]). Used by [_lookupEntityRepository]. Iterable _whereNotClosed( - Iterable self, int length, bool removeClosedProviders, ) { - var notClosed = self.where((e) => !e.isClosed); + var notClosed = where((e) => !e.isClosed); if (removeClosedProviders) { notClosed = notClosed.toList(growable: false); @@ -5252,17 +5270,7 @@ extension EntityRepositoryProviderExtension on EntityRepositoryProvider { if (json == null) return >{}; var map = (json as Map).map((key, value) { - return MapEntry( - key.toString(), - (value as Iterable) - .map( - (e) => (e as Map).map( - (key, value) => - MapEntry(key.toString(), value as dynamic), - ), - ) - .toList(), - ); + return MapEntry(key.toString(), _toEntitiesJson(value as Iterable)); }); return storeAllFromJson( @@ -5281,6 +5289,12 @@ extension EntityRepositoryProviderExtension on EntityRepositoryProvider { }) async { var results = >{}; + // The `%name%` placeholders in the entries (see + // `resolveEntitySourceValueVariables`): + if (variables != null && variables.isNotEmpty) { + entries = resolveEntitiesSourceVariables(entries, variables); + } + var allRepositoriesBuildOrder = allRepositories().values.toList( growable: false, ); @@ -5344,6 +5358,7 @@ extension EntityRepositoryProviderExtension on EntityRepositoryProvider { try { var os = await entityRepository.storeAllFromJson( typeEntries, + transaction: transaction, resolutionRules: resolutionRules, ); results[typeName] = os; @@ -5371,16 +5386,15 @@ extension EntityRepositoryProviderExtension on EntityRepositoryProvider { EntityResolutionRules? resolutionRules, Object? variables, }) { - var sourceMap = resolveEntitiesSource(source); - var variablesMap = resolveSource(variables); + var sourceMap = resolveEntitiesSource(source, workingPath: workingPath); + var variablesMap = resolveSource(variables, workingPath: workingPath); return sourceMap.resolveOther(variablesMap, (sourceMap, variablesMap) { - sourceMap = resolveEntitiesSourceVariables(sourceMap, variablesMap); - _log.info( 'Populating adapter ($this) [entries: ${sourceMap.length}; variables: ${variablesMap.length}]...$_logSectionOpen', ); + // `storeAllFromJson` resolves the `variables` in the entries: return storeAllFromJson( sourceMap, resolutionRules: resolutionRules, @@ -5438,35 +5452,35 @@ extension EntityRepositoryProviderExtension on EntityRepositoryProvider { } if (value is String) { + // A `%name%` placeholder: the variable value (the placeholder itself + // if the variable isn't defined). if (value.startsWith('%') && value.endsWith('%')) { var varName = value.substring(1, value.length - 1); - var varValue = variables[varName] ?? value; - return varValue; - } else { - return value; + return variables[varName] ?? value; } - } else if (value is num || value is bool) { - return value; + } else if (value is Map) { + // Kept apart from a generic `Map` so the result stays typed as a + // `Map`: + return value.map( + (k, v) => MapEntry( + k, + resolveEntitySourceValueVariables(k, v, variables), + ), + ); } else if (value is Map) { - if (value is Map) { - return value.map((k, v) { - var val = resolveEntitySourceValueVariables(k, v, variables); - return MapEntry(k, val); - }); - } else { - return value.map((k, v) { - var val = resolveEntitySourceValueVariables(k, v, variables); - return MapEntry(k, val); - }); - } + return value.map( + (k, v) => + MapEntry(k, resolveEntitySourceValueVariables(k, v, variables)), + ); } else if (value is List) { - return value.map((e) { - var val = resolveEntitySourceValueVariables(key, e, variables); - return val; - }).toList(); - } else { - return value; + return value + .map((e) => resolveEntitySourceValueVariables(key, e, variables)) + .toList(); } + + // Any other value (a plain `String`, a `num`, a `bool`...) has no + // variable to resolve: + return value; } FutureOr>>> resolveEntitiesSource( @@ -5476,20 +5490,23 @@ extension EntityRepositoryProviderExtension on EntityRepositoryProvider { return json.map((k, v) { var key = k.toString(); var values = v is Iterable ? v : [v]; - - var entities = values - .map( - (e) => (e as Map).map( - (key, value) => - MapEntry(key.toString(), value as dynamic), - ), - ) - .toList(); - - return MapEntry(key, entities); + return MapEntry(key, _toEntitiesJson(values)); }); }); + /// Converts each element of [values] (a decoded JSON object, a [Map]) to + /// a `Map` (keys as [String]s). Used to coerce the + /// decoded seed JSON by [storeAllFromJsonEncoded] and + /// [resolveEntitiesSource]. + static List> _toEntitiesJson(Iterable values) => values + .map( + (e) => (e as Map).map( + (key, value) => + MapEntry(key.toString(), value as dynamic), + ), + ) + .toList(); + FutureOr> resolveSource( Object? source, { String? workingPath, @@ -5799,30 +5816,27 @@ abstract class EntityRepository extends EntityAccessor parent: provider, ); + /// Up to this many JSON entries, [storeAllFromJson] creates and stores + /// them all at once; above it, in blocks of [_storeAllFromJsonBlockSize] + /// (logging the progress of a long population). + static const int _storeAllFromJsonSingleBlockMaxLength = 100; + + /// The block size of a [storeAllFromJson] above + /// [_storeAllFromJsonSingleBlockMaxLength] entries: bounds the entities + /// created (and held) before each `storeAll`. + static const int _storeAllFromJsonBlockSize = 50; + FutureOr> _storeAllFromJsonImpl( Iterable> entitiesJson, Transaction? transaction, EntityResolutionRules? resolutionRules, ) => Transaction.executeBlock((transaction) { - if (entitiesJson.length <= 100) { - var osAsync = entitiesJson - .map( - (e) => createFromMap( - e, - entityCache: transaction, - entityProvider: provider, - resolutionRules: resolutionRules, - ), - ) - .resolveAll(); - - return osAsync.resolveMapped((os) { - return storeAll(os, transaction: transaction).resolveWithValue(os); - }); + if (entitiesJson.length <= _storeAllFromJsonSingleBlockMaxLength) { + return _createAndStoreAll(entitiesJson, transaction, resolutionRules); } var entitiesJsonBlocks = entitiesJson - .splitBeforeIndexed((i, e) => i % 50 == 0) + .splitBeforeIndexed((i, e) => i % _storeAllFromJsonBlockSize == 0) .toList(); final entitiesJsonLength = entitiesJson.length; @@ -5849,26 +5863,41 @@ abstract class EntityRepository extends EntityAccessor 'Populating `$name`... $count / $entitiesJsonLength (${progress.toStringAsFixed(2)}%) -- ${elapsedTime.toHumanReadable()} / ${estimatedTime.toHumanReadable()}', ); - var osAsync = entitiesBlock - .map( - (e) => createFromMap( - e, - entityCache: transaction, - entityProvider: provider, - resolutionRules: resolutionRules, - ), - ) - .resolveAll(); - - return osAsync.resolveMapped((os) { - return storeAll(os, transaction: transaction).resolveWithValue(os); - }); + return _createAndStoreAll( + entitiesBlock, + transaction, + resolutionRules, + ); }) .resolveMapped((results) { return results.expand((l) => l).toList(); }); }, transaction: transaction); + /// Creates the entities of [entitiesJson] (see [createFromMap]) and then + /// stores them all in [transaction], returning them. Used by + /// [_storeAllFromJsonImpl] for all the entries at once, or for each block. + FutureOr> _createAndStoreAll( + Iterable> entitiesJson, + Transaction transaction, + EntityResolutionRules? resolutionRules, + ) { + var osAsync = entitiesJson + .map( + (e) => createFromMap( + e, + entityCache: transaction, + entityProvider: provider, + resolutionRules: resolutionRules, + ), + ) + .resolveAll(); + + return osAsync.resolveMapped((os) { + return storeAll(os, transaction: transaction).resolveWithValue(os); + }); + } + FutureOr storeFromJson( Map json, { Transaction? transaction, @@ -5911,25 +5940,27 @@ abstract class EntityRepository extends EntityAccessor return ret.then( (o) => o, onError: (e, s) { - _log.severe( - "Error creating `${entityHandler.type}` from fields Map: $fields", - e, - s, - ); - + _logCreateFromMapError(fields, e, s); throw e; }, ); } catch (e, s) { - _log.severe( - "Error creating `${entityHandler.type}` from fields Map: $fields", - e, - s, - ); + _logCreateFromMapError(fields, e, s); rethrow; } } + /// Logs an error of [createFromMap] (sync or async) with the [fields]. + void _logCreateFromMapError( + Map fields, + Object error, + StackTrace stackTrace, + ) => _log.severe( + "Error creating `${entityHandler.type}` from fields Map: $fields", + error, + stackTrace, + ); + @override Object? getEntityID(O o) => entityHandler.getID(o); @@ -6031,13 +6062,7 @@ abstract class EntityRepository extends EntityAccessor var fields = getEntityFields(o); - var changed = fields.entries.where((e) { - var key = e.key; - var val = e.value; - var prevVal = prevFields[key]; - var eq = entityHandler.equalsFieldValues(key, val, prevVal); - return !eq; - }); + var changed = fields.entries.where((e) => _isFieldChanged(e, prevFields)); var changedFields = changed.map((e) => e.key).toList(); return changedFields; @@ -6051,15 +6076,19 @@ abstract class EntityRepository extends EntityAccessor var fields = getEntityFields(o); - var changed = fields.entries.any((e) { - var key = e.key; - var val = e.value; - var prevVal = prevFields[key]; - var eq = entityHandler.equalsFieldValues(key, val, prevVal); - return !eq; - }); + return fields.entries.any((e) => _isFieldChanged(e, prevFields)); + } - return changed; + /// Returns `true` if the [field] value differs from its tracked value in + /// [prevFields] (see [EntityHandler.equalsFieldValues]). The per-field + /// predicate of [getEntityChangedFields] and [entityHasChangedFields]. + bool _isFieldChanged( + MapEntry field, + Map prevFields, + ) { + var key = field.key; + var prevVal = prevFields[key]; + return !entityHandler.equalsFieldValues(key, field.value, prevVal); } @override @@ -6527,8 +6556,23 @@ class Transaction extends JsonEntityCacheSimple implements EntityProvider { /// Returns the `Transaction` that are open. static List get openInstances => _openInstances.toList(); + /// The operations, in the order they were added (an operation's `id` is + /// its position). See [_operationsSetCache]. final List _operations = []; + /// Cache of the [_operations] as an identity set, for O(1) membership + /// checks: `_operations.contains` on every [addOperation] made a bulk + /// store (`storeAll`, thousands of operations) O(n²). + /// ([TransactionOperation] doesn't override `==`: identity is the same + /// equality `List.contains` used.) + /// + /// `null` until the list grows past [_operationsSetThreshold] (see + /// [_isOperationIn]): most transactions hold one or a few operations, + /// where a scan is cheaper than allocating and filling a set (an eager + /// set cost a `count` ~4% in the benchmark). Kept in sync by + /// [addOperation] (through [_trackOperationIn]) and reset by [dispose]. + Set? _operationsSetCache; + List get operations => UnmodifiableListView(_operations); @@ -6554,9 +6598,64 @@ class Transaction extends JsonEntityCacheSimple implements EntityProvider { /// Returns the executor of this transaction. Object? get executor => mainOperation?.executor; + /// The executed operations, in execution order. + /// See [_executedOperationsSetCache]. final List _executedOperations = []; + /// Cache of the [_executedOperations] as an identity set, for O(1) + /// membership checks: `_executedOperations.contains` on every executed + /// operation made a bulk store O(n²). + /// + /// Created lazily as [_operationsSetCache] (see [_isOperationIn]); kept + /// in sync by [_markOperationExecuted] (through [_trackOperationIn]) and + /// reset by [dispose]. + Set? _executedOperationsSetCache; + + /// The list size above which [_isOperationIn] uses an identity set + /// instead of scanning the list. Up to it a scan is at most a few + /// pointer comparisons; past it, the set keeps a bulk store linear. + static const int _operationsSetThreshold = 16; + + /// `true` if [op] is in [operations] (by identity). Uses [set] when it's + /// been created (the list grew past [_operationsSetThreshold]), otherwise + /// scans the list. Used with [_operations]/[_operationsSetCache] and + /// [_executedOperations]/[_executedOperationsSetCache]. + static bool _isOperationIn( + TransactionOperation op, + List operations, + Set? set, + ) { + if (set != null) return set.contains(op); + for (var o in operations) { + if (identical(o, op)) return true; + } + return false; + } + + /// Keeps an identity-set cache in sync after [op] was added to + /// [operations]: adds it to [set] when the set exists, or creates the + /// set (with the whole list) once the list grows past + /// [_operationsSetThreshold]. Returns the set to store back in the cache + /// field (`null` while the list is small). + static Set? _trackOperationIn( + TransactionOperation op, + List operations, + Set? set, + ) { + if (set != null) return set..add(op); + if (operations.length <= _operationsSetThreshold) return null; + return Set.identity()..addAll(operations); + } + + /// `true` if [op] was added to this transaction ([operations]). + bool _hasOperation(TransactionOperation op) => + _isOperationIn(op, _operations, _operationsSetCache); + + /// `true` if [op] was executed ([executedOperations]). + bool _hasExecutedOperation(TransactionOperation op) => + _isOperationIn(op, _executedOperations, _executedOperationsSetCache); + /// Returns the list of [operations] that were fully executed. List get executedOperations => UnmodifiableListView(_executedOperations); @@ -6732,12 +6831,12 @@ class Transaction extends JsonEntityCacheSimple implements EntityProvider { return false; } - if (!_operations.contains(op)) { + if (!_hasOperation(op)) { throw StateError("Operation not in transaction: $op"); } if (notExecuted == 1) { - return !_executedOperations.contains(op); + return !_hasExecutedOperation(op); } else { return _executedOperations.last == op; } @@ -6871,7 +6970,7 @@ class Transaction extends JsonEntityCacheSimple implements EntityProvider { Object? get result => _result; void addOperation(TransactionOperation op) { - if (_operations.contains(op)) { + if (_hasOperation(op)) { throw StateError("Operation already in transaction: $op"); } @@ -6894,6 +6993,11 @@ class Transaction extends JsonEntityCacheSimple implements EntityProvider { } _operations.add(op); + _operationsSetCache = _trackOperationIn( + op, + _operations, + _operationsSetCache, + ); op.id = _operations.length; } @@ -6940,7 +7044,7 @@ class Transaction extends JsonEntityCacheSimple implements EntityProvider { Object? result, bool allowAutoCommit, ) { - if (_executedOperations.contains(op)) { + if (_hasExecutedOperation(op)) { throw StateError("Operation already executed in transaction: $op"); } @@ -6951,6 +7055,11 @@ class Transaction extends JsonEntityCacheSimple implements EntityProvider { op._endTime = DateTime.now(); _executedOperations.add(op); + _executedOperationsSetCache = _trackOperationIn( + op, + _executedOperations, + _executedOperationsSetCache, + ); if (!_aborted) { _lastResult = result; @@ -7001,13 +7110,13 @@ class Transaction extends JsonEntityCacheSimple implements EntityProvider { ); } - if (!_operations.contains(operation)) { + if (!_hasOperation(operation)) { throw StateError( "Operation not in transaction:\n-- Operation> $operation\n-- $this", ); } - if (_executedOperations.contains(operation)) { + if (_hasExecutedOperation(operation)) { return true; } @@ -7039,12 +7148,12 @@ class Transaction extends JsonEntityCacheSimple implements EntityProvider { return true; } - if (_executedOperations.contains(operation)) { + if (_hasExecutedOperation(operation)) { return true; } } - return _executedOperations.contains(operation); + return _hasExecutedOperation(operation); } FutureOr _doAutoCommit() { @@ -7338,7 +7447,7 @@ class Transaction extends JsonEntityCacheSimple implements EntityProvider { _abortError = abortError; _aborted = true; - for (var op in _operations.whereNotIn(_executedOperations)) { + for (var op in _operations.where((op) => !_hasExecutedOperation(op))) { finishOperationVoid(op, null, allowAutoCommit: false); } @@ -7732,7 +7841,9 @@ class Transaction extends JsonEntityCacheSimple implements EntityProvider { _executionsFutures.clear(); _executedOperations.clear(); + _executedOperationsSetCache = null; _operations.clear(); + _operationsSetCache = null; clearCachedEntities(); @@ -7967,6 +8078,17 @@ abstract class TransactionOperation implements RecursiveToString { String get _executedStatus => isExecuted ? '(executed)' : ''; + /// The common head of the subclasses' [toStringSimple] and [toString]: + /// `[#@#:@]`. + /// - [className]: only [TransactionOperationSubTransaction] shows its own. + /// - [target]: a suffix after the repository name (e.g. the value + /// repository of [TransactionOperationSelectRelationships]). + String _describe({ + String className = 'TransactionOperation', + String target = '', + }) => + '$className$_executedStatus[#$id@#${transaction.id}:${type.name}@$repositoryName$target]'; + @override String toString({Set? processedObjects}); } @@ -7988,9 +8110,14 @@ class TransactionOperationSubTransaction extends TransactionOperation { transaction: parentTransaction, ); + /// The [toStringSimple]/[toString] prefix of a sub-transaction + /// ([_describe]): with its full class name, as it always was (not the + /// `TransactionOperation[...]` of the other operations). + String get _describeSubTransaction => + _describe(className: 'TransactionOperationSubTransaction'); + @override - String toStringSimple() => - 'TransactionOperationSubTransaction$_executedStatus[#$id@#${transaction.id}:subTransaction@$repositoryName]->$durationMsInfo'; + String toStringSimple() => '$_describeSubTransaction->$durationMsInfo'; @override String toString({ @@ -7999,7 +8126,7 @@ class TransactionOperationSubTransaction extends TransactionOperation { processedObjects, this, () => - 'TransactionOperationSubTransaction$_executedStatus[#$id@#${transaction.id}:subTransaction@$repositoryName]->${subTransaction.toString(compact: true)}$durationMsInfo', + '$_describeSubTransaction->${subTransaction.toString(compact: true)}$durationMsInfo', )!; } @@ -8021,18 +8148,16 @@ class TransactionOperationSelect extends TransactionOperation { ); @override - String toStringSimple() => - 'TransactionOperation$_executedStatus[#$id@#${transaction.id}:select@$repositoryName]$durationMsInfo'; + String toStringSimple() => '${_describe()}$durationMsInfo'; @override - String toString({ - Set? processedObjects, - }) => RecursiveToString.recursiveToString( - processedObjects, - this, - () => - 'TransactionOperation$_executedStatus[#$id@#${transaction.id}:select@$repositoryName]{matcher: $matcher$_commandToString}$durationMsInfo', - )!; + String toString({Set? processedObjects}) => + RecursiveToString.recursiveToString( + processedObjects, + this, + () => + '${_describe()}{matcher: $matcher$_commandToString}$durationMsInfo', + )!; } class TransactionOperationCount extends TransactionOperation { @@ -8047,18 +8172,16 @@ class TransactionOperationCount extends TransactionOperation { }) : super(TransactionOperationType.count, repositoryName, false, executor); @override - String toStringSimple() => - 'TransactionOperation$_executedStatus[#$id@#${transaction.id}:count@$repositoryName]$durationMsInfo'; + String toStringSimple() => '${_describe()}$durationMsInfo'; @override - String toString({ - Set? processedObjects, - }) => RecursiveToString.recursiveToString( - processedObjects, - this, - () => - 'TransactionOperation$_executedStatus[#$id@#${transaction.id}:count@$repositoryName]{matcher: $matcher$_commandToString}$durationMsInfo', - )!; + String toString({Set? processedObjects}) => + RecursiveToString.recursiveToString( + processedObjects, + this, + () => + '${_describe()}{matcher: $matcher$_commandToString}$durationMsInfo', + )!; } abstract class TransactionOperationWithEntity extends TransactionOperation { @@ -8105,18 +8228,15 @@ class TransactionOperationStore extends TransactionOperationSaveEntity { ); @override - String toStringSimple() => - 'TransactionOperation$_executedStatus[#$id@#${transaction.id}:store@$repositoryName]$durationMsInfo'; + String toStringSimple() => '${_describe()}$durationMsInfo'; @override - String toString({ - Set? processedObjects, - }) => RecursiveToString.recursiveToString( - processedObjects, - this, - () => - 'TransactionOperation$_executedStatus[#$id@#${transaction.id}:store@$repositoryName]{entity: $entity$_commandToString}$durationMsInfo', - )!; + String toString({Set? processedObjects}) => + RecursiveToString.recursiveToString( + processedObjects, + this, + () => '${_describe()}{entity: $entity$_commandToString}$durationMsInfo', + )!; } class TransactionOperationUpdate extends TransactionOperationSaveEntity { @@ -8136,18 +8256,15 @@ class TransactionOperationUpdate extends TransactionOperationSaveEntity { ); @override - String toStringSimple() => - 'TransactionOperation$_executedStatus[#$id@#${transaction.id}:update@$repositoryName]$durationMsInfo'; + String toStringSimple() => '${_describe()}$durationMsInfo'; @override - String toString({ - Set? processedObjects, - }) => RecursiveToString.recursiveToString( - processedObjects, - this, - () => - 'TransactionOperation$_executedStatus[#$id@#${transaction.id}:update@$repositoryName]{entity: $entity$_commandToString}$durationMsInfo', - )!; + String toString({Set? processedObjects}) => + RecursiveToString.recursiveToString( + processedObjects, + this, + () => '${_describe()}{entity: $entity$_commandToString}$durationMsInfo', + )!; } class TransactionOperationStoreRelationship @@ -8176,8 +8293,7 @@ class TransactionOperationStoreRelationship ); @override - String toStringSimple() => - 'TransactionOperation$_executedStatus[#$id@#${transaction.id}:storeRelationship@$repositoryName]$durationMsInfo'; + String toStringSimple() => '${_describe()}$durationMsInfo'; @override String toString({ @@ -8186,7 +8302,7 @@ class TransactionOperationStoreRelationship processedObjects, this, () => - 'TransactionOperation$_executedStatus[#$id@#${transaction.id}:storeRelationship@$repositoryName]{entity: $entity, other: $others$_commandToString}$durationMsInfo', + '${_describe()}{entity: $entity, other: $others$_commandToString}$durationMsInfo', )!; } @@ -8210,8 +8326,7 @@ class TransactionOperationConstrainRelationship ); @override - String toStringSimple() => - 'TransactionOperation$_executedStatus[#$id@#${transaction.id}:constrainRelationship@$repositoryName]$durationMsInfo'; + String toStringSimple() => '${_describe()}$durationMsInfo'; @override String toString({ @@ -8220,7 +8335,7 @@ class TransactionOperationConstrainRelationship processedObjects, this, () => - 'TransactionOperation$_executedStatus[#$id@#${transaction.id}:constrainRelationship@$repositoryName]{entity: $entity, other: $others$_commandToString}$durationMsInfo', + '${_describe()}{entity: $entity, other: $others$_commandToString}$durationMsInfo', )!; } @@ -8241,18 +8356,15 @@ class TransactionOperationSelectRelationship ); @override - String toStringSimple() => - 'TransactionOperation$_executedStatus[#$id@#${transaction.id}:selectRelationship@$repositoryName]$durationMsInfo'; + String toStringSimple() => '${_describe()}$durationMsInfo'; @override - String toString({ - Set? processedObjects, - }) => RecursiveToString.recursiveToString( - processedObjects, - this, - () => - 'TransactionOperation$_executedStatus[#$id@#${transaction.id}:selectRelationship@$repositoryName]{entity: $entity$_commandToString}$durationMsInfo', - )!; + String toString({Set? processedObjects}) => + RecursiveToString.recursiveToString( + processedObjects, + this, + () => '${_describe()}{entity: $entity$_commandToString}$durationMsInfo', + )!; } class TransactionOperationSelectRelationships extends TransactionOperation { @@ -8275,7 +8387,7 @@ class TransactionOperationSelectRelationships extends TransactionOperation { @override String toStringSimple() => - 'TransactionOperation$_executedStatus[#$id@#${transaction.id}:selectRelationships@$repositoryName->$valueRepositoryName]$durationMsInfo'; + '${_describe(target: '->$valueRepositoryName')}$durationMsInfo'; @override String toString({ @@ -8284,7 +8396,7 @@ class TransactionOperationSelectRelationships extends TransactionOperation { processedObjects, this, () => - 'TransactionOperation$_executedStatus[#$id@#${transaction.id}:selectRelationships@$repositoryName->$valueRepositoryName]{entities: $entities$_commandToString}$durationMsInfo', + '${_describe(target: '->$valueRepositoryName')}{entities: $entities$_commandToString}$durationMsInfo', )!; } @@ -8306,18 +8418,16 @@ class TransactionOperationDelete extends TransactionOperation { ); @override - String toStringSimple() => - 'TransactionOperation$_executedStatus[#$id@#${transaction.id}:delete@$repositoryName]$durationMsInfo'; + String toStringSimple() => '${_describe()}$durationMsInfo'; @override - String toString({ - Set? processedObjects, - }) => RecursiveToString.recursiveToString( - processedObjects, - this, - () => - 'TransactionOperation$_executedStatus[#$id@#${transaction.id}:delete@$repositoryName]{matcher: $matcher$_commandToString}$durationMsInfo', - )!; + String toString({Set? processedObjects}) => + RecursiveToString.recursiveToString( + processedObjects, + this, + () => + '${_describe()}{matcher: $matcher$_commandToString}$durationMsInfo', + )!; } enum TransactionOperationType { @@ -8334,30 +8444,11 @@ enum TransactionOperationType { } extension TransactionOperationTypeExtension on TransactionOperationType { - String get name { - switch (this) { - case TransactionOperationType.subTransaction: - return 'subTransaction'; - case TransactionOperationType.select: - return 'select'; - case TransactionOperationType.count: - return 'count'; - case TransactionOperationType.store: - return 'store'; - case TransactionOperationType.storeRelationship: - return 'storeRelationship'; - case TransactionOperationType.constrainRelationship: - return 'constrainRelationship'; - case TransactionOperationType.selectRelationship: - return 'selectRelationship'; - case TransactionOperationType.selectRelationships: - return 'selectRelationships'; - case TransactionOperationType.update: - return 'update'; - case TransactionOperationType.delete: - return 'delete'; - } - } + /// The same as [Enum.name] (each value's name is its identifier), kept as + /// public API. (`Enum.name` is itself an extension of the SDK, `EnumName`: + /// on a [TransactionOperationType] value this more specific extension is + /// the one called, so it returns the same value as before.) + String get name => (this as Enum).name; /// `true` for an operation that only reads. bool get isRead => switch (this) { @@ -8482,6 +8573,11 @@ abstract class IterableEntityRepository fieldsListEntity.length - fieldsListEntityRef.length; if (fieldsListEntityNoRef > 0) return true; + // From here, every entity field is a reference field (`EntityReference` + // or `EntityReferenceList`), and there's at least one: + // - Only `EntityReferenceList` fields: `false` (the operation doesn't + // propagate for them). + // - Any `EntityReference` field: `true` (it can hold a loaded entity). if (fieldsEntityRef.isEmpty && fieldsListEntityRef.isNotEmpty) return false; return true; @@ -8501,17 +8597,36 @@ abstract class IterableEntityRepository return iterable().length; } - return iterable().where((o) { - return matcher.matchesEntity( - o, - entityHandler: entityHandler, - parameters: parameters, - positionalParameters: positionalParameters, - namedParameters: namedParameters, - ); - }).length; + return iterable() + .where( + (o) => _matchesEntity( + matcher, + o, + parameters, + positionalParameters, + namedParameters, + ), + ) + .length; } + /// [matcher].`matchesEntity` for [o] with this [entityHandler]: the + /// entity predicate of the [iterable] scans, shared by [count] and + /// [matches] (used by the selects and deletes). + bool _matchesEntity( + EntityMatcher matcher, + O o, + Object? parameters, + List? positionalParameters, + Map? namedParameters, + ) => matcher.matchesEntity( + o, + entityHandler: entityHandler, + parameters: parameters, + positionalParameters: positionalParameters, + namedParameters: namedParameters, + ); + @override Object? getEntityID(O o) => getID(o, entityHandler: entityHandler); @@ -8761,11 +8876,7 @@ abstract class IterableEntityRepository }) { checkNotClosed(); - fieldType ??= entityHandler.getFieldType(o, field, resolveFiledName: true)!; - - if (!fieldType.isListEntityOrReference) { - throw StateError("Field `$field` not a `List` entity type: $fieldType"); - } + fieldType = _relationshipFieldType(o, field, fieldType); var op = TransactionOperationStoreRelationship( name, @@ -8792,6 +8903,20 @@ abstract class IterableEntityRepository }); } + /// Returns [fieldType] (or [field]'s type, when `null`), which must be a + /// list of entities (or an [EntityReferenceList]): throws a [StateError] + /// otherwise. The shared validation of [setRelationship] and + /// [selectRelationship]. + TypeInfo _relationshipFieldType(O? o, String field, TypeInfo? fieldType) { + fieldType ??= entityHandler.getFieldType(o, field, resolveFiledName: true)!; + + if (!fieldType.isListEntityOrReference) { + throw StateError("Field `$field` not a `List` entity type: $fieldType"); + } + + return fieldType; + } + FutureOr putRelationship( Object oId, Type valuesType, @@ -8808,11 +8933,7 @@ abstract class IterableEntityRepository }) { checkNotClosed(); - fieldType ??= entityHandler.getFieldType(o, field, resolveFiledName: true)!; - - if (!fieldType.isListEntityOrReference) { - throw StateError("Field `$field` not a `List` entity type: $fieldType"); - } + fieldType = _relationshipFieldType(o, field, fieldType); oId ??= getID(o!, entityHandler: entityHandler)!; @@ -9000,7 +9121,7 @@ abstract class IterableEntityRepository @override FutureOr deleteEntity(O o, {Transaction? transaction}) => - deleteByID(getEntityID(o)); + deleteByID(getEntityID(o), transaction: transaction); @override FutureOr tryDeleteEntity(O o, {Transaction? transaction}) => @@ -9043,15 +9164,15 @@ abstract class IterableEntityRepository bool? orderByID, OrderDirection? orderDirection, }) { - var itr = iterable().where((o) { - return matcher.matchesEntity( + var itr = iterable().where( + (o) => _matchesEntity( + matcher, o, - entityHandler: entityHandler, - parameters: parameters, - positionalParameters: positionalParameters, - namedParameters: namedParameters, - ); - }); + parameters, + positionalParameters, + namedParameters, + ), + ); return _applyOrderAndPagination( itr, diff --git a/lib/src/bones_api_entity_db.dart b/lib/src/bones_api_entity_db.dart index 8c827ede..b8c88508 100644 --- a/lib/src/bones_api_entity_db.dart +++ b/lib/src/bones_api_entity_db.dart @@ -3,6 +3,7 @@ import 'package:bones_api/bones_api.dart'; import 'package:bones_api/src/bones_api_entity_db_object.dart'; import 'package:collection/collection.dart'; import 'package:logging/logging.dart' as logging; +import 'package:meta/meta.dart'; import 'package:statistics/statistics.dart' hide IterableIntExtension; import 'bones_api_mixin.dart'; @@ -445,7 +446,12 @@ abstract class DBAdapter extends SchemeProvider .map((e) => MapEntry(e, e.information(extended: true))) .toMapFromEntries(); + @Deprecated("Renamed to `getEntityRepositoriesTables` (typo)") FutureOr> getEntityRepositoresTables() => + getEntityRepositoriesTables(); + + /// The table of each registered [EntityRepository]. + FutureOr> getEntityRepositoriesTables() => entityRepositories .map( (r) => MapEntry( @@ -478,7 +484,7 @@ abstract class DBAdapter extends SchemeProvider } } - EntityRepository? _geAdapterEntityRepository({ + EntityRepository? _getAdapterEntityRepository({ String? entityName, String? tableName, Type? entityType, @@ -509,7 +515,7 @@ abstract class DBAdapter extends SchemeProvider String? tableName, Type? entityType, }) { - var entityRepository = _geAdapterEntityRepository( + var entityRepository = _getAdapterEntityRepository( entityName: entityName, tableName: tableName, entityType: entityType, @@ -562,7 +568,7 @@ abstract class DBAdapter extends SchemeProvider String? tableName, Type? entityType, }) { - var entityRepository = _geAdapterEntityRepository( + var entityRepository = _getAdapterEntityRepository( entityName: entityName, tableName: tableName, entityType: entityType, @@ -1599,19 +1605,44 @@ class DBEntityRepository extends EntityRepository return _ensureStoredImpl(o, transaction, operation); } else { if (isTrackingEntity(o)) { - return id; + return onEnsureStoredExisting(o, id, transaction); } return existsID(id, transaction: transaction).resolveMapped((exists) { if (!exists) { return _ensureStoredImpl(o, transaction, operation); } else { - return id; + return onEnsureStoredExisting(o, id, transaction); } }); } } + /// Called by [ensureStored] when [o] (with [id]) is already stored (and + /// unchanged): returns the result of [ensureStored]. Default: [id]. + /// + /// A relational repository overrides it to also ensure that the entities + /// referenced by [o] are stored. + @protected + FutureOr onEnsureStoredExisting( + O o, + Object id, + Transaction? transaction, + ) => id; + + /// Stores [o] as a store of [ensureStored] (with [parentOperation], the + /// operation that required it). Each repository implementation stores + /// with its own `store` implementation. + @protected + FutureOr storeWithParentOperation( + O o, + Transaction? transaction, + TransactionOperation? parentOperation, + ) => _storeImpl(o, transaction, parentOperation); + + /// Stores [o] for [ensureStored], unless it's already being stored by + /// [transaction] (then waits for that store, avoiding a recursive + /// relationship loop). The store is [storeWithParentOperation]. FutureOr _ensureStoredImpl( dynamic o, Transaction? transaction, @@ -1637,7 +1668,7 @@ class DBEntityRepository extends EntityRepository } } - return _storeImpl(o, transaction, parentOperation); + return storeWithParentOperation(o, transaction, parentOperation); } @override @@ -1753,7 +1784,7 @@ class DBEntityRepository extends EntityRepository if (matcher is KeyConditionEQ) { var keys = matcher.keys; if (keys.length == 1) { - var conditionKey = matcher.keys.first; + var conditionKey = keys.first; if (conditionKey is ConditionKeyField) { var key = conditionKey.name; @@ -1824,20 +1855,19 @@ class DBEntityRepository extends EntityRepository ); } - FutureOr _selectByID( + /// Creates the [TransactionOperationSelect] of a select of [matcher] (in + /// [transaction]), and resolves its [resolutionRules]. + /// + /// The operation can propagate (to the same transaction) only if the + /// resolved rules select referenced entities: their selects are issued + /// by the operation's `preFinish` (see [resolveEntities]). + @protected + (TransactionOperationSelect, EntityResolutionRulesResolved) + newSelectOperation( Transaction? transaction, - ConditionID matcher, - Object? parameters, + EntityMatcher matcher, EntityResolutionRules? resolutionRules, ) { - var id = matcher.resolveIDValue(parameters: parameters); - - if (id == null && parameters != null) { - id = matcher.getID(parameters); - } - - if (id == null) return null; - final resolutionRulesResolved = resolveEntityResolutionRules( resolutionRules, ); @@ -1852,6 +1882,29 @@ class DBEntityRepository extends EntityRepository transaction: transaction, ); + return (op, resolutionRulesResolved); + } + + FutureOr _selectByID( + Transaction? transaction, + ConditionID matcher, + Object? parameters, + EntityResolutionRules? resolutionRules, + ) { + var id = matcher.resolveIDValue(parameters: parameters); + + if (id == null && parameters != null) { + id = matcher.getID(parameters); + } + + if (id == null) return null; + + final (op, resolutionRulesResolved) = newSelectOperation( + transaction, + matcher, + resolutionRules, + ); + try { return repositoryAdapter.doSelectByID( op, @@ -1888,18 +1941,10 @@ class DBEntityRepository extends EntityRepository var ids = matcher.idsValues.nonNulls.toList(); if (ids.isEmpty) return []; - final resolutionRulesResolved = resolveEntityResolutionRules( - resolutionRules, - ); - - var canPropagate = hasReferencedEntities(resolutionRulesResolved); - - var op = TransactionOperationSelect( - name, - canPropagate, - operationExecutor, + final (op, resolutionRulesResolved) = newSelectOperation( + transaction, matcher, - transaction: transaction, + resolutionRules, ); try { @@ -1938,18 +1983,10 @@ class DBEntityRepository extends EntityRepository bool? orderByID, OrderDirection? orderDirection, }) { - final resolutionRulesResolved = resolveEntityResolutionRules( - resolutionRules, - ); - - var canPropagate = hasReferencedEntities(resolutionRulesResolved); - - var op = TransactionOperationSelect( - name, - canPropagate, - operationExecutor, + final (op, resolutionRulesResolved) = newSelectOperation( + transaction, matcher, - transaction: transaction, + resolutionRules, ); try { @@ -2300,10 +2337,16 @@ class DBEntityRepository extends EntityRepository int get _fieldsListEntityNoRefLength => _fieldsListEntityNoRefLengthCache ??= _fieldsListEntity.length - _fieldsListEntityRef.length; - Map>? _fieldsEntityRepositories; + /// The [EntityRepository] of each entity field (by field name), for the + /// fields whose repository resolves. The entity fields are fixed per + /// [entityHandler], so the (provider lookup) resolution is done once and + /// reused by every select that resolves the sub-entities. + Map>? _fieldsEntityRepositoriesCache; + /// The [EntityRepository] of each entity field (see + /// [_fieldsEntityRepositoriesCache]). Map> _fieldsEntityRepositoriesAll() => - _fieldsEntityRepositories ??= _fieldsEntity.entries + _fieldsEntityRepositoriesCache ??= _fieldsEntity.entries .map((e) { var repo = _resolveEntityRepository(e.value); return repo != null ? MapEntry(e.key, repo) : null; @@ -2482,6 +2525,17 @@ class DBEntityRepository extends EntityRepository return tableField; }); + /// The relationship of each list-of-entities field (see + /// [_getRelationshipFieldsImpl]), by the [TableScheme] they were resolved + /// from: the key is the `fieldsListEntity` map they were resolved for, and + /// the value the relationships by field name. + /// + /// Resolving them looks up each field's target repository and searches the + /// scheme's relationship tables; it's needed by every select/store of + /// entities with list fields. Keyed by the + /// [TableScheme] (an [Expando], weakly), so a reloaded scheme (e.g. after + /// a schema change) resolves them again; the `fieldsListEntity` identity + /// check ensures the cached entry is for the same fields. final Expando< MapEntry, Map> > diff --git a/lib/src/bones_api_entity_db_memory.dart b/lib/src/bones_api_entity_db_memory.dart index 22f9cd06..ecb8a930 100644 --- a/lib/src/bones_api_entity_db_memory.dart +++ b/lib/src/bones_api_entity_db_memory.dart @@ -238,14 +238,20 @@ class DBSQLMemoryAdapter extends DBSQLAdapter >>{}; void _onTablesModification() { - _tablesVersions = null; - _tablesIndexesVersions = null; + _tablesVersionsCache = null; + _tablesIndexesVersionsCache = null; } - Map? _tablesVersions; + /// A cache of [tablesVersions], filled on its first call and reset by + /// [_onTablesModification]. + /// + /// Why: each new connection ([DBSQLMemoryAdapterContext]) snapshots the + /// versions (to roll the tables back to them), usually more often than + /// the tables change: the map isn't rebuilt for each one. + Map? _tablesVersionsCache; Map get tablesVersions => - _tablesVersions ??= UnmodifiableMapView( + _tablesVersionsCache ??= UnmodifiableMapView( _tables.map((table, map) => MapEntry(table, map.version)), ); @@ -609,10 +615,12 @@ class DBSQLMemoryAdapter extends DBSQLAdapter final _tablesIndexes = >>>{}; - Map>? _tablesIndexesVersions; + /// A cache of [tablesIndexesVersions], filled on its first call and reset + /// by [_onTablesModification]. As [_tablesVersionsCache]. + Map>? _tablesIndexesVersionsCache; Map> get tablesIndexesVersions => - _tablesIndexesVersions ??= UnmodifiableMapView( + _tablesIndexesVersionsCache ??= UnmodifiableMapView( _tablesIndexes.map( (table, fieldIndex) => MapEntry( table, @@ -1046,26 +1054,28 @@ class DBSQLMemoryAdapter extends DBSQLAdapter // Resolved by ID or by an index. } else if (condition == null) { itr = map.values; - } else if (tableScheme == null || - (tableScheme.fieldsReferencedTablesLength == 0 && - tableScheme.tableRelationshipReferenceLength == 0) || - condition.isIDCondition) { - itr = map.values.where((e) { - return condition.matchesEntityMap( - e, - positionalParameters: sql.positionalParameters, - namedParameters: sql.namedParameters ?? sql.parametersByPlaceholder, - entityHandler: entityHandler, - ); - }); } else { + // The table's scheme to resolve each row's references with before + // matching it (`null`: matched as stored): not needed without + // references, nor by an ID condition (the row's own ID). + var resolveReferencesScheme = + _hasNoReferences(tableScheme) || condition.isIDCondition + ? null + : tableScheme; + + // A select prefers the SQL's `namedParameters` (a delete only uses the + // `parametersByPlaceholder`: see `doDeleteSQL`). + var namedParameters = sql.namedParameters ?? sql.parametersByPlaceholder; + itr = map.values.where((obj) { - obj = _resolveEntityMap(obj, entityHandler, tableScheme); + if (resolveReferencesScheme != null) { + obj = _resolveEntityMap(obj, entityHandler, resolveReferencesScheme); + } return condition.matchesEntityMap( obj, positionalParameters: sql.positionalParameters, - namedParameters: sql.namedParameters ?? sql.parametersByPlaceholder, + namedParameters: namedParameters, entityHandler: entityHandler, ); }); @@ -1136,10 +1146,7 @@ class DBSQLMemoryAdapter extends DBSQLAdapter // As a select: also a to-many relationship (a table without to-one // references) needs its references resolved to be matched. - final noReferences = - tableScheme == null || - (tableScheme.fieldsReferencedTablesLength == 0 && - tableScheme.tableRelationshipReferenceLength == 0); + final noReferences = _hasNoReferences(tableScheme); // A delete by ID needs neither a scan nor resolving every row (below): // only the deleted row. A miss falls back to the scan, as in a select. @@ -1150,13 +1157,11 @@ class DBSQLMemoryAdapter extends DBSQLAdapter ? byID : MapEntry( byID.key, - _resolveEntityMap(byID.value, entityHandler, tableScheme), + _resolveEntityMap(byID.value, entityHandler, tableScheme!), ), ]; - _checkNotReferencedEntities(entries, table, sql); - - return _removeEntries(table, entries, map); + return _deleteMatchedEntries(table, entries, map, sql); } // A relationship delete (e.g. the roles a user no longer has) only needs @@ -1172,6 +1177,13 @@ class DBSQLMemoryAdapter extends DBSQLAdapter if (byIndex != null) candidates = byIndex; } + // Unlike a select (`sql.namedParameters ?? sql.parametersByPlaceholder`, + // see `_selectEntries`), a delete matches with the + // `parametersByPlaceholder` only (the condition's values as encoded for + // its SQL). The difference predates this code and its reason isn't + // recorded; it's kept, not unified, because unifying could change which + // entries a delete matches when the caller's named parameters differ + // from the encoded ones (a behavior change, out of a refactor's scope). if (noReferences) { var entries = candidates.where((e) { return sql.condition!.matchesEntityMap( @@ -1182,14 +1194,12 @@ class DBSQLMemoryAdapter extends DBSQLAdapter ); }).toList(); - _checkNotReferencedEntities(entries, table, sql); - - return _removeEntries(table, entries, map); + return _deleteMatchedEntries(table, entries, map, sql); } var entries = candidates .map((e) { - var obj = _resolveEntityMap(e.value, entityHandler, tableScheme); + var obj = _resolveEntityMap(e.value, entityHandler, tableScheme!); var match = sql.condition!.matchesEntityMap( obj, @@ -1203,9 +1213,30 @@ class DBSQLMemoryAdapter extends DBSQLAdapter .nonNulls .toList(growable: false); - _checkNotReferencedEntities(entries, table, sql); + return _deleteMatchedEntries(table, entries, map, sql); + } + + /// If [tableScheme] has no references (to-one fields referencing other + /// tables, or relationship tables), or is unknown: its rows are matched as + /// stored, without resolving their references ([_resolveEntityMap]). Used + /// by [_selectEntries] and [doDeleteSQL]. + static bool _hasNoReferences(TableScheme? tableScheme) => + tableScheme == null || + (tableScheme.fieldsReferencedTablesLength == 0 && + tableScheme.tableRelationshipReferenceLength == 0); - return _removeEntries(table, entries, map); + /// Deletes the matched [entries] of [table] (its [tableMap]), after + /// checking that no other table references them (see + /// [_checkNotReferencedEntities]). Returns the deleted rows. Used by the + /// branches of [doDeleteSQL]. + List> _deleteMatchedEntries( + String table, + List>> entries, + Map> tableMap, + SQL sql, + ) { + _checkNotReferencedEntities(entries, table, sql); + return _removeEntries(table, entries, tableMap); } void _checkNotReferencedEntities( @@ -1660,7 +1691,10 @@ class DBSQLMemoryAdapter extends DBSQLAdapter ?.type ?? int; - var relTable = '${sourceTable}__${sourceField}__rel'; + var relTable = SQLGenerator.relationshipTableName( + sourceTable, + sourceField, + ); var relSourceField = '${sourceTable}__$sourceFieldId'; var relTargetField = '${targetTable}__$targetField'; diff --git a/lib/src/bones_api_entity_db_mysql.dart b/lib/src/bones_api_entity_db_mysql.dart index 465b87ca..0d9c03a5 100644 --- a/lib/src/bones_api_entity_db_mysql.dart +++ b/lib/src/bones_api_entity_db_mysql.dart @@ -1,6 +1,7 @@ import 'package:async_extension/async_extension.dart'; import 'package:collection/collection.dart'; import 'package:logging/logging.dart' as logging; +import 'package:meta/meta.dart'; import 'package:mysql1/mysql1.dart'; import 'package:reflection_factory/reflection_factory.dart'; import 'package:statistics/statistics.dart' show Decimal, DynamicInt; @@ -17,7 +18,6 @@ import 'bones_api_logging.dart'; import 'bones_api_sql_builder.dart'; import 'bones_api_types.dart'; import 'bones_api_utils.dart'; -import 'bones_api_utils_timedmap.dart'; final _log = logging.Logger('DBMySQLAdapter')..registerAsDbLogger(); @@ -412,16 +412,15 @@ class DBMySQLAdapter extends DBSQLAdapter notifyTableFieldTypes(table, fieldsTypes); - var fieldsReferencedTables = await _findFieldsReferencedTables( + var fieldsReferencedTables = await findFieldsReferencedTablesCached( connection, table, contextID: contextID, ); - var relationshipTables = await _findRelationshipTables( + var relationshipTables = await findRelationshipTables( connection, table, - idFieldName, contextID: contextID, ); @@ -519,78 +518,8 @@ class DBMySQLAdapter extends DBSQLAdapter } } - Future> _findRelationshipTables( - DBMySqlConnectionWrapper connection, - String table, - String idFieldName, { - Object? contextID, - }) async { - var tablesNames = await _listTablesNames(connection, contextID: contextID); - - var tablesReferences = >[]; - - for (var t in tablesNames) { - var refs = await _findFieldsReferencedTables( - connection, - t, - contextID: contextID, - ); - tablesReferences.add(refs); - } - - tablesReferences = tablesReferences.where((m) { - return m.length > 1 && - m.values.where((r) => r.targetTable == table).isNotEmpty && - m.values.where((r) => r.targetTable != table).isNotEmpty; - }).toList(); - - var relationships = tablesReferences - .map((e) { - var refToTables = e.values - .where((r) => r.targetTable == table) - .toList(growable: false); - - var otherRefs = e.values - .where((r) => r.targetTable != table) - .toList(growable: false); - - if (refToTables.length != 1 || otherRefs.length != 1) { - return null; - } - - var refToTable = refToTables.first; - var otherRef = otherRefs.first; - - return TableRelationshipReference( - refToTable.sourceTable, - refToTable.targetTable, - refToTable.targetField, - refToTable.targetFieldType, - refToTable.sourceField, - otherRef.targetTable, - otherRef.targetField, - otherRef.targetFieldType, - otherRef.sourceField, - ); - }) - .nonNulls - .toList(); - - return relationships; - } - - final Expando>> _listTablesNamesContextCache = - Expando(); - - FutureOr> _listTablesNames( - DBMySqlConnectionWrapper connection, { - Object? contextID, - }) => _listTablesNamesContextCache.putIfAbsentAsync( - contextID, - () => _listTablesNamesImpl(connection), - ); - - Future> _listTablesNamesImpl( + @override + Future> listTablesNamesImpl( DBMySqlConnectionWrapper connection, ) async { var sql = ''' @@ -604,49 +533,6 @@ class DBMySQLAdapter extends DBSQLAdapter return names; } - final TimedMap> - _findFieldsReferencedTablesCache = - TimedMap>(Duration(seconds: 30)); - - @override - TableScheme? disposeTableSchemeCache( - String table, { - bool fieldsTypes = false, - }) { - _findFieldsReferencedTablesCache.remove(table); - return super.disposeTableSchemeCache(table, fieldsTypes: fieldsTypes); - } - - @override - void disposeTablesSchemesCaches({bool fieldsTypes = false}) { - _findFieldsReferencedTablesCache.clear(); - super.disposeTablesSchemesCaches(fieldsTypes: fieldsTypes); - } - - final Expando>>> - _findFieldsReferencedTablesContextCache = Expando(); - - FutureOr> _findFieldsReferencedTables( - DBMySqlConnectionWrapper connection, - String table, { - Object? contextID, - }) { - if (contextID != null) { - var cache = _findFieldsReferencedTablesContextCache[contextID] ??= {}; - return cache[table] ??= _findFieldsReferencedTablesImpl(connection, table) - .then((ret) { - cache[table] = ret; - _findFieldsReferencedTablesCache[table] = ret; - return ret; - }); - } - - return _findFieldsReferencedTablesCache.putIfAbsentCheckedAsync( - table, - () => _findFieldsReferencedTablesImpl(connection, table), - ); - } - /// The indexes of [table]: by name, its columns (in order), and the names /// of the unique ones (`Non_unique = 0`: also `UNIQUE` constraints and the /// `PRIMARY` key). Only plain-column indexes: functional indexes (a key @@ -721,16 +607,14 @@ class DBMySQLAdapter extends DBSQLAdapter String table, Iterable indexesNames, ) { - var names = indexesNames - .map((n) => "'${n.replaceAll("'", "''")}'") - .join(', '); + var names = indexesNames.map(mysqlStringLiteral).join(', '); if (names.isEmpty) return {}; var sql = "SELECT DISTINCT INDEX_NAME AS index_name " "FROM information_schema.STATISTICS " "WHERE TABLE_SCHEMA = DATABASE() " - "AND TABLE_NAME = '${table.replaceAll("'", "''")}' " + "AND TABLE_NAME = ${mysqlStringLiteral(table)} " "AND INDEX_NAME IN ($names)"; return executeWithPool( @@ -744,22 +628,40 @@ class DBMySQLAdapter extends DBSQLAdapter ); } - Future> _findFieldsReferencedTablesImpl( - DBMySqlConnectionWrapper connection, + /// [value] as a MySQL string literal ([DBSQLAdapter.sqlStringLiteral] + /// with `escapeBackslashes`): MySQL's default `sql_mode` treats `\` as an + /// escape character in literals, so a `\` must be escaped too (or `\'` + /// would end the literal). + @visibleForTesting + static String mysqlStringLiteral(String value) => + DBSQLAdapter.sqlStringLiteral(value, escapeBackslashes: true); + + /// The SQL of [findFieldsReferencedTablesImpl]: the foreign keys of + /// [table] in the database [databaseName] (both escaped as string + /// literals, see [mysqlStringLiteral]). + @visibleForTesting + static String findFieldsReferencedTablesSQL( + String databaseName, String table, - ) async { - var sql = - ''' - SELECT + ) => + ''' + SELECT CONSTRAINT_NAME, TABLE_NAME, COLUMN_NAME, REFERENCED_TABLE_NAME, REFERENCED_COLUMN_NAME FROM INFORMATION_SCHEMA.KEY_COLUMN_USAGE WHERE - TABLE_SCHEMA = '$databaseName' AND TABLE_NAME = '$table' + TABLE_SCHEMA = ${mysqlStringLiteral(databaseName)} AND TABLE_NAME = ${mysqlStringLiteral(table)} AND REFERENCED_TABLE_NAME IS NOT NULL AND REFERENCED_COLUMN_NAME IS NOT NULL '''; + @override + Future> findFieldsReferencedTablesImpl( + DBMySqlConnectionWrapper connection, + String table, + ) async { + var sql = findFieldsReferencedTablesSQL(databaseName, table); + var results = await connection.query(sql); var referenceFields = results @@ -1063,32 +965,15 @@ class DBMySQLAdapter extends DBSQLAdapter .resolveMapped((results) { var affectedRows = results.affectedRows; if (affectedRows == 0) { - var entry = sql.parametersByPlaceholder; - if (!allowAutoInsert) { - throw StateError( - "Can't update not stored entity into table `$table`: $entry", - ); - } - - var fields = sql.namedParameters!; - - return generateInsertSQL( + return autoInsertAfterNoopUpdate( transaction, entityName, table, - fields, - ).resolveMapped((insertSQL) { - _log.info( - 'Update not affecting any row! Auto inserting: $insertSQL', - ); - return doInsertSQL( - entityName, - table, - insertSQL, - transaction, - connection, - ); - }); + sql, + connection, + allowAutoInsert: allowAutoInsert, + logger: _log, + ); } return _resolveResultID(results, table, sql, id); @@ -1195,61 +1080,22 @@ class DBMySQLAdapter extends DBSQLAdapter } @override - Future openTransaction(Transaction transaction) { - var contextCompleter = Completer(); - - var result = executeWithPool( - (connection) { - return connection.openTransaction((transactionWrap) { - contextCompleter.complete(transactionWrap); - - return transaction.transactionFuture.then( - (res) => resolveTransactionResult(res, transaction, connection), - onError: (e, s) { - cancelTransaction(transaction, connection, e, s); - throw e; - }, - ); - }); - }, - validator: (c) => !transaction.isAborted, - onError: (e, s) => transaction.notifyExecutionError( - e, - s, - errorResolver: resolveError, - debugInfo: () => transaction.toString(withExecutedOperations: false), - ), - ); - - transaction.transactionResult = result; - - return contextCompleter.future; - } - - @override - bool get cancelTransactionResultWithError => false; - - @override - bool get throwTransactionResultWithError => true; - - @override - bool cancelTransaction( - Transaction transaction, - DBMySqlConnectionWrapper? connection, - Object? error, - StackTrace? stackTrace, - ) { - return true; - } - - @override - bool get callCloseTransactionRequired => false; - - @override - FutureOr closeTransaction( - Transaction transaction, - DBMySqlConnectionWrapper? connection, - ) {} + Future openTransaction(Transaction transaction) => + openTransactionWithPool( + transaction, + (connection, onTransactionContext) => + connection.openTransaction((transactionWrap) { + onTransactionContext(transactionWrap); + + return transaction.transactionFuture.then( + (res) => resolveTransactionResult(res, transaction, connection), + onError: (e, s) { + cancelTransaction(transaction, connection, e, s); + throw e; + }, + ); + }), + ); @override String toString() { diff --git a/lib/src/bones_api_entity_db_object.dart b/lib/src/bones_api_entity_db_object.dart index f75643e7..4ed783b3 100644 --- a/lib/src/bones_api_entity_db_object.dart +++ b/lib/src/bones_api_entity_db_object.dart @@ -1,6 +1,16 @@ +import 'dart:typed_data'; + +import 'package:async_extension/async_extension.dart'; import 'package:bones_api/bones_api.dart'; +import 'package:logging/logging.dart' as logging; +import 'package:meta/meta.dart'; +import 'package:statistics/statistics.dart' show StringExtension; import 'package:swiss_knife/swiss_knife.dart' show parseBool; +import 'bones_api_logging.dart'; + +final _log = logging.Logger('DBObjectAdapter')..registerAsDbLogger(); + /// Base class for Object DB adapters. /// /// A [DBObjectAdapter] implementation is responsible to connect to the database @@ -129,6 +139,506 @@ abstract class DBObjectAdapter extends DBAdapter { var parentRepositoryProvider = this.parentRepositoryProvider; return [?parentRepositoryProvider]; } + + // ----------------------------------------------------------------------- + // Implementation shared by the object adapters (memory, directory, GCS). + // + // The members below were copy-pasted in each adapter, differing only in the + // context type `C`, the logger, the exception type and the encoding of the + // bytes fields. Those differences are the hooks [adapterLogger], + // [createAdapterException] and [encodeBytesAsDataURL]. + // ----------------------------------------------------------------------- + + /// The logger of this adapter's messages (e.g. the resolved [TableScheme]s). + /// + /// Each adapter overrides it with its own logger, so its messages keep + /// their logger name (e.g. `DBObjectDirectoryAdapter`). + @protected + logging.Logger get adapterLogger => _log; + + /// Creates the error of a failed operation of this adapter (see + /// [resolveError]). + /// + /// Each adapter overrides it to throw its own exception type (e.g. + /// `DBObjectGCSAdapterException`): callers can catch an adapter's errors by + /// type. + @protected + DBObjectAdapterException createAdapterException( + String type, + String message, { + Object? parentError, + StackTrace? parentStackTrace, + Object? operation, + Object? previousError, + }) => DBObjectAdapterException( + type, + message, + parentError: parentError, + parentStackTrace: parentStackTrace, + operation: operation, + previousError: previousError, + ); + + /// If `true`, a bytes ([Uint8List]) field is stored as a [DataURLBase64] + /// string, by [normalizeEntityJSON]. + /// + /// The adapters that encode the objects as JSON (directory, GCS) need it, + /// as JSON has no bytes type. [DBObjectMemoryAdapter] keeps the objects in + /// memory, so it stores the bytes as they are (overriding it as `false`). + @protected + bool get encodeBytesAsDataURL => true; + + /// The registered [TableScheme]s, by table name (see [addTableSchemes]). + /// + /// A table without a registered scheme has it resolved from its + /// [EntityHandler] (see [getTableSchemeImpl]). + final Map tablesSchemes = {}; + + /// Registers [tablesSchemes] (replacing the ones with the same name). + void addTableSchemes(Iterable tablesSchemes) { + for (var s in tablesSchemes) { + this.tablesSchemes[s.name] = s; + } + } + + @override + TableScheme? getTableSchemeImpl( + String table, + TableRelationshipReference? relationship, { + Object? contextID, + }) { + var tableScheme = tablesSchemes[table]; + if (tableScheme != null) return tableScheme; + + var entityHandler = getEntityHandler(tableName: table); + if (entityHandler == null) { + // A relationship table has no entity: its scheme is the 2 ID fields + // (source and target) of the relationship. + if (relationship != null) { + var sourceId = + '${relationship.sourceTable}_${relationship.sourceField}'; + var targetId = + '${relationship.targetTable}_${relationship.targetField}'; + + tableScheme = TableScheme( + table, + relationship: true, + idFieldName: sourceId, + fieldsTypes: { + sourceId: relationship.sourceFieldType, + targetId: relationship.targetFieldType, + }, + ); + + adapterLogger.info('relationship> $tableScheme'); + + return tableScheme; + } + + throw StateError( + "Can't resolve `TableScheme` for table `$table`. No `EntityHandler` found for table `$table`!", + ); + } + + var idFieldName = entityHandler.idFieldName(); + + var entityFieldsTypes = entityHandler.fieldsTypes(); + + var fieldsTypes = entityFieldsTypes.map( + (key, value) => MapEntry(key, value.type), + ); + + tableScheme = TableScheme( + table, + relationship: relationship != null, + idFieldName: idFieldName, + fieldsTypes: fieldsTypes, + ); + + adapterLogger.info('$tableScheme'); + + return tableScheme; + } + + @override + FutureOr?> getTableFieldsTypesImpl(String table) { + return tablesSchemes[table]?.fieldsTypes; + } + + @override + FutureOr isConnectionValid(C connection, {bool checkUsage = true}) => + true; + + /// The ID field of [table]: from its registered [TableScheme], or `id`. + @protected + String getTableIDFieldName(String table) { + var tablesScheme = tablesSchemes[table]; + + var idField = tablesScheme?.idFieldName ?? 'id'; + return idField; + } + + /// Orders (by the ID field of [table]) and paginates the selected + /// [entries]: the object adapters select all the objects (no query + /// engine), and then apply the select's order and limits here. + @protected + List> applyTableOrderAndPagination( + String table, + Iterable> entries, { + int? limit, + int? offset, + bool? orderByID, + OrderDirection? orderDirection, + }) { + var idField = getTableIDFieldName(table); + + return applySelectOrderAndPagination( + entries, + (e) => e[idField], + limit: limit, + offset: offset, + orderByID: orderByID, + orderDirection: orderDirection, + ).toList(); + } + + static final _regExpNormalizeIDInvalidChars = RegExp(r'[^\w.-]+'); + static final _regExpNormalizeIDInvalidPrefix = RegExp(r'^\.+'); + + /// Normalizes the object ID [o] into a safe file/object name: only word + /// characters, `.` and `-`, no leading `.` (no hidden files, nor `..`), + /// and at most 220 characters (below the usual file name limits). + @protected + String normalizeObjectID(Object? o) { + var id = o?.toString() ?? ''; + if (id.isEmpty) { + throw StateError("Empty ID string."); + } + + id = id + .replaceAll(_regExpNormalizeIDInvalidChars, '') + .replaceAll(_regExpNormalizeIDInvalidPrefix, ''); + id = id.truncate(220); + return id; + } + + static final _regExpNormalizeTableNonWord = RegExp(r'\W'); + + /// Normalizes [table] into a safe directory/prefix name (non-word + /// characters replaced by `_`). + @protected + String normalizeTableName(String table) { + table = table.trim().replaceAll(_regExpNormalizeTableNonWord, '_'); + if (table.isEmpty) { + throw StateError("Empty table name."); + } + return table; + } + + /// Normalizes an entity JSON to be stored: the references to other + /// entities (single, [EntityReference], lists and [EntityReferenceList]) + /// become their IDs, and a bytes field becomes a [DataURLBase64] string + /// (if [encodeBytesAsDataURL]). + @protected + Map normalizeEntityJSON( + Map entityJson, { + String? entityName, + String? table, + EntityRepository? entityRepository, + }) { + entityRepository ??= getEntityRepository( + name: entityName, + tableName: table, + ); + + if (entityRepository == null) { + throw StateError( + "Can't determine `EntityRepository` for: entityName=$entityName ; tableName=$table", + ); + } + + var entityHandler = entityRepository.entityHandler; + + var encodeBytesAsDataURL = this.encodeBytesAsDataURL; + + var entityJsonNormalized = entityJson.map((key, value) { + if (value == null || (value as Object).isPrimitiveValue) { + return MapEntry(key, value); + } else if (encodeBytesAsDataURL && value is Uint8List) { + var dataURLBase64 = DataURLBase64.from(value); + return MapEntry(key, dataURLBase64.toString()); + } + + var fieldType = entityHandler.getFieldType( + null, + key, + resolveFiledName: false, + ); + + if (fieldType != null) { + if (fieldType.isEntityReferenceType) { + var entityType = fieldType.arguments0!.type; + var fieldEntityRepository = getEntityRepositoryByType(entityType); + + if (fieldEntityRepository == null) { + throw StateError( + "Can't determine `EntityRepository` for field `$key`: fieldType=$fieldType", + ); + } + + if (value is EntityReference) { + if (value.isNull) { + value = null; + } else if (value.isIdSet) { + value = value.id; + } else if (value.isEntitySet) { + value = fieldEntityRepository.getEntityID(value.entity); + } else { + throw StateError("Invalid state: $value"); + } + } else if (!value.isPrimitiveValue) { + value = fieldEntityRepository.getEntityID(value); + } + } else if (fieldType.isEntityReferenceListType) { + var listEntityType = fieldType.arguments0!; + + var fieldListEntityRepository = getEntityRepositoryByTypeInfo( + listEntityType, + ); + if (fieldListEntityRepository == null) { + throw StateError( + "Can't determine `EntityRepository` for field `$key` List type: fieldType=$fieldType", + ); + } + + if (value is EntityReferenceList) { + // Its IDs (from its entities, if only those are set): + value = value.isNull ? null : value.ids; + } else { + var valIter = value is Iterable ? value : [value]; + + value = valIter + .map( + (v) => fieldListEntityRepository.isOfEntityType(v) + ? fieldListEntityRepository.getEntityID(v) + : v, + ) + .toList(); + } + } else if (fieldType.isListEntity) { + var listEntityType = fieldType.listEntityType!; + + var fieldListEntityRepository = getEntityRepositoryByTypeInfo( + listEntityType, + ); + if (fieldListEntityRepository == null) { + throw StateError( + "Can't determine `EntityRepository` for field `$key` List type: fieldType=$fieldType", + ); + } + + var valIter = value is Iterable ? value : [value]; + + value = valIter + .map( + (v) => fieldListEntityRepository.isOfEntityType(v) + ? fieldListEntityRepository.getEntityID(v) + : v, + ) + .toList(); + } else if (!fieldType.isPrimitiveType && fieldType.entityType != null) { + var entityType = fieldType.entityType!; + var fieldEntityRepository = getEntityRepositoryByType(entityType); + + if (fieldEntityRepository == null) { + throw StateError( + "Can't determine `EntityRepository` for field `$key`: fieldType=$fieldType", + ); + } + + value = fieldEntityRepository.getEntityID(value); + } + } + + return MapEntry(key, value); + }); + + return entityJsonNormalized; + } + + /// The open transactions' contexts, with their opening time. + /// + /// Registered by [openTransaction], and removed when the transaction is + /// canceled ([cancelTransaction]) or closed ([closeTransaction]). An + /// adapter can delay the consolidation of a closed transaction while + /// others are still open (see [onTransactionContextClosed]). + final Map _openTransactionsContexts = {}; + + @override + FutureOr openTransaction(Transaction transaction) => + createConnection().resolveMapped((conn) { + _openTransactionsContexts[conn] = DateTime.now(); + + transaction.transactionFuture + // ignore: discarded_futures + .then( + // ignore: discarded_futures + (res) => resolveTransactionResult(res, transaction, conn), + onError: (e, s) { + cancelTransaction(transaction, conn, e, s); + throw e; + }, + ); + + return conn; + }); + + @override + bool get cancelTransactionResultWithError => true; + + @override + bool get throwTransactionResultWithError => false; + + @override + bool cancelTransaction( + Transaction transaction, + C? connection, + Object? error, + StackTrace? stackTrace, + ) { + _openTransactionsContexts.remove(connection); + if (connection != null) { + onTransactionContextCanceled(connection); + } + return true; + } + + /// Called when the transaction of [context] is canceled (after removing it + /// from the open ones). Default: does nothing (the object adapters write + /// immediately). [DBObjectMemoryAdapter] rolls back its tables. + @protected + void onTransactionContextCanceled(C context) {} + + @override + bool get callCloseTransactionRequired => true; + + @override + FutureOr closeTransaction(Transaction transaction, C? connection) { + if (connection != null) { + _consolidateTransactionContext(connection); + } + } + + /// Removes the closed transaction [context] from the open ones, and + /// notifies [onTransactionContextClosed]. Used by [closeTransaction]. + void _consolidateTransactionContext(C context) { + _openTransactionsContexts.remove(context); + + onTransactionContextClosed( + context, + otherTransactionsOpen: _openTransactionsContexts.isNotEmpty, + ); + } + + /// Called when the transaction of [context] is closed (after removing it + /// from the open ones). [otherTransactionsOpen] tells if other + /// transactions are still open. Default: does nothing. + /// [DBObjectMemoryAdapter] consolidates its tables (once all the + /// transactions are closed). + @protected + void onTransactionContextClosed( + C context, { + required bool otherTransactionsOpen, + }) {} + + /// Resolves an operation [error] into this adapter's exception (see + /// [createAdapterException]). + Object resolveError( + Object error, + StackTrace stackTrace, + Object? operation, + Object? previousError, + ) => createAdapterException( + 'error', + '$error', + parentError: error, + parentStackTrace: stackTrace, + previousError: previousError, + operation: operation, + ); + + /// Finishes [op] with [res] (see [finishOperation]), resolving the errors + /// with [resolveError]. Used by [executeOperation]. + FutureOr _finishOperation( + TransactionOperation op, + T res, + PreFinishDBOperation? preFinish, + ) => finishOperation(op, res, preFinish, errorResolver: resolveError); + + /// Executes [f] in the transaction of [op], and finishes [op] (with + /// [preFinish]) after that execution, not inside it: a [preFinish] can + /// issue other operations in the same transaction (e.g. selecting the + /// sub-entities), executed after this one (inside it, they would wait for + /// it, that waits for them). + @protected + FutureOr executeOperation( + TransactionOperation op, + FutureOr Function(C connection) f, + PreFinishDBOperation? preFinish, + ) => executeTransactionOperation( + op, + f, + ).resolveMapped((res) => _finishOperation(op, res, preFinish)); + + /// Executes [f] with a connection of the transaction of [op]: from the + /// pool if it's a single operation transaction, otherwise as an execution + /// of the transaction (opening it if needed). + FutureOr executeTransactionOperation( + TransactionOperation op, + FutureOr Function(C connection) f, + ) { + var transaction = op.transaction; + + if (isTransactionWithSingleOperation(op)) { + return executeWithPool( + f, + onError: (e, s) => transaction.notifyExecutionError( + e, + s, + errorResolver: resolveError, + operation: op, + debugInfo: () => op.toString(), + ), + ); + } + + if (transaction.isOpen) { + return transaction.addExecution( + f, + errorResolver: resolveError, + operation: op, + debugInfo: () => op.toString(), + ); + } + + if (!transaction.isOpening) { + transaction.open( + () => openTransaction(transaction), + callCloseTransactionRequired + ? () => closeTransaction(transaction, transaction.context as C?) + : null, + ); + } + + return transaction.onOpen(() { + return transaction.addExecution( + f, + errorResolver: resolveError, + operation: op, + debugInfo: () => op.toString(), + ); + }); + } } /// Error thrown by [DBObjectAdapter] operations. diff --git a/lib/src/bones_api_entity_db_object_directory.dart b/lib/src/bones_api_entity_db_object_directory.dart index 259a8b47..062ce189 100644 --- a/lib/src/bones_api_entity_db_object_directory.dart +++ b/lib/src/bones_api_entity_db_object_directory.dart @@ -1,21 +1,15 @@ import 'dart:convert' as dart_convert; import 'dart:io'; import 'dart:math' as math; -import 'dart:typed_data'; import 'package:async_extension/async_extension.dart'; import 'package:logging/logging.dart' as logging; import 'package:path/path.dart' as pack_path; -import 'package:reflection_factory/reflection_factory.dart'; -import 'package:statistics/statistics.dart'; -import 'package:swiss_knife/swiss_knife.dart'; import 'bones_api_condition.dart'; -import 'bones_api_condition_encoder.dart'; import 'bones_api_entity.dart'; import 'bones_api_entity_db.dart'; import 'bones_api_entity_db_object.dart'; -import 'bones_api_entity_reference.dart'; import 'bones_api_extension.dart'; import 'bones_api_extension_io.dart'; import 'bones_api_logging.dart'; @@ -211,6 +205,14 @@ class DBObjectDirectoryAdapter int _connectionCount = 0; + /// The shared [DBObjectAdapter.openTransaction], typed as this adapter's + /// context: it opens synchronously ([createConnection] is synchronous), as + /// this adapter's own `openTransaction` was declared before the shared + /// implementation (a caller may rely on the concrete type). + @override + DBObjectDirectoryAdapterContext openTransaction(Transaction transaction) => + super.openTransaction(transaction) as DBObjectDirectoryAdapterContext; + @override DBObjectDirectoryAdapterContext createConnection() { var id = ++_connectionCount; @@ -233,138 +235,25 @@ class DBObjectDirectoryAdapter return info; } - final Map tablesSchemes = {}; - - void addTableSchemes(Iterable tablesSchemes) { - for (var s in tablesSchemes) { - this.tablesSchemes[s.name] = s; - } - } - - @override - TableScheme? getTableSchemeImpl( - String table, - TableRelationshipReference? relationship, { - Object? contextID, - }) { - //_log.info('getTableSchemeImpl> $table ; relationship: $relationship'); - - var tableScheme = tablesSchemes[table]; - if (tableScheme != null) return tableScheme; - - var entityHandler = getEntityHandler(tableName: table); - if (entityHandler == null) { - if (relationship != null) { - var sourceId = - '${relationship.sourceTable}_${relationship.sourceField}'; - var targetId = - '${relationship.targetTable}_${relationship.targetField}'; - - tableScheme = TableScheme( - table, - relationship: true, - idFieldName: sourceId, - fieldsTypes: { - sourceId: relationship.sourceFieldType, - targetId: relationship.targetFieldType, - }, - ); - - _log.info('relationship> $tableScheme'); - - return tableScheme; - } - - throw StateError( - "Can't resolve `TableScheme` for table `$table`. No `EntityHandler` found for table `$table`!", - ); - } - - var idFieldName = entityHandler.idFieldName(); - - var entityFieldsTypes = entityHandler.fieldsTypes(); - - var fieldsTypes = entityFieldsTypes.map( - (key, value) => MapEntry(key, value.type), - ); - - tableScheme = TableScheme( - table, - relationship: relationship != null, - idFieldName: idFieldName, - fieldsTypes: fieldsTypes, - ); - - _log.info('$tableScheme'); - - return tableScheme; - } - - @override - FutureOr?> getTableFieldsTypesImpl(String table) { - return tablesSchemes[table]?.fieldsTypes; - } - - @override - FutureOr isConnectionValid(connection, {bool checkUsage = true}) => - true; - - final Map - _openTransactionsContexts = {}; - - @override - DBObjectDirectoryAdapterContext openTransaction(Transaction transaction) { - var conn = createConnection(); - - _openTransactionsContexts[conn] = DateTime.now(); - - transaction.transactionFuture - // ignore: discarded_futures - .then( - // ignore: discarded_futures - (res) => resolveTransactionResult(res, transaction, conn), - onError: (e, s) { - cancelTransaction(transaction, conn, e, s); - throw e; - }, - ); - - return conn; - } - - @override - bool get cancelTransactionResultWithError => true; - - @override - bool get throwTransactionResultWithError => false; - - @override - bool cancelTransaction( - Transaction transaction, - DBObjectDirectoryAdapterContext? connection, - Object? error, - StackTrace? stackTrace, - ) { - _openTransactionsContexts.remove(connection); - return true; - } - @override - bool get callCloseTransactionRequired => true; + logging.Logger get adapterLogger => _log; @override - FutureOr closeTransaction( - Transaction transaction, - DBObjectDirectoryAdapterContext? connection, - ) { - if (connection != null) { - _consolidateTransactionContext(connection); - } - } - - void _consolidateTransactionContext(DBObjectDirectoryAdapterContext context) { - _openTransactionsContexts.remove(context); - } + DBObjectDirectoryAdapterException createAdapterException( + String type, + String message, { + Object? parentError, + StackTrace? parentStackTrace, + Object? operation, + Object? previousError, + }) => DBObjectDirectoryAdapterException( + type, + message, + parentError: parentError, + parentStackTrace: parentStackTrace, + operation: operation, + previousError: previousError, + ); @override String toString() { @@ -383,7 +272,7 @@ class DBObjectDirectoryAdapter Map? namedParameters, PreFinishDBOperation? preFinish, }) { - return _executeOperation( + return executeOperation( op, (conn) => _doCountImpl(op, table, matcher, parameters ?? namedParameters), preFinish, @@ -437,7 +326,7 @@ class DBObjectDirectoryAdapter String table, List ids, ) { - return _executeOperation, List>( + return executeOperation, List>( op, (conn) => _doExistIDsImpl(op, table, ids), null, @@ -467,7 +356,7 @@ class DBObjectDirectoryAdapter String table, Object id, { PreFinishDBOperation?, R?>? preFinish, - }) => _executeOperation?, R?>( + }) => executeOperation?, R?>( op, (conn) => _doSelectByIDImpl(table, id, entityName), preFinish, @@ -496,7 +385,7 @@ class DBObjectDirectoryAdapter bool? orderByID, OrderDirection? orderDirection, PreFinishDBOperation>, List>? preFinish, - }) => _executeOperation>, List>( + }) => executeOperation>, List>( op, (conn) => _doSelectByIDsImpl( table, @@ -524,7 +413,7 @@ class DBObjectDirectoryAdapter var entries = await _readObjects(table, ids); - return _applyOrderAndPagination( + return applyTableOrderAndPagination( table, entries, limit: limit, @@ -544,7 +433,7 @@ class DBObjectDirectoryAdapter bool? orderByID, OrderDirection? orderDirection, PreFinishDBOperation>, List>? preFinish, - }) => _executeOperation>, List>( + }) => executeOperation>, List>( op, (conn) => _doSelectAllImpl( table, @@ -578,7 +467,7 @@ class DBObjectDirectoryAdapter var entries = _readObjects(table, ids); return entries.resolveMapped( - (entries) => _applyOrderAndPagination( + (entries) => applyTableOrderAndPagination( table, entries, limit: limit, @@ -589,26 +478,6 @@ class DBObjectDirectoryAdapter ); } - List> _applyOrderAndPagination( - String table, - Iterable> entries, { - int? limit, - int? offset, - bool? orderByID, - OrderDirection? orderDirection, - }) { - var idField = _getTableIDFieldName(table); - - return applySelectOrderAndPagination( - entries, - (e) => e[idField], - limit: limit, - offset: offset, - orderByID: orderByID, - orderDirection: orderDirection, - ).toList(); - } - @override FutureOr doDelete( TransactionOperation op, @@ -619,7 +488,7 @@ class DBObjectDirectoryAdapter List? positionalParameters, Map? namedParameters, PreFinishDBOperation>, R>? preFinish, - }) => _executeOperation>, R>( + }) => executeOperation>, R>( op, (conn) => _doDeleteImpl(op, table, matcher, parameters ?? namedParameters), @@ -656,13 +525,6 @@ class DBObjectDirectoryAdapter throw UnsupportedError("Relationship delete not supported for: $matcher"); } - String _getTableIDFieldName(String table) { - var tablesScheme = tablesSchemes[table]; - - var idField = tablesScheme?.idFieldName ?? 'id'; - return idField; - } - @override FutureOr doInsert( TransactionOperation op, @@ -672,7 +534,7 @@ class DBObjectDirectoryAdapter Map fields, { String? idFieldName, PreFinishDBOperation? preFinish, - }) => _executeOperation( + }) => executeOperation( op, (conn) => _doInsertImpl(op, table, fields, entityName), preFinish, @@ -689,13 +551,13 @@ class DBObjectDirectoryAdapter tableDir.createSync(); } - var entry = _normalizeEntityJSON( + var entry = normalizeEntityJSON( fields, entityName: entityName, table: table, ); - var idField = _getTableIDFieldName(table); + var idField = getTableIDFieldName(table); var id = entry[idField]; _log.info( @@ -722,7 +584,7 @@ class DBObjectDirectoryAdapter String? idFieldName, PreFinishDBOperation? preFinish, bool allowAutoInsert = false, - }) => _executeOperation( + }) => executeOperation( op, (conn) => _doUpdateImpl(op, table, fields, entityName, id), preFinish, @@ -744,7 +606,7 @@ class DBObjectDirectoryAdapter '[transaction:${op.transactionId}] doUpdate> UPDATE INTO $table OBJECT `$id`', ); - var changes = _normalizeEntityJSON( + var changes = normalizeEntityJSON( fields, entityName: entityName, table: table, @@ -757,7 +619,7 @@ class DBObjectDirectoryAdapter ...changes, }; - var idField = _getTableIDFieldName(table); + var idField = getTableIDFieldName(table); entry[idField] = id; _saveObject(table, id, entry); @@ -847,251 +709,17 @@ class DBObjectDirectoryAdapter } File _resolveObjectFile(String table, Object? id) { - var idStr = _normalizeID(id); + var idStr = normalizeObjectID(id); var tableDir = _resolveTableDirectory(table); var file = File(pack_path.join(tableDir.path, '$idStr.json')); return file; } Directory _resolveTableDirectory(String table) { - var tableStr = _normalizeTableName(table); + var tableStr = normalizeTableName(table); var tableDir = Directory(pack_path.join(directory.path, tableStr)); return tableDir; } - - static final _regExpNormalizeIDInvalidChars = RegExp(r'[^\w.-]+'); - static final _regExpNormalizeIDInvalidPrefix = RegExp(r'^\.+'); - - String _normalizeID(Object? o) { - var id = o?.toString() ?? ''; - if (id.isEmpty) { - throw StateError("Empty ID string."); - } - - id = id - .replaceAll(_regExpNormalizeIDInvalidChars, '') - .replaceAll(_regExpNormalizeIDInvalidPrefix, ''); - id = id.truncate(220); - return id; - } - - static final _regExpNormalizeTableNonWord = RegExp(r'\W'); - - String _normalizeTableName(String table) { - table = table.trim().replaceAll(_regExpNormalizeTableNonWord, '_'); - if (table.isEmpty) { - throw StateError("Empty table name."); - } - return table; - } - - Map _normalizeEntityJSON( - Map entityJson, { - String? entityName, - String? table, - EntityRepository? entityRepository, - }) { - entityRepository ??= getEntityRepository( - name: entityName, - tableName: table, - ); - - if (entityRepository == null) { - throw StateError( - "Can't determine `EntityRepository` for: entityName=$entityName ; tableName=$table", - ); - } - - var entityHandler = entityRepository.entityHandler; - - var entityJsonNormalized = entityJson.map((key, value) { - if (value == null || (value as Object).isPrimitiveValue) { - return MapEntry(key, value); - } else if (value is Uint8List) { - var dataURLBase64 = DataURLBase64.from(value); - return MapEntry(key, dataURLBase64.toString()); - } - - var fieldType = entityHandler.getFieldType( - null, - key, - resolveFiledName: false, - ); - - if (fieldType != null) { - if (fieldType.isEntityReferenceType) { - var entityType = fieldType.arguments0!.type; - var fieldEntityRepository = getEntityRepositoryByType(entityType); - - if (fieldEntityRepository == null) { - throw StateError( - "Can't determine `EntityRepository` for field `$key`: fieldType=$fieldType", - ); - } - - if (value is EntityReference) { - if (value.isNull) { - value = null; - } else if (value.isIdSet) { - value = value.id; - } else if (value.isEntitySet) { - value = fieldEntityRepository.getEntityID(value.entity); - } else { - throw StateError("Invalid state: $value"); - } - } else if (!value.isPrimitiveValue) { - value = fieldEntityRepository.getEntityID(value); - } - } else if (fieldType.isEntityReferenceListType) { - var listEntityType = fieldType.arguments0!; - - var fieldListEntityRepository = getEntityRepositoryByTypeInfo( - listEntityType, - ); - if (fieldListEntityRepository == null) { - throw StateError( - "Can't determine `EntityRepository` for field `$key` List type: fieldType=$fieldType", - ); - } - - if (value is EntityReferenceList) { - // Its IDs (from its entities, if only those are set): - value = value.isNull ? null : value.ids; - } else { - var valIter = value is Iterable ? value : [value]; - - value = valIter - .map( - (v) => fieldListEntityRepository.isOfEntityType(v) - ? fieldListEntityRepository.getEntityID(v) - : v, - ) - .toList(); - } - } else if (fieldType.isListEntity) { - var listEntityType = fieldType.listEntityType!; - - var fieldListEntityRepository = getEntityRepositoryByTypeInfo( - listEntityType, - ); - if (fieldListEntityRepository == null) { - throw StateError( - "Can't determine `EntityRepository` for field `$key` List type: fieldType=$fieldType", - ); - } - - var valIter = value is Iterable ? value : [value]; - - value = valIter - .map( - (v) => fieldListEntityRepository.isOfEntityType(v) - ? fieldListEntityRepository.getEntityID(v) - : v, - ) - .toList(); - } else if (!fieldType.isPrimitiveType && fieldType.entityType != null) { - var entityType = fieldType.entityType!; - var fieldEntityRepository = getEntityRepositoryByType(entityType); - - if (fieldEntityRepository == null) { - throw StateError( - "Can't determine `EntityRepository` for field `$key`: fieldType=$fieldType", - ); - } - - value = fieldEntityRepository.getEntityID(value); - } - } - - return MapEntry(key, value); - }); - - return entityJsonNormalized; - } - - Object resolveError( - Object error, - StackTrace stackTrace, - Object? operation, - Object? previousError, - ) => DBObjectDirectoryAdapterException( - 'error', - '$error', - parentError: error, - parentStackTrace: stackTrace, - previousError: previousError, - operation: operation, - ); - - FutureOr _finishOperation( - TransactionOperation op, - T res, - PreFinishDBOperation? preFinish, - ) => finishOperation(op, res, preFinish, errorResolver: resolveError); - - /// Executes [f] in the transaction of [op], and finishes [op] (with - /// [preFinish]) after that execution, not inside it: a [preFinish] can - /// issue other operations in the same transaction (e.g. selecting the - /// sub-entities), executed after this one (inside it, they would wait for - /// it, that waits for them). - FutureOr _executeOperation( - TransactionOperation op, - FutureOr Function(DBObjectDirectoryAdapterContext connection) f, - PreFinishDBOperation? preFinish, - ) => executeTransactionOperation( - op, - f, - ).resolveMapped((res) => _finishOperation(op, res, preFinish)); - - FutureOr executeTransactionOperation( - TransactionOperation op, - FutureOr Function(DBObjectDirectoryAdapterContext connection) f, - ) { - var transaction = op.transaction; - - if (isTransactionWithSingleOperation(op)) { - return executeWithPool( - f, - onError: (e, s) => transaction.notifyExecutionError( - e, - s, - errorResolver: resolveError, - operation: op, - debugInfo: () => op.toString(), - ), - ); - } - - if (transaction.isOpen) { - return transaction.addExecution( - (c) => f(c), - errorResolver: resolveError, - operation: op, - debugInfo: () => op.toString(), - ); - } - - if (!transaction.isOpening) { - transaction.open( - () => openTransaction(transaction), - callCloseTransactionRequired - ? () => closeTransaction( - transaction, - transaction.context as DBObjectDirectoryAdapterContext?, - ) - : null, - ); - } - - return transaction.onOpen(() { - return transaction.addExecution( - (c) => f(c), - errorResolver: resolveError, - operation: op, - debugInfo: () => op.toString(), - ); - }); - } } /// Error thrown by [DBObjectDirectoryAdapter] operations. diff --git a/lib/src/bones_api_entity_db_object_gcs.dart b/lib/src/bones_api_entity_db_object_gcs.dart index 3766ebec..3c9439cc 100644 --- a/lib/src/bones_api_entity_db_object_gcs.dart +++ b/lib/src/bones_api_entity_db_object_gcs.dart @@ -1,7 +1,6 @@ import 'dart:convert' as dart_convert; import 'dart:io'; import 'dart:math' as math; -import 'dart:typed_data'; import 'package:_discoveryapis_commons/_discoveryapis_commons.dart' show DetailedApiRequestError; @@ -13,16 +12,12 @@ import 'package:googleapis_auth/auth_io.dart' as auth; import 'package:http/http.dart' as http; import 'package:logging/logging.dart' as logging; import 'package:path/path.dart' as pack_path; -import 'package:reflection_factory/reflection_factory.dart'; import 'package:statistics/statistics.dart'; -import 'package:swiss_knife/swiss_knife.dart'; import 'bones_api_condition.dart'; -import 'bones_api_condition_encoder.dart'; import 'bones_api_entity.dart'; import 'bones_api_entity_db.dart'; import 'bones_api_entity_db_object.dart'; -import 'bones_api_entity_reference.dart'; import 'bones_api_extension.dart'; import 'bones_api_extension_io.dart'; import 'bones_api_logging.dart'; @@ -302,6 +297,14 @@ class DBObjectGCSAdapter extends DBObjectAdapter { int _connectionCount = 0; + /// The shared [DBObjectAdapter.openTransaction], typed as this adapter's + /// context: it opens synchronously ([createConnection] is synchronous), as + /// this adapter's own `openTransaction` was declared before the shared + /// implementation (a caller may rely on the concrete type). + @override + DBObjectGCSAdapterContext openTransaction(Transaction transaction) => + super.openTransaction(transaction) as DBObjectGCSAdapterContext; + @override DBObjectGCSAdapterContext createConnection() { var id = ++_connectionCount; @@ -324,138 +327,25 @@ class DBObjectGCSAdapter extends DBObjectAdapter { return info; } - final Map tablesSchemes = {}; - - void addTableSchemes(Iterable tablesSchemes) { - for (var s in tablesSchemes) { - this.tablesSchemes[s.name] = s; - } - } - - @override - TableScheme? getTableSchemeImpl( - String table, - TableRelationshipReference? relationship, { - Object? contextID, - }) { - //_log.info('getTableSchemeImpl> $table ; relationship: $relationship'); - - var tableScheme = tablesSchemes[table]; - if (tableScheme != null) return tableScheme; - - var entityHandler = getEntityHandler(tableName: table); - if (entityHandler == null) { - if (relationship != null) { - var sourceId = - '${relationship.sourceTable}_${relationship.sourceField}'; - var targetId = - '${relationship.targetTable}_${relationship.targetField}'; - - tableScheme = TableScheme( - table, - relationship: true, - idFieldName: sourceId, - fieldsTypes: { - sourceId: relationship.sourceFieldType, - targetId: relationship.targetFieldType, - }, - ); - - _log.info('relationship> $tableScheme'); - - return tableScheme; - } - - throw StateError( - "Can't resolve `TableScheme` for table `$table`. No `EntityHandler` found for table `$table`!", - ); - } - - var idFieldName = entityHandler.idFieldName(); - - var entityFieldsTypes = entityHandler.fieldsTypes(); - - var fieldsTypes = entityFieldsTypes.map( - (key, value) => MapEntry(key, value.type), - ); - - tableScheme = TableScheme( - table, - relationship: relationship != null, - idFieldName: idFieldName, - fieldsTypes: fieldsTypes, - ); - - _log.info('$tableScheme'); - - return tableScheme; - } - @override - FutureOr?> getTableFieldsTypesImpl(String table) { - return tablesSchemes[table]?.fieldsTypes; - } - - @override - FutureOr isConnectionValid(connection, {bool checkUsage = true}) => - true; - - final Map _openTransactionsContexts = - {}; - - @override - DBObjectGCSAdapterContext openTransaction(Transaction transaction) { - var conn = createConnection(); - - _openTransactionsContexts[conn] = DateTime.now(); - - transaction.transactionFuture - // ignore: discarded_futures - .then( - // ignore: discarded_futures - (res) => resolveTransactionResult(res, transaction, conn), - onError: (e, s) { - cancelTransaction(transaction, conn, e, s); - throw e; - }, - ); - - return conn; - } + logging.Logger get adapterLogger => _log; @override - bool get cancelTransactionResultWithError => true; - - @override - bool get throwTransactionResultWithError => false; - - @override - bool cancelTransaction( - Transaction transaction, - DBObjectGCSAdapterContext? connection, - Object? error, - StackTrace? stackTrace, - ) { - _openTransactionsContexts.remove(connection); - return true; - } - - @override - bool get callCloseTransactionRequired => true; - - @override - FutureOr closeTransaction( - Transaction transaction, - DBObjectGCSAdapterContext? connection, - ) { - if (connection != null) { - _consolidateTransactionContext(connection); - } - } - - void _consolidateTransactionContext(DBObjectGCSAdapterContext context) { - _openTransactionsContexts.remove(context); - } + DBObjectGCSAdapterException createAdapterException( + String type, + String message, { + Object? parentError, + StackTrace? parentStackTrace, + Object? operation, + Object? previousError, + }) => DBObjectGCSAdapterException( + type, + message, + parentError: parentError, + parentStackTrace: parentStackTrace, + operation: operation, + previousError: previousError, + ); @override String toString() { @@ -477,7 +367,7 @@ class DBObjectGCSAdapter extends DBObjectAdapter { Map? namedParameters, PreFinishDBOperation? preFinish, }) { - return _executeOperation( + return executeOperation( op, (conn) => _doCountImpl(op, table, matcher, parameters ?? namedParameters), preFinish, @@ -523,7 +413,7 @@ class DBObjectGCSAdapter extends DBObjectAdapter { String table, List ids, ) { - return _executeOperation, List>( + return executeOperation, List>( op, (conn) => _doExistIDsImpl(op, table, ids), null, @@ -555,7 +445,7 @@ class DBObjectGCSAdapter extends DBObjectAdapter { String table, Object id, { PreFinishDBOperation?, R?>? preFinish, - }) => _executeOperation?, R?>( + }) => executeOperation?, R?>( op, (conn) => _doSelectByIDImpl(table, id, entityName), preFinish, @@ -578,7 +468,7 @@ class DBObjectGCSAdapter extends DBObjectAdapter { bool? orderByID, OrderDirection? orderDirection, PreFinishDBOperation>, List>? preFinish, - }) => _executeOperation>, List>( + }) => executeOperation>, List>( op, (conn) => _doSelectByIDsImpl( table, @@ -605,7 +495,7 @@ class DBObjectGCSAdapter extends DBObjectAdapter { .map((id) => _readObject(table, id)) .resolveAllNotNull(); - return _applyOrderAndPagination( + return applyTableOrderAndPagination( table, entries, limit: limit, @@ -625,7 +515,7 @@ class DBObjectGCSAdapter extends DBObjectAdapter { bool? orderByID, OrderDirection? orderDirection, PreFinishDBOperation>, List>? preFinish, - }) => _executeOperation>, List>( + }) => executeOperation>, List>( op, (conn) => _doSelectAllImpl( table, @@ -654,7 +544,7 @@ class DBObjectGCSAdapter extends DBObjectAdapter { return _readObject(table, id); }).resolveAllNotNull(); - return _applyOrderAndPagination( + return applyTableOrderAndPagination( table, entries, limit: limit, @@ -664,26 +554,6 @@ class DBObjectGCSAdapter extends DBObjectAdapter { ); } - List> _applyOrderAndPagination( - String table, - Iterable> entries, { - int? limit, - int? offset, - bool? orderByID, - OrderDirection? orderDirection, - }) { - var idField = _getTableIDFieldName(table); - - return applySelectOrderAndPagination( - entries, - (e) => e[idField], - limit: limit, - offset: offset, - orderByID: orderByID, - orderDirection: orderDirection, - ).toList(); - } - @override FutureOr doDelete( TransactionOperation op, @@ -694,7 +564,7 @@ class DBObjectGCSAdapter extends DBObjectAdapter { List? positionalParameters, Map? namedParameters, PreFinishDBOperation>, R>? preFinish, - }) => _executeOperation>, R>( + }) => executeOperation>, R>( op, (conn) => _doDeleteImpl(op, table, matcher, parameters ?? namedParameters), @@ -718,13 +588,6 @@ class DBObjectGCSAdapter extends DBObjectAdapter { throw UnsupportedError("Relationship delete not supported for: $matcher"); } - String _getTableIDFieldName(String table) { - var tablesScheme = tablesSchemes[table]; - - var idField = tablesScheme?.idFieldName ?? 'id'; - return idField; - } - @override FutureOr doInsert( TransactionOperation op, @@ -734,7 +597,7 @@ class DBObjectGCSAdapter extends DBObjectAdapter { Map fields, { String? idFieldName, PreFinishDBOperation? preFinish, - }) => _executeOperation( + }) => executeOperation( op, (conn) => _doInsertImpl(op, table, fields, entityName), preFinish, @@ -746,13 +609,13 @@ class DBObjectGCSAdapter extends DBObjectAdapter { Map fields, String entityName, ) async { - var entry = _normalizeEntityJSON( + var entry = normalizeEntityJSON( fields, entityName: entityName, table: table, ); - var idField = _getTableIDFieldName(table); + var idField = getTableIDFieldName(table); var id = entry[idField]; _log.info( @@ -779,7 +642,7 @@ class DBObjectGCSAdapter extends DBObjectAdapter { String? idFieldName, PreFinishDBOperation? preFinish, bool allowAutoInsert = false, - }) => _executeOperation( + }) => executeOperation( op, (conn) => _doUpdateImpl(op, table, fields, entityName, id), preFinish, @@ -796,7 +659,7 @@ class DBObjectGCSAdapter extends DBObjectAdapter { '[transaction:${op.transactionId}] doUpdate> UPDATE INTO $table OBJECT `$id`', ); - var changes = _normalizeEntityJSON( + var changes = normalizeEntityJSON( fields, entityName: entityName, table: table, @@ -810,7 +673,7 @@ class DBObjectGCSAdapter extends DBObjectAdapter { var entry = {...?stored, ...changes}; - var idField = _getTableIDFieldName(table); + var idField = getTableIDFieldName(table); entry[idField] = id; await _saveObject(table, id, entry); @@ -1212,14 +1075,14 @@ class DBObjectGCSAdapter extends DBObjectAdapter { } String _resolveObjectFilePath(String table, Object? id) { - var idStr = _normalizeID(id); + var idStr = normalizeObjectID(id); var tablePath = _resolveTablePath(table); var file = '$tablePath/$idStr.json'; return file; } String _resolveTablePath(String table) { - var tableStr = _normalizeTableName(table); + var tableStr = normalizeTableName(table); return tableStr; } @@ -1235,7 +1098,7 @@ class DBObjectGCSAdapter extends DBObjectAdapter { autoCreateDir: autoCreateDir, ).resolveMapped((tableDir) { if (tableDir == null) return null; - var idStr = _normalizeID(id); + var idStr = normalizeObjectID(id); var file = File(pack_path.join(tableDir.path, '$idStr.json')); return file; }); @@ -1250,7 +1113,7 @@ class DBObjectGCSAdapter extends DBObjectAdapter { var cacheDirectory = this.cacheDirectory; if (cacheDirectory == null) return null; - var tableStr = _normalizeTableName(table); + var tableStr = normalizeTableName(table); var tableDir = Directory(pack_path.join(cacheDirectory.path, tableStr)); if (autoCreateDir) { @@ -1269,240 +1132,6 @@ class DBObjectGCSAdapter extends DBObjectAdapter { } return tableDir; } - - static final _regExpNormalizeIDInvalidChars = RegExp(r'[^\w.-]+'); - static final _regExpNormalizeIDInvalidPrefix = RegExp(r'^\.+'); - - String _normalizeID(Object? o) { - var id = o?.toString() ?? ''; - if (id.isEmpty) { - throw StateError("Empty ID string."); - } - - id = id - .replaceAll(_regExpNormalizeIDInvalidChars, '') - .replaceAll(_regExpNormalizeIDInvalidPrefix, ''); - id = id.truncate(220); - return id; - } - - static final _regExpNormalizeTableNonWord = RegExp(r'\W'); - - String _normalizeTableName(String table) { - table = table.trim().replaceAll(_regExpNormalizeTableNonWord, '_'); - if (table.isEmpty) { - throw StateError("Empty table name."); - } - return table; - } - - Map _normalizeEntityJSON( - Map entityJson, { - String? entityName, - String? table, - EntityRepository? entityRepository, - }) { - entityRepository ??= getEntityRepository( - name: entityName, - tableName: table, - ); - - if (entityRepository == null) { - throw StateError( - "Can't determine `EntityRepository` for: entityName=$entityName ; tableName=$table", - ); - } - - var entityHandler = entityRepository.entityHandler; - - var entityJsonNormalized = entityJson.map((key, value) { - if (value == null || (value as Object).isPrimitiveValue) { - return MapEntry(key, value); - } else if (value is Uint8List) { - var dataURLBase64 = DataURLBase64.from(value); - return MapEntry(key, dataURLBase64.toString()); - } - - var fieldType = entityHandler.getFieldType( - null, - key, - resolveFiledName: false, - ); - - if (fieldType != null) { - if (fieldType.isEntityReferenceType) { - var entityType = fieldType.arguments0!.type; - var fieldEntityRepository = getEntityRepositoryByType(entityType); - - if (fieldEntityRepository == null) { - throw StateError( - "Can't determine `EntityRepository` for field `$key`: fieldType=$fieldType", - ); - } - - if (value is EntityReference) { - if (value.isNull) { - value = null; - } else if (value.isIdSet) { - value = value.id; - } else if (value.isEntitySet) { - value = fieldEntityRepository.getEntityID(value.entity); - } else { - throw StateError("Invalid state: $value"); - } - } else if (!value.isPrimitiveValue) { - value = fieldEntityRepository.getEntityID(value); - } - } else if (fieldType.isEntityReferenceListType) { - var listEntityType = fieldType.arguments0!; - - var fieldListEntityRepository = getEntityRepositoryByTypeInfo( - listEntityType, - ); - if (fieldListEntityRepository == null) { - throw StateError( - "Can't determine `EntityRepository` for field `$key` List type: fieldType=$fieldType", - ); - } - - if (value is EntityReferenceList) { - // Its IDs (from its entities, if only those are set): - value = value.isNull ? null : value.ids; - } else { - var valIter = value is Iterable ? value : [value]; - - value = valIter - .map( - (v) => fieldListEntityRepository.isOfEntityType(v) - ? fieldListEntityRepository.getEntityID(v) - : v, - ) - .toList(); - } - } else if (fieldType.isListEntity) { - var listEntityType = fieldType.listEntityType!; - - var fieldListEntityRepository = getEntityRepositoryByTypeInfo( - listEntityType, - ); - if (fieldListEntityRepository == null) { - throw StateError( - "Can't determine `EntityRepository` for field `$key` List type: fieldType=$fieldType", - ); - } - - var valIter = value is Iterable ? value : [value]; - - value = valIter - .map( - (v) => fieldListEntityRepository.isOfEntityType(v) - ? fieldListEntityRepository.getEntityID(v) - : v, - ) - .toList(); - } else if (!fieldType.isPrimitiveType && fieldType.entityType != null) { - var entityType = fieldType.entityType!; - var fieldEntityRepository = getEntityRepositoryByType(entityType); - - if (fieldEntityRepository == null) { - throw StateError( - "Can't determine `EntityRepository` for field `$key`: fieldType=$fieldType", - ); - } - - value = fieldEntityRepository.getEntityID(value); - } - } - - return MapEntry(key, value); - }); - - return entityJsonNormalized; - } - - Object resolveError( - Object error, - StackTrace stackTrace, - Object? operation, - Object? previousError, - ) => DBObjectGCSAdapterException( - 'error', - '$error', - parentError: error, - parentStackTrace: stackTrace, - previousError: previousError, - operation: operation, - ); - - FutureOr _finishOperation( - TransactionOperation op, - T res, - PreFinishDBOperation? preFinish, - ) => finishOperation(op, res, preFinish, errorResolver: resolveError); - - /// Executes [f] in the transaction of [op], and finishes [op] (with - /// [preFinish]) after that execution, not inside it: a [preFinish] can - /// issue other operations in the same transaction (e.g. selecting the - /// sub-entities), executed after this one (inside it, they would wait for - /// it, that waits for them). - FutureOr _executeOperation( - TransactionOperation op, - FutureOr Function(DBObjectGCSAdapterContext connection) f, - PreFinishDBOperation? preFinish, - ) => executeTransactionOperation( - op, - f, - ).resolveMapped((res) => _finishOperation(op, res, preFinish)); - - FutureOr executeTransactionOperation( - TransactionOperation op, - FutureOr Function(DBObjectGCSAdapterContext connection) f, - ) { - var transaction = op.transaction; - - if (isTransactionWithSingleOperation(op)) { - return executeWithPool( - f, - onError: (e, s) => transaction.notifyExecutionError( - e, - s, - errorResolver: resolveError, - operation: op, - debugInfo: () => op.toString(), - ), - ); - } - - if (transaction.isOpen) { - return transaction.addExecution( - f, - errorResolver: resolveError, - operation: op, - debugInfo: () => op.toString(), - ); - } - - if (!transaction.isOpening) { - transaction.open( - () => openTransaction(transaction), - callCloseTransactionRequired - ? () => closeTransaction( - transaction, - transaction.context as DBObjectGCSAdapterContext?, - ) - : null, - ); - } - - return transaction.onOpen(() { - return transaction.addExecution( - f, - errorResolver: resolveError, - operation: op, - debugInfo: () => op.toString(), - ); - }); - } } /// Error thrown by [DBObjectGCSAdapter] operations. diff --git a/lib/src/bones_api_entity_db_object_memory.dart b/lib/src/bones_api_entity_db_object_memory.dart index 30bf44ab..488eff2b 100644 --- a/lib/src/bones_api_entity_db_object_memory.dart +++ b/lib/src/bones_api_entity_db_object_memory.dart @@ -7,11 +7,9 @@ import 'package:reflection_factory/reflection_factory.dart'; import 'package:statistics/statistics.dart'; import 'bones_api_condition.dart'; -import 'bones_api_condition_encoder.dart'; import 'bones_api_entity.dart'; import 'bones_api_entity_db.dart'; import 'bones_api_entity_db_object.dart'; -import 'bones_api_entity_reference.dart'; import 'bones_api_extension.dart'; import 'bones_api_logging.dart'; import 'bones_api_types.dart'; @@ -161,6 +159,14 @@ class DBObjectMemoryAdapter int _connectionCount = 0; + /// The shared [DBObjectAdapter.openTransaction], typed as this adapter's + /// context: it opens synchronously ([createConnection] is synchronous), as + /// this adapter's own `openTransaction` was declared before the shared + /// implementation (a caller may rely on the concrete type). + @override + DBObjectMemoryAdapterContext openTransaction(Transaction transaction) => + super.openTransaction(transaction) as DBObjectMemoryAdapterContext; + @override DBObjectMemoryAdapterContext createConnection() { var id = ++_connectionCount; @@ -235,146 +241,55 @@ class DBObjectMemoryAdapter return info; } - final Map tablesSchemes = {}; - - void addTableSchemes(Iterable tablesSchemes) { - for (var s in tablesSchemes) { - this.tablesSchemes[s.name] = s; - } - } - bool _hasTableScheme(String table) => tablesSchemes.containsKey(table); @override - TableScheme? getTableSchemeImpl( - String table, - TableRelationshipReference? relationship, { - Object? contextID, - }) { - //_log.info('getTableSchemeImpl> $table ; relationship: $relationship'); - - var tableScheme = tablesSchemes[table]; - if (tableScheme != null) return tableScheme; - - var entityHandler = getEntityHandler(tableName: table); - if (entityHandler == null) { - if (relationship != null) { - var sourceId = - '${relationship.sourceTable}_${relationship.sourceField}'; - var targetId = - '${relationship.targetTable}_${relationship.targetField}'; - - tableScheme = TableScheme( - table, - relationship: true, - idFieldName: sourceId, - fieldsTypes: { - sourceId: relationship.sourceFieldType, - targetId: relationship.targetFieldType, - }, - ); - - _log.info('relationship> $tableScheme'); - - return tableScheme; - } - - throw StateError( - "Can't resolve `TableScheme` for table `$table`. No `EntityHandler` found for table `$table`!", - ); - } - - var idFieldName = entityHandler.idFieldName(); - - var entityFieldsTypes = entityHandler.fieldsTypes(); - - var fieldsTypes = entityFieldsTypes.map( - (key, value) => MapEntry(key, value.type), - ); - - tableScheme = TableScheme( - table, - relationship: relationship != null, - idFieldName: idFieldName, - fieldsTypes: fieldsTypes, - ); - - _log.info('$tableScheme'); - - return tableScheme; - } - - @override - FutureOr?> getTableFieldsTypesImpl(String table) { - return tablesSchemes[table]?.fieldsTypes; - } + logging.Logger get adapterLogger => _log; @override - FutureOr isConnectionValid(connection, {bool checkUsage = true}) => - true; - - final Map _openTransactionsContexts = - {}; - - @override - DBObjectMemoryAdapterContext openTransaction(Transaction transaction) { - var conn = createConnection(); - - _openTransactionsContexts[conn] = DateTime.now(); - - transaction.transactionFuture - // ignore: discarded_futures - .then( - // ignore: discarded_futures - (res) => resolveTransactionResult(res, transaction, conn), - onError: (e, s) { - cancelTransaction(transaction, conn, e, s); - throw e; - }, - ); - - return conn; - } - - @override - bool get cancelTransactionResultWithError => true; + DBObjectMemoryAdapterException createAdapterException( + String type, + String message, { + Object? parentError, + StackTrace? parentStackTrace, + Object? operation, + Object? previousError, + }) => DBObjectMemoryAdapterException( + type, + message, + parentError: parentError, + parentStackTrace: parentStackTrace, + operation: operation, + previousError: previousError, + ); + /// The objects are kept in memory (not encoded as JSON): a bytes field is + /// stored as it is. @override - bool get throwTransactionResultWithError => false; + bool get encodeBytesAsDataURL => false; + /// A canceled transaction rolls back its changes of the tables. @override - bool cancelTransaction( - Transaction transaction, - DBObjectMemoryAdapterContext? connection, - Object? error, - StackTrace? stackTrace, - ) { - if (connection == null) return true; - _openTransactionsContexts.remove(connection); - _rollbackTables(connection.tablesVersions); - return true; - } - - @override - bool get callCloseTransactionRequired => true; - - @override - FutureOr closeTransaction( - Transaction transaction, - DBObjectMemoryAdapterContext? connection, - ) { - if (connection != null) { - _consolidateTransactionContext(connection); - } + void onTransactionContextCanceled(DBObjectMemoryAdapterContext context) { + _rollbackTables(context.tablesVersions); } + /// The closed transactions' contexts waiting for the other open + /// transactions to close, to consolidate their tables' changes together + /// (in context order). See [onTransactionContextClosed]. final ListQueue _consolidateContextQueue = ListQueue(); - void _consolidateTransactionContext(DBObjectMemoryAdapterContext context) { - _openTransactionsContexts.remove(context); - - if (_openTransactionsContexts.isNotEmpty) { + /// Consolidates the tables' changes of the closed [context]: only when no + /// other transaction is open (a consolidation would also make the open + /// ones' changes permanent); otherwise [context] waits in + /// [_consolidateContextQueue]. + @override + void onTransactionContextClosed( + DBObjectMemoryAdapterContext context, { + required bool otherTransactionsOpen, + }) { + if (otherTransactionsOpen) { _consolidateContextQueue.add(context); return; } @@ -436,7 +351,7 @@ class DBObjectMemoryAdapter Map? namedParameters, PreFinishDBOperation? preFinish, }) { - return _executeOperation( + return executeOperation( op, (conn) => _doCountImpl(op, table, matcher, parameters ?? namedParameters), preFinish, @@ -471,7 +386,7 @@ class DBObjectMemoryAdapter String table, List ids, ) { - return _executeOperation, List>( + return executeOperation, List>( op, (conn) => _doExistIDsImpl(op, table, ids), null, @@ -498,7 +413,7 @@ class DBObjectMemoryAdapter String table, Object id, { PreFinishDBOperation?, R?>? preFinish, - }) => _executeOperation?, R?>( + }) => executeOperation?, R?>( op, (conn) => _doSelectByIDImpl(table, id, entityName), preFinish, @@ -532,7 +447,7 @@ class DBObjectMemoryAdapter bool? orderByID, OrderDirection? orderDirection, PreFinishDBOperation>, List>? preFinish, - }) => _executeOperation>, List>( + }) => executeOperation>, List>( op, (conn) => _doSelectByIDsImpl( table, @@ -571,7 +486,7 @@ class DBObjectMemoryAdapter .nonNulls .toList(); - return _applyOrderAndPagination( + return applyTableOrderAndPagination( table, entries, limit: limit, @@ -591,7 +506,7 @@ class DBObjectMemoryAdapter bool? orderByID, OrderDirection? orderDirection, PreFinishDBOperation>, List>? preFinish, - }) => _executeOperation>, List>( + }) => executeOperation>, List>( op, (conn) => _doSelectAllImpl( table, @@ -615,7 +530,7 @@ class DBObjectMemoryAdapter var map = _getTableMap(table, false); if (map == null) return []; - return _applyOrderAndPagination( + return applyTableOrderAndPagination( table, map.values, limit: limit, @@ -625,26 +540,6 @@ class DBObjectMemoryAdapter ); } - List> _applyOrderAndPagination( - String table, - Iterable> entries, { - int? limit, - int? offset, - bool? orderByID, - OrderDirection? orderDirection, - }) { - var idField = _getTableIDFieldName(table); - - return applySelectOrderAndPagination( - entries, - (e) => e[idField], - limit: limit, - offset: offset, - orderByID: orderByID, - orderDirection: orderDirection, - ).toList(); - } - @override FutureOr doDelete( TransactionOperation op, @@ -655,7 +550,7 @@ class DBObjectMemoryAdapter List? positionalParameters, Map? namedParameters, PreFinishDBOperation>, R>? preFinish, - }) => _executeOperation>, R>( + }) => executeOperation>, R>( op, (conn) => _doDeleteImpl(op, table, matcher, parameters ?? namedParameters), @@ -680,13 +575,6 @@ class DBObjectMemoryAdapter throw UnsupportedError("Relationship delete not supported for: $matcher"); } - String _getTableIDFieldName(String table) { - var tablesScheme = tablesSchemes[table]; - - var idField = tablesScheme?.idFieldName ?? 'id'; - return idField; - } - @override FutureOr doInsert( TransactionOperation op, @@ -696,7 +584,7 @@ class DBObjectMemoryAdapter Map fields, { String? idFieldName, PreFinishDBOperation? preFinish, - }) => _executeOperation( + }) => executeOperation( op, (conn) => _doInsertImpl(op, table, fields, entityName), preFinish, @@ -710,13 +598,13 @@ class DBObjectMemoryAdapter ) { var map = _getTableMap(table, true)!; - var entry = _normalizeEntityJSON( + var entry = normalizeEntityJSON( fields, entityName: entityName, table: table, ); - var idField = _getTableIDFieldName(table); + var idField = getTableIDFieldName(table); var id = entry[idField]; _logTransactionOperation('doInsert', op, 'INSERT INTO $table OBJECT `$id`'); @@ -751,7 +639,7 @@ class DBObjectMemoryAdapter String? idFieldName, PreFinishDBOperation? preFinish, bool allowAutoInsert = false, - }) => _executeOperation( + }) => executeOperation( op, (conn) => _doUpdateImpl(op, table, fields, entityName, id), preFinish, @@ -768,7 +656,7 @@ class DBObjectMemoryAdapter var map = _getTableMap(table, true)!; - var changes = _normalizeEntityJSON( + var changes = normalizeEntityJSON( fields, entityName: entityName, table: table, @@ -778,218 +666,13 @@ class DBObjectMemoryAdapter // entry (if any: an auto-insert). var entry = {...?map[id], ...changes}; - var idField = _getTableIDFieldName(table); + var idField = getTableIDFieldName(table); entry[idField] = id; map[id] = entry; return id; } - - Map _normalizeEntityJSON( - Map entityJson, { - String? entityName, - String? table, - EntityRepository? entityRepository, - }) { - entityRepository ??= getEntityRepository( - name: entityName, - tableName: table, - ); - - if (entityRepository == null) { - throw StateError( - "Can't determine `EntityRepository` for: entityName=$entityName ; tableName=$table", - ); - } - - var entityHandler = entityRepository.entityHandler; - - var entityJsonNormalized = entityJson.map((key, value) { - if (value == null || (value as Object).isPrimitiveValue) { - return MapEntry(key, value); - } - - var fieldType = entityHandler.getFieldType( - null, - key, - resolveFiledName: false, - ); - - if (fieldType != null) { - if (fieldType.isEntityReferenceType) { - var entityType = fieldType.arguments0!.type; - var fieldEntityRepository = getEntityRepositoryByType(entityType); - - if (fieldEntityRepository == null) { - throw StateError( - "Can't determine `EntityRepository` for field `$key`: fieldType=$fieldType", - ); - } - - if (value is EntityReference) { - if (value.isNull) { - value = null; - } else if (value.isIdSet) { - value = value.id; - } else if (value.isEntitySet) { - value = fieldEntityRepository.getEntityID(value.entity); - } else { - throw StateError("Invalid state: $value"); - } - } else if (!value.isPrimitiveValue) { - value = fieldEntityRepository.getEntityID(value); - } - } else if (fieldType.isEntityReferenceListType) { - var listEntityType = fieldType.arguments0!; - - var fieldListEntityRepository = getEntityRepositoryByTypeInfo( - listEntityType, - ); - if (fieldListEntityRepository == null) { - throw StateError( - "Can't determine `EntityRepository` for field `$key` List type: fieldType=$fieldType", - ); - } - - if (value is EntityReferenceList) { - // Its IDs (from its entities, if only those are set): - value = value.isNull ? null : value.ids; - } else { - var valIter = value is Iterable ? value : [value]; - - value = valIter - .map( - (v) => fieldListEntityRepository.isOfEntityType(v) - ? fieldListEntityRepository.getEntityID(v) - : v, - ) - .toList(); - } - } else if (fieldType.isListEntity) { - var listEntityType = fieldType.listEntityType!; - - var fieldListEntityRepository = getEntityRepositoryByTypeInfo( - listEntityType, - ); - if (fieldListEntityRepository == null) { - throw StateError( - "Can't determine `EntityRepository` for field `$key` List type: fieldType=$fieldType", - ); - } - - var valIter = value is Iterable ? value : [value]; - - value = valIter - .map( - (v) => fieldListEntityRepository.isOfEntityType(v) - ? fieldListEntityRepository.getEntityID(v) - : v, - ) - .toList(); - } else if (!fieldType.isPrimitiveType && fieldType.entityType != null) { - var entityType = fieldType.entityType!; - var fieldEntityRepository = getEntityRepositoryByType(entityType); - - if (fieldEntityRepository == null) { - throw StateError( - "Can't determine `EntityRepository` for field `$key`: fieldType=$fieldType", - ); - } - - value = fieldEntityRepository.getEntityID(value); - } - } - - return MapEntry(key, value); - }); - - return entityJsonNormalized; - } - - Object resolveError( - Object error, - StackTrace stackTrace, - Object? operation, - Object? previousError, - ) => DBObjectMemoryAdapterException( - 'error', - '$error', - parentError: error, - parentStackTrace: stackTrace, - previousError: previousError, - operation: operation, - ); - - FutureOr _finishOperation( - TransactionOperation op, - T res, - PreFinishDBOperation? preFinish, - ) => finishOperation(op, res, preFinish, errorResolver: resolveError); - - /// Executes [f] in the transaction of [op], and finishes [op] (with - /// [preFinish]) after that execution, not inside it: a [preFinish] can - /// issue other operations in the same transaction (e.g. selecting the - /// sub-entities), executed after this one (inside it, they would wait for - /// it, that waits for them). - FutureOr _executeOperation( - TransactionOperation op, - FutureOr Function(DBObjectMemoryAdapterContext connection) f, - PreFinishDBOperation? preFinish, - ) => executeTransactionOperation( - op, - f, - ).resolveMapped((res) => _finishOperation(op, res, preFinish)); - - FutureOr executeTransactionOperation( - TransactionOperation op, - FutureOr Function(DBObjectMemoryAdapterContext connection) f, - ) { - var transaction = op.transaction; - - if (isTransactionWithSingleOperation(op)) { - return executeWithPool( - f, - onError: (e, s) => transaction.notifyExecutionError( - e, - s, - errorResolver: resolveError, - operation: op, - debugInfo: () => op.toString(), - ), - ); - } - - if (transaction.isOpen) { - return transaction.addExecution( - f, - errorResolver: resolveError, - operation: op, - debugInfo: () => op.toString(), - ); - } - - if (!transaction.isOpening) { - transaction.open( - () => openTransaction(transaction), - callCloseTransactionRequired - ? () => closeTransaction( - transaction, - transaction.context as DBObjectMemoryAdapterContext?, - ) - : null, - ); - } - - return transaction.onOpen(() { - return transaction.addExecution( - f, - errorResolver: resolveError, - operation: op, - debugInfo: () => op.toString(), - ); - }); - } } @Deprecated("Renamed to 'DBObjectMemoryAdapterException'") diff --git a/lib/src/bones_api_entity_db_postgres.dart b/lib/src/bones_api_entity_db_postgres.dart index b622f07e..bc107f8f 100644 --- a/lib/src/bones_api_entity_db_postgres.dart +++ b/lib/src/bones_api_entity_db_postgres.dart @@ -1,6 +1,7 @@ import 'package:async_extension/async_extension.dart'; import 'package:collection/collection.dart'; import 'package:logging/logging.dart' as logging; +import 'package:meta/meta.dart'; import 'package:postgres/postgres.dart' hide Time, Type; import 'package:reflection_factory/reflection_factory.dart'; import 'package:statistics/statistics.dart'; @@ -18,7 +19,6 @@ import 'bones_api_logging.dart'; import 'bones_api_sql_builder.dart'; import 'bones_api_types.dart'; import 'bones_api_utils.dart'; -import 'bones_api_utils_timedmap.dart'; final _log = logging.Logger('DBPostgreSQLAdapter')..registerAsDbLogger(); @@ -647,7 +647,7 @@ class DBPostgreSQLAdapter extends DBSQLAdapter _log.info('getTableFieldsTypesImpl> $table'); var sql = - "SELECT column_name, data_type, column_default, is_updatable FROM information_schema.columns WHERE table_name = '$table'"; + "SELECT column_name, data_type, column_default, is_updatable FROM information_schema.columns WHERE table_name = ${DBSQLAdapter.sqlStringLiteral(table)}"; var scheme = await connection.mappedResultsQuery(sql); @@ -682,7 +682,7 @@ class DBPostgreSQLAdapter extends DBSQLAdapter //_log.info('getTableSchemeImpl> $table ; relationship: $relationship'); var sql = - "SELECT column_name, data_type, column_default, is_updatable FROM information_schema.columns WHERE table_name = '$table'"; + "SELECT column_name, data_type, column_default, is_updatable FROM information_schema.columns WHERE table_name = ${DBSQLAdapter.sqlStringLiteral(table)}"; var scheme = await connection.mappedResultsQuery(sql); @@ -703,16 +703,15 @@ class DBPostgreSQLAdapter extends DBSQLAdapter notifyTableFieldTypes(table, fieldsTypes); - var fieldsReferencedTables = await _findFieldsReferencedTables( + var fieldsReferencedTables = await findFieldsReferencedTablesCached( connection, table, contextID: contextID, ); - var relationshipTables = await _findRelationshipTables( + var relationshipTables = await findRelationshipTables( connection, table, - idFieldName, contextID: contextID, ); @@ -802,7 +801,7 @@ class DBPostgreSQLAdapter extends DBSQLAdapter ON true JOIN pg_attribute a ON a.attrelid = t.oid AND a.attnum = k.attnum WHERE - t.relname = '$table' + t.relname = ${DBSQLAdapter.sqlStringLiteral(table)} AND t.relkind IN ('r', 'p') AND pg_table_is_visible(t.oid) AND ix.indisvalid @@ -844,9 +843,7 @@ class DBPostgreSQLAdapter extends DBSQLAdapter String table, Iterable indexesNames, ) { - var names = indexesNames - .map((n) => "'${n.replaceAll("'", "''")}'") - .join(', '); + var names = indexesNames.map(DBSQLAdapter.sqlStringLiteral).join(', '); if (names.isEmpty) return {}; var sql = @@ -897,7 +894,7 @@ class DBPostgreSQLAdapter extends DBSQLAdapter FROM pg_constraint con JOIN pg_class tbl ON con.conrelid = tbl.oid WHERE - tbl.relname = '$table' + tbl.relname = ${DBSQLAdapter.sqlStringLiteral(table)} AND pg_table_is_visible(tbl.oid); '''; @@ -1002,7 +999,7 @@ class DBPostgreSQLAdapter extends DBSQLAdapter JOIN information_schema.columns AS c ON c.table_schema = tc.constraint_schema AND tc.table_name = c.table_name AND ccu.column_name = c.column_name WHERE - constraint_type = 'PRIMARY KEY' and tc.table_name = '$table'; + constraint_type = 'PRIMARY KEY' and tc.table_name = ${DBSQLAdapter.sqlStringLiteral(table)}; '''; var columns = await connection.mappedResultsQuery(sql); @@ -1075,95 +1072,8 @@ class DBPostgreSQLAdapter extends DBSQLAdapter } } - Future> _findRelationshipTables( - PostgreSQLConnectionWrapper connection, - String table, - String idFieldName, { - Object? contextID, - }) async { - final tablesReferences = await _findAllTableFieldsReferences( - connection, - table, - contextID: contextID, - ); - - var relationships = tablesReferences - .map((e) { - var refToTables = e.values - .where((r) => r.targetTable == table) - .toList(growable: false); - var otherRefs = e.values - .where((r) => r.targetTable != table) - .toList(growable: false); - - if (refToTables.length != 1 || otherRefs.length != 1) { - return null; - } - - var refToTable = refToTables.single; - var otherRef = otherRefs.single; - - var tableRelationshipReference = TableRelationshipReference( - refToTable.sourceTable, - refToTable.targetTable, - refToTable.targetField, - refToTable.targetFieldType, - refToTable.sourceField, - otherRef.targetTable, - otherRef.targetField, - otherRef.targetFieldType, - otherRef.sourceField, - sourceRelationshipFieldIndex: refToTable.indexName, - targetRelationshipFieldIndex: otherRef.indexName, - ); - - return tableRelationshipReference; - }) - .nonNulls - .toList(); - - return relationships; - } - - FutureOr>> - _findAllTableFieldsReferences( - PostgreSQLConnectionWrapper connection, - String table, { - Object? contextID, - }) async { - var tablesNames = await _listTablesNames(connection, contextID: contextID); - - var tablesReferences = >[]; - - for (var t in tablesNames) { - var refs = await _findFieldsReferencedTables( - connection, - t, - contextID: contextID, - ); - tablesReferences.add(refs); - } - - tablesReferences = tablesReferences.where((m) { - return m.length > 1 && - m.values.where((r) => r.targetTable == table).isNotEmpty && - m.values.where((r) => r.targetTable != table).isNotEmpty; - }).toList(); - return tablesReferences; - } - - final Expando>> _listTablesNamesContextCache = - Expando(); - - FutureOr> _listTablesNames( - PostgreSQLConnectionWrapper connection, { - Object? contextID, - }) => _listTablesNamesContextCache.putIfAbsentAsync( - contextID, - () => _listTablesNamesImpl(connection), - ); - - Future> _listTablesNamesImpl( + @override + Future> listTablesNamesImpl( PostgreSQLConnectionWrapper connection, ) async { var sql = ''' @@ -1177,55 +1087,12 @@ class DBPostgreSQLAdapter extends DBSQLAdapter return names; } - final TimedMap> - _findFieldsReferencedTablesCache = - TimedMap>(Duration(seconds: 30)); - - @override - TableScheme? disposeTableSchemeCache( - String table, { - bool fieldsTypes = false, - }) { - _findFieldsReferencedTablesCache.remove(table); - return super.disposeTableSchemeCache(table, fieldsTypes: fieldsTypes); - } - - @override - void disposeTablesSchemesCaches({bool fieldsTypes = false}) { - _findFieldsReferencedTablesCache.clear(); - super.disposeTablesSchemesCaches(fieldsTypes: fieldsTypes); - } - - final Expando>>> - _findFieldsReferencedTablesContextCache = Expando(); - - FutureOr> _findFieldsReferencedTables( - PostgreSQLConnectionWrapper connection, - String table, { - Object? contextID, - }) { - if (contextID != null) { - var cache = _findFieldsReferencedTablesContextCache[contextID] ??= {}; - return cache[table] ??= _findFieldsReferencedTablesImpl(connection, table) - .then((ret) { - cache[table] = ret; - _findFieldsReferencedTablesCache[table] = ret; - return ret; - }); - } - - return _findFieldsReferencedTablesCache.putIfAbsentCheckedAsync( - table, - () => _findFieldsReferencedTablesImpl(connection, table), - ); - } - - Future> _findFieldsReferencedTablesImpl( - PostgreSQLConnectionWrapper connection, - String table, - ) async { - var sql = - ''' + /// The SQL of [findFieldsReferencedTablesImpl]: the foreign keys of + /// [table] (escaped as a string literal, see + /// [DBSQLAdapter.sqlStringLiteral]). + @visibleForTesting + static String findFieldsReferencedTablesSQL(String table) => + ''' SELECT o.conname AS constraint_name, @@ -1280,12 +1147,19 @@ class DBPostgreSQLAdapter extends DBSQLAdapter WHERE o.contype = 'f' - AND m.relname = '$table' + AND m.relname = ${DBSQLAdapter.sqlStringLiteral(table)} AND o.conrelid IN ( SELECT oid FROM pg_class c WHERE c.relkind = 'r' ); '''; + @override + Future> findFieldsReferencedTablesImpl( + PostgreSQLConnectionWrapper connection, + String table, + ) async { + var sql = findFieldsReferencedTablesSQL(table); + var referenceFields = await connection.mappedResultsQuery(sql); var map = Map.fromEntries( @@ -1437,20 +1311,23 @@ class DBPostgreSQLAdapter extends DBSQLAdapter sql.sql, substitutionValues: sql.parametersByPlaceholder, ) - .resolveMapped((results) { - var count = results - .map((row) { - var count = row['count'] ?? 0; - return count is int - ? count - : int.tryParse(count.toString().trim()); - }) - .whereType() - .first; - return count; - }); + .resolveMapped(countFromResults); } + /// The count of a `COUNT` query's [results]: the first row's parseable + /// `count` column, or `0` without one (no row, as the SQLite and MySQL + /// adapters; a `.first` used to throw a `StateError` here). + @visibleForTesting + static int countFromResults(Iterable> results) => + results + .map((row) { + var count = row['count'] ?? 0; + return count is int ? count : int.tryParse(count.toString().trim()); + }) + .whereType() + .firstOrNull ?? + 0; + @override FutureOr> doExistIDsSQL( String entityName, @@ -1627,14 +1504,26 @@ class DBPostgreSQLAdapter extends DBSQLAdapter return lastInsertResult; } - var fixSql = - "SELECT setval(pg_get_serial_sequence('$table', '$idFieldName'), coalesce(max(id),0) + 1, false) FROM \"$table\""; + var fixSql = fixTableSequenceSQL(table, idFieldName); _log.info("Fixing table PRIMARY KEY sequence: <$fixSql>"); return connection.query(fixSql).then((r) => lastInsertResult); } + /// The SQL that moves the ID sequence of [table] after its greatest ID + /// (after an auto-insert with an explicit ID, see `_fixeTableSequence`). + /// + /// The greatest ID is of the column [idFieldName] (quoted, as the table), + /// the column of the sequence: not a literal `id` column, that a table + /// with another ID column doesn't have. + @visibleForTesting + static String fixTableSequenceSQL(String table, String idFieldName) => + "SELECT setval(pg_get_serial_sequence(" + "${DBSQLAdapter.sqlStringLiteral(table)}, " + "${DBSQLAdapter.sqlStringLiteral(idFieldName)}), " + "coalesce(max(\"$idFieldName\"),0) + 1, false) FROM \"$table\""; + dynamic _resolveResultID( List> results, String table, @@ -1671,13 +1560,11 @@ class DBPostgreSQLAdapter extends DBSQLAdapter @override Future openTransaction( Transaction transaction, - ) { - var contextCompleter = Completer(); - - var result = executeWithPool( - (connection) { - return connection.openTransaction((connTransaction) { - contextCompleter.complete(connTransaction); + ) => openTransactionWithPool( + transaction, + (connection, onTransactionContext) => + connection.openTransaction((connTransaction) { + onTransactionContext(connTransaction); return transaction.transactionFuture.then( (res) => resolveTransactionResult(res, transaction, connection), @@ -1686,28 +1573,11 @@ class DBPostgreSQLAdapter extends DBSQLAdapter throw e; }, ); - }); - }, - validator: (c) => !transaction.isAborted, - onError: (e, s) => transaction.notifyExecutionError( - e, - s, - errorResolver: resolveError, - debugInfo: () => transaction.toString(withExecutedOperations: false), - ), - ); - - transaction.transactionResult = result; - - return contextCompleter.future; - } - - @override - bool get cancelTransactionResultWithError => false; - - @override - bool get throwTransactionResultWithError => true; + }), + ); + /// Cancels the driver's transaction (see + /// [PostgreSQLConnectionTransactionWrapper.cancelTransaction]). @override bool cancelTransaction( Transaction transaction, @@ -1720,15 +1590,6 @@ class DBPostgreSQLAdapter extends DBSQLAdapter return true; } - @override - bool get callCloseTransactionRequired => false; - - @override - FutureOr closeTransaction( - Transaction transaction, - PostgreSQLConnectionWrapper? connection, - ) {} - @override String toString() { var closedStr = isClosed ? ', closed' : ''; diff --git a/lib/src/bones_api_entity_db_relational.dart b/lib/src/bones_api_entity_db_relational.dart index 60ce9d40..7e639b13 100644 --- a/lib/src/bones_api_entity_db_relational.dart +++ b/lib/src/bones_api_entity_db_relational.dart @@ -320,71 +320,35 @@ class DBRelationalEntityRepository super.type, }); + /// An already stored entity also has its referenced entities ensured + /// stored (see [ensureReferencesStored]). @override - FutureOr ensureStored( - o, { + FutureOr onEnsureStoredExisting( + O o, + Object id, Transaction? transaction, - TransactionOperation? operation, - }) { - checkNotClosed(); - - var id = getID(o, entityHandler: entityHandler); + ) => ensureReferencesStored(o, transaction: transaction).resolveWithValue(id); - if (id == null || entityHasChangedFields(o)) { - return _ensureStoredImpl(o, transaction, operation); - } else { - if (isTrackingEntity(o)) { - return ensureReferencesStored( - o, - transaction: transaction, - ).resolveWithValue(id); - } - - return existsID(id, transaction: transaction).resolveMapped((exists) { - if (!exists) { - return _ensureStoredImpl(o, transaction, operation); - } else { - return ensureReferencesStored( - o, - transaction: transaction, - ).resolveWithValue(id); - } - }); - } - } - - FutureOr _ensureStoredImpl( - dynamic o, + @override + FutureOr storeWithParentOperation( + O o, Transaction? transaction, TransactionOperation? parentOperation, - ) { - if (transaction != null) { - var storeOp = transaction - .firstOperationWithEntity(o); - - if (storeOp != null) { - return storeOp.waitFinish(parentOperation: parentOperation).then((ok) { - var id = getEntityID(storeOp.entity) ?? getEntityID(o); - if (id == null && !ok) { - throw RecursiveRelationshipLoopError.fromTransaction( - transaction, - storeOp, - parentOperation, - o, - ); - } - return id; - }); - } - } - - return _storeImpl(o, transaction, parentOperation); - } - - Map? _nonPrimitiveFields; - + ) => _storeImpl(o, transaction, parentOperation); + + /// The non-primitive fields of the entities of this repository (the ones + /// that may reference other entities), by field name, with their types. + /// + /// Built from the first entity's `fieldsNames` and reused for the whole + /// repository (all its entities have the same fields), so + /// [ensureReferencesStored] doesn't resolve the fields types of each + /// entity. + Map? _nonPrimitiveFieldsCache; + + /// The non-primitive fields (see [_nonPrimitiveFieldsCache]). Used by + /// [ensureReferencesStored]. Map _getNonPrimitiveFields(O o) => - _nonPrimitiveFields ??= entityHandler + _nonPrimitiveFieldsCache ??= entityHandler .fieldsNames(o) .map((fieldName) { var fieldType = entityHandler.getFieldType( @@ -558,18 +522,10 @@ class DBRelationalEntityRepository offset = resolveSelectOffset(page: page, offset: offset, limit: limit); - final resolutionRulesResolved = resolveEntityResolutionRules( - resolutionRules, - ); - - var canPropagate = hasReferencedEntities(resolutionRulesResolved); - - var op = TransactionOperationSelect( - name, - canPropagate, - operationExecutor, + final (op, resolutionRulesResolved) = newSelectOperation( + transaction, matcher, - transaction: transaction, + resolutionRules, ); try { diff --git a/lib/src/bones_api_entity_db_sql.dart b/lib/src/bones_api_entity_db_sql.dart index d2f54cf7..7aa0bdb3 100644 --- a/lib/src/bones_api_entity_db_sql.dart +++ b/lib/src/bones_api_entity_db_sql.dart @@ -1,6 +1,7 @@ import 'package:async_extension/async_extension.dart'; import 'package:collection/collection.dart'; import 'package:logging/logging.dart' as logging; +import 'package:meta/meta.dart'; import 'package:reflection_factory/reflection_factory.dart'; import 'package:statistics/statistics.dart'; import 'package:swiss_knife/swiss_knife.dart' show parseBool; @@ -23,6 +24,7 @@ import 'bones_api_sql_builder.dart'; import 'bones_api_types.dart'; import 'bones_api_utils_collections.dart'; import 'bones_api_utils_json.dart'; +import 'bones_api_utils_timedmap.dart'; export 'bones_api_entity_db.dart' show DBAdapterCapabilityConnectivity; @@ -192,6 +194,14 @@ class SQL implements SQLWrapper { static final RegExp _defaultPlaceholderRegexp = RegExp(r'@(\w+)'); + // The lazily computed [sqlPositional], [parametersKeysByPosition] and + // [parametersValuesByPosition] (`null`: not computed yet), filled together + // by [_computeSQLPositional] on the first access of any of them: only an + // adapter with positional placeholders (`?`) needs them. They can also be + // given to the constructor (a [copy]). Not named `*Cache` because they're + // also private named constructor parameters (`sqlPositional:` ...), whose + // public names would change. + String? _sqlPositional; List? _parametersKeysByPosition; @@ -290,6 +300,17 @@ class SQL implements SQLWrapper { return _parametersValuesByPosition!; } + /// A cache of [sqlPositional] and [parametersKeysByPosition] by SQL text + /// (default placeholder only), filled by [_computeSQLPositional]. + /// + /// Why: the same statements repeat, and converting their placeholders is a + /// regexp replace per statement: only the values are resolved per + /// instance. Cleared when it reaches [_positionalCacheLimit] (e.g. SQLs + /// with inlined values, all different). + static final Map)> _positionalCache = {}; + + static const int _positionalCacheLimit = 2000; + void _computeSQLPositional() { var keys = []; var values = []; @@ -339,13 +360,6 @@ class SQL implements SQLWrapper { _parametersValuesByPosition ??= values; } - /// [sqlPositional] and [parametersKeysByPosition] by SQL text (default - /// placeholder only). Cleared when it reaches [_positionalCacheLimit] - /// (e.g. SQLs with inlined values, all different). - static final Map)> _positionalCache = {}; - - static const int _positionalCacheLimit = 2000; - @override String toString() { var s = StringBuffer(); @@ -1242,13 +1256,10 @@ abstract class DBSQLAdapter extends DBRelationalAdapter var repoType = repository.type; if (scheme == null) { - _log.severe( - 'No scheme for repository `${repository.name}` (`$repoType`) in this adapter> generateTables: $_generateTables @ $this', - ); - - return _DBTableCheck.error( - 'No scheme for repository `${repository.name}` (`$repoType`) in this adapter> generateTables: $_generateTables @ $this', - ); + var errorMsg = + 'No scheme for repository `${repository.name}` (`$repoType`) in this adapter> generateTables: $_generateTables @ $this'; + _log.severe(errorMsg); + return _DBTableCheck.error(errorMsg); } var schemeTableName = scheme.name; @@ -1389,108 +1400,16 @@ abstract class DBSQLAdapter extends DBRelationalAdapter if (capability.constraintSupport) { // REF CONSTRAINTS // - var fieldEntityTypes = entityHandler - .getFieldsEntityTypes() - .entries - .map((e) { - var entityType = e.value.entityType; - if (entityType == null) return null; - - var refRepository = repository.provider.getEntityRepositoryByType( - entityType, - ); - - if (refRepository == null || - !refRepository.isSameEntityManager(repository)) { - return null; - } - - return MapEntry(e.key, entityType); - }) - .nonNulls - .toMapFromEntries(); - - if (fieldEntityTypes.isNotEmpty) { - var fieldsReferencedTables = scheme.fieldsReferencedTables; - - var fieldsReferences = fieldsReferencedTables.entries - .map((e) { - var colum = e.key; - var field = schemeToRepoMap[colum] ?? colum; - - var entityType = fieldEntityTypes[field]; - if (entityType == null) return null; - - var refTable = e.value.targetTable; - return MapEntry(field, refTable); - }) - .nonNulls - .toMapFromEntries(); - - missingFieldReferenceConstraints = fieldEntityTypes.keys - .where((f) => !fieldsReferences.containsKey(f)) - .toSet(); - - if (!dialect.foreignKeyCreatesImplicitIndex) { - missingFieldReferenceIndexes = fieldsReferencedTables.entries - .where((e) => e.value.indexName == null) - .map((e) => e.key) - .toSet(); - } - } + (missingFieldReferenceConstraints, missingFieldReferenceIndexes) = + _findMissingReferenceConstraints(repository, scheme, schemeToRepoMap); // RELATIONSHIP CONSTRAINTS // - var fieldListEntityTypes = entityHandler - .getFieldsListEntityTypes() - .entries - .map((e) { - var entityType = e.value.entityType; - if (entityType == null) return null; - - var refRepository = repository.provider.getEntityRepositoryByType( - entityType, - ); - - if (refRepository == null || - !refRepository.isSameEntityManager(repository)) { - return null; - } - - return MapEntry(e.key, entityType); - }) - .nonNulls - .toMapFromEntries(); - - if (fieldListEntityTypes.isNotEmpty) { - if (!dialect.foreignKeyCreatesImplicitIndex) { - missingRelationshipFieldReferenceIndexes = collectionReferenceFields - .entries - .expand((e) { - var relFieldRef = e.value; - if (relFieldRef == null) { - return <(String, String, TypeInfo)>[]; - } - - return [ - if (relFieldRef.sourceRelationshipFieldIndex == null) - ( - relFieldRef.relationshipTable, - relFieldRef.sourceRelationshipField, - relFieldRef.sourceFieldEntityType, - ), - if (relFieldRef.targetRelationshipFieldIndex == null) - ( - relFieldRef.relationshipTable, - relFieldRef.targetRelationshipField, - relFieldRef.targetFieldEntityType, - ), - ]; - }) - .nonNulls - .toSet(); - } - } + missingRelationshipFieldReferenceIndexes = + _findMissingRelationshipIndexes( + repository, + collectionReferenceFields, + ); // UNIQUE CONSTRAINTS (unique indexes unknown) // @@ -1510,49 +1429,11 @@ abstract class DBSQLAdapter extends DBRelationalAdapter // ENUM CONSTRAINTS // - var enumConstraints = scheme.constraints - .whereType() - .toList(); - - var enumConstraintsFields = enumConstraints.toFields( - fieldMap: schemeToRepoMap, + (missingEnumConstraints, missingEnumValues) = _findMissingEnumConstraints( + entityHandler, + scheme, + schemeToRepoMap, ); - - var entityEnumFields = entityHandler.getFieldsEnumTypes().keys.toList(); - - missingEnumConstraints = entityEnumFields - .whereNotIn(enumConstraintsFields) - .toSet(); - - // A column's enum values are missing only if no `CHECK` over it has - // them all: with a duplicated `CHECK` (an old one never dropped), the - // current one has them, and the duplicate is warned - // (`findTableDuplicates`). - var columnsWithAllValues = {}; - var columnsMissingValues = >{}; - - for (var e in enumConstraints) { - var valuesNames = _columnEnumValues( - entityHandler, - e.field, - schemeToRepoMap, - ); - if (valuesNames == null) continue; - - var missingValues = valuesNames.whereNotIn(e.values).toList(); - - if (missingValues.isEmpty) { - columnsWithAllValues.add(e.field); - } else { - columnsMissingValues.putIfAbsent(e.field, () => missingValues); - } - } - - for (var e in columnsMissingValues.entries) { - if (!columnsWithAllValues.contains(e.key)) { - missingEnumValues[e.key] = e.value; - } - } } // The entity's indexes (`@EntityIndex`) without an index over their @@ -1579,51 +1460,17 @@ abstract class DBSQLAdapter extends DBRelationalAdapter (f) => missingColumns.contains(f) || missingReferenceColumns.contains(f), ); - // As in the `CREATE TABLE`, not one over the same columns as another - // missing one: a unique field's constraint (reported with the field), a - // unique `@EntityIndex` (also serving a non-unique one), or the same - // index twice. - var missingUniqueColumns = [ - for (var f in missingUniqueConstraints) [fieldColumn(f)], - ]; - var missingEntityIndexesKept = []; - for (var sql in [ - ...missingEntityIndexes.where((e) => e.unique), - ...missingEntityIndexes.where((e) => !e.unique), - ]) { - var columns = sql.columns; - if (missingUniqueColumns.any((c) => c.equals(columns))) continue; - if (missingEntityIndexesKept.any( - (e) => e.columns.equals(columns) && (e.unique || !sql.unique), - )) { - continue; - } - missingEntityIndexesKept.add(sql); - } - // In the declared order: - missingEntityIndexes = missingEntityIndexes - .where(missingEntityIndexesKept.contains) - .toList(); - - // A missing index whose name is in use by another index (other - // columns, or not unique): `CREATE INDEX IF NOT EXISTS` would do - // nothing; it must be dropped first. By index name, where it's in use - // (other tables: [findIndexesNamesInUse], in `_checkDBTablesImpl`). - var entityIndexesNameConflicts = { - for (var sql in missingEntityIndexes) - if (sql.indexName case var indexName?) - if (scheme.indexes?[indexName] case var columns?) - indexName: - 'over (${columns.join(', ')})' - '${sql.unique && !(scheme.uniqueIndexes?.contains(indexName) ?? false) ? ', not unique' : ''}', - }; + missingEntityIndexes = _dedupeMissingEntityIndexes( + missingEntityIndexes, + missingUniqueColumns: [ + for (var f in missingUniqueConstraints) [fieldColumn(f)], + ], + ); - if (entityIndexesNameConflicts.isNotEmpty) { - _log.severe( - "Table `$schemeTableName`: `@EntityIndex` names in use by other " - "indexes (drop them first): $entityIndexesNameConflicts", - ); - } + var entityIndexesNameConflicts = _findEntityIndexesNameConflicts( + scheme, + missingEntityIndexes, + ); if (missingColumns.isNotEmpty || missingReferenceColumns.isNotEmpty || @@ -1637,45 +1484,36 @@ abstract class DBSQLAdapter extends DBRelationalAdapter missingEntityIndexes.isNotEmpty) { _log.severe( "ERROR Checking table `$schemeTableName`> entityType: `$repoType`\n" - "${missingEntityIndexes.isNotEmpty ? '-- missingEntityIndexes: ${missingEntityIndexes.map((e) => e.indexName).toList()}\n' : ''}" - "${missingColumns.isNotEmpty ? '-- missingColumns: $missingColumns\n' : ''}" - "${missingReferenceColumns.isNotEmpty ? '-- missingReferenceColumns: $missingReferenceColumns\n' : ''}" - "${missingCollectionReferenceColumns.isNotEmpty ? '-- missingCollectionReferenceColumns: $missingCollectionReferenceColumns\n' : ''}" - "${missingFieldReferenceConstraints.isNotEmpty ? '-- missingFieldReferenceConstraints: $missingFieldReferenceConstraints\n' : ''}" - "${missingFieldReferenceIndexes.isNotEmpty ? '-- missingFieldReferenceIndexes: $missingFieldReferenceIndexes\n' : ''}" - "${missingRelationshipFieldReferenceIndexes.isNotEmpty ? '-- missingRelationshipFieldReferenceIndexes: $missingRelationshipFieldReferenceIndexes\n' : ''}" - "${missingUniqueConstraints.isNotEmpty ? '-- missingUniqueConstraints: $missingUniqueConstraints\n' : ''}" - "${missingEnumConstraints.isNotEmpty ? '-- missingEnumConstraints: $missingEnumConstraints\n' : ''}" - "${missingEnumValues.isNotEmpty ? '-- missingEnumValues: $missingEnumValues\n' : ''}", + "${_tableCheckErrorLine('missingEntityIndexes', missingEntityIndexes.map((e) => e.indexName).toList())}" + "${_tableCheckErrorLine('missingColumns', missingColumns)}" + "${_tableCheckErrorLine('missingReferenceColumns', missingReferenceColumns)}" + "${_tableCheckErrorLine('missingCollectionReferenceColumns', missingCollectionReferenceColumns)}" + "${_tableCheckErrorLine('missingFieldReferenceConstraints', missingFieldReferenceConstraints)}" + "${_tableCheckErrorLine('missingFieldReferenceIndexes', missingFieldReferenceIndexes)}" + "${_tableCheckErrorLine('missingRelationshipFieldReferenceIndexes', missingRelationshipFieldReferenceIndexes)}" + "${_tableCheckErrorLine('missingUniqueConstraints', missingUniqueConstraints)}" + "${_tableCheckErrorLine('missingEnumConstraints', missingEnumConstraints)}" + "${_tableCheckErrorLine('missingEnumValues', missingEnumValues)}", ); + // The columns of [fields], as reported by the [_DBTableCheck]: + List<_DBTableColumn> columnsOf(Iterable fields) => fields + .map((f) => _DBTableColumn.fromEntityHandler(entityHandler, f)) + .toList(); + return _DBTableCheck.missing( repository, scheme, - missingColumns - .map((f) => _DBTableColumn.fromEntityHandler(entityHandler, f)) - .toList(), - missingReferenceColumns - .map((f) => _DBTableColumn.fromEntityHandler(entityHandler, f)) - .toList(), - missingCollectionReferenceColumns - .map((f) => _DBTableColumn.fromEntityHandler(entityHandler, f)) - .toList(), - missingFieldReferenceConstraints - .map((f) => _DBTableColumn.fromEntityHandler(entityHandler, f)) - .toList(), - missingFieldReferenceIndexes - .map((f) => _DBTableColumn.fromEntityHandler(entityHandler, f)) - .toList(), + columnsOf(missingColumns), + columnsOf(missingReferenceColumns), + columnsOf(missingCollectionReferenceColumns), + columnsOf(missingFieldReferenceConstraints), + columnsOf(missingFieldReferenceIndexes), missingRelationshipFieldReferenceIndexes .map((f) => _DBRelationshipTableColumn(f.$1, f.$2, f.$3, [])) .toList(), - missingUniqueConstraints - .map((f) => _DBTableColumn.fromEntityHandler(entityHandler, f)) - .toList(), - missingEnumConstraints - .map((f) => _DBTableColumn.fromEntityHandler(entityHandler, f)) - .toList(), + columnsOf(missingUniqueConstraints), + columnsOf(missingEnumConstraints), missingEnumValues.entries .map( (e) => MapEntry( @@ -1694,6 +1532,252 @@ abstract class DBSQLAdapter extends DBRelationalAdapter return _DBTableCheck.ok(); } + /// A line of the table check's error log (see [_checkDBTableScheme]): + /// `--