v0.8.58: desktop app catchas, insights dashboard, kb fixes - #8200
Merged
Merged
Conversation
…8189) A statement, lock, or transaction timeout while indexing a workspace file means the database had no capacity for the build, not that the file is bad. The task retried it within seconds, so all three attempts landed in the same slow window and the revision was marked failed for good. Capacity timeouts now retry after about 2, 4, 8, 16 and 30 minutes (six attempts); other failures keep three attempts with the default delays.
…f eight attempts (#8190) * fix(db): retry migration lock timeouts within a time budget instead of eight attempts * fix(db): measure the migration lock budget on a monotonic clock and never start an attempt past it
…urnstile passes (#8192) * fix(desktop): set the browser user agent process-wide so Cloudflare Turnstile passes * test(desktop): cover the process-wide user agent from the first launched request
…essing commit (#8191) * fix(knowledge): check connector and knowledge base at the end of the processing commit * fix(knowledge): skip index writes for an inactive source and probe connector locks after the insert * test(knowledge): arm the pre-commit source check in the processing utils test
…sign the overview (#8193) * improvement(insights): serve org usage from settled segments and redesign the overview * fix(insights): bound usage settling by the stream cap, discard unreadable segments, neutral zero delta * fix(insights): align rolling windows to the viewer's hour and clear stale legend highlights * improvement(emcn): draw each stacked column as one bar split by color * improvement(insights): brand-aligned, colorblind-validated chart palette
…etries (#8194) * fix(jobs): classify transient database failures once for background retries - classifyDatabaseFailure in @sim/utils/errors: capacity / conflict / connection / permanent - shared Trigger catchError decision helper with minute-scale database backoff - workspace file search, document processing, connector sync, and the 0021 backfill adopt it - a scheduled document retry leaves the document pending instead of failed - a transient database failure no longer counts toward connector auto-disable * fix(jobs): escalate database retries without spending the breaker, cover members mode - connector and members-mode syncs climb the failure ladder by the failed-run streak read from their run logs; database failures never advance the auto-disable counter - classify 25P03 as capacity and ECONNREFUSED/EHOSTUNREACH/ENOTFOUND/EAI_AGAIN (with a query) as connection - in-process document processing keeps recording database failures as failed, documented - offer Retry for a pending document whose dispatch or deferred retry is past the retry API's grace - correct the processing task's retry-ceiling comment * revert(knowledge): drop the stale-deferred Retry action and its wire fields Restores the document list, contract, serializer, and Retry condition to staging; Retry stays failed-only. * fix(knowledge): fail an uploaded document whose scheduled database retry never ran - the deferral write and a deferred-retry-check outbox event commit together for uploaded documents - the check fails the document only if the same deferral is still pending, overdue past the queue grace, and no run is live; otherwise it rechecks without spending an attempt, bounded by the recovery window - a database deferral stamps processingQueuedAt when unset, so a dispatch cannot claim the row as never-queued during the retry window * fix(knowledge): keep the deferred-retry watchdog alive through database failures A transient database failure while checking postpones the check without spending an attempt, paced by how overdue it is and capped at the recheck interval, until a terminal bound past the recovery window; other errors, and any error past that bound, spend attempts as before. * fix(jobs): progress-aware connector database retries, stuck-sync signal, strict query shapes - the database-failure streak counts only failed runs that made no progress; a run that wrote documents (or completed a member) retries in minutes - ten zero-progress database failures in a row log an alertable error; the connector is never disabled for them - the classifier's query check matches only Drizzle's query error and the postgres.js query error shapes, so a client error carrying its own query stays a source failure * fix(jobs): bound the retry-history read, count member purges, match driver connection errors - the connector run-history read runs under short statement and lock timeouts and falls back to this run alone - members-mode progress counts lifecycle purges (docs_purged in the run log) - postgres.js connection codes count only on an error the driver built, or under a query error; the driver's query signature is its four own properties, which a refused connection carries with no SQL yet - real-error PostgreSQL test for terminated transactions, refused connections, and an ending pool, wired into CI - knowledge-processing tests use the static task import * fix(jobs): classify ENETDOWN, ENETRESET and ENETUNREACH on a database query as connection failures * test(knowledge): import the connector retry helpers statically
…8196) Co-authored-by: Sim Pi Agent <pi@sim.ai>
…ort page (#8195) * fix(knowledge): bound every connector-lease ACL transaction to one short page * fix(knowledge): isolate stale sweep lock failures, close failed disables, update ACL test callers * fix(knowledge): prove connector leases last, page ACL writes by projection rows, and skip unchanged work * fix(knowledge): row-bound every observation page, size ACL pages from locked chunk counts, and check the sweep budget per page * fix(knowledge): stop every ACL page walker at the run budget and keep pending rewrites until they finish * test(knowledge): seed filled projection rows so the fan-out test holds on any provisioned database * fix(knowledge): check the page deadline after the lease heartbeat * test(knowledge): expire the restore budget at the heartbeat before the next window
#8199) * feat(knowledge): project document ACL and chunk changes asynchronously Document ACL/source changes and chunk writes mark their document in knowledge_projection_dirty (one upsert with a generation bump) in the writer's transaction. A knowledge projector converges every search projection per document in short pages bounded by chunk rows, several documents at once, and removes a mark only on the generation it read. Writers that declare sim.projection_mode = 'async' skip the synchronous embedding and document fan-out triggers. The processing commit and every connector-lease ACL page declare it while the knowledge-async-projection flag is on; every other writer, and every release before this one, keeps writing projection rows itself. Marks are written in both modes so a projector pass can never settle over a concurrent synchronous write. Migrations 0021-0023 keep their trigger body; 0024 alone installs the marking. Search decides a marked document's rows on the document itself, so a pending projection never admits a revoked grant. A document that moved sources joins the new source's per-source ranking after its pass. The projector runs as a Trigger.dev task, requested after writes and swept every minute while there is work, or in-process without Trigger.dev. A pass opens one worker per mark up to KB_CONFIG_PROJECTION_CONCURRENCY. The separate source/ACL backfill is folded into it behind knowledge-projection-fill. * fix(knowledge): keep projection guards out of historical migrations Restores 0016, 0019 and 0021 to their staging bodies; 0024 alone re-creates the projection triggers with the mode guard, in the same transaction that installs the marks. The fill starts only inside the pass budget and reports what it marked as remaining. Passes dispatch to Trigger.dev by the rule document processing uses, and the inline coalescer clears its running flag in the same step it reads the owed flag. Bumps the Helm chart for the projection cron job. * test(knowledge): count projection requests per connector ACL case
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Contributor
 |
![cubic-dev-ai[bot]](https://avatars.githubusercontent.com/in/1082092?s=60&v=4){kind=small}
Contributor
There was a problem hiding this comment.
All reported issues were addressed across 149 files
Tip: instead of fixing issues one by one fix them all with cubic
Re-trigger cubic
…026 (#8203) Co-authored-by: Sim Pi Agent <pi@sim.ai>
Co-authored-by: Sim Pi Agent <pi@sim.ai>
…ection fill, and sync retries (#8202) * fix(knowledge): address release review follow-ups for ACL pages, projection fill, and sync retries - Scope the observed-ACL aggregate to members of the document's own connector - Advance a member's listing checkpoint only after the ACLs it decides are written, and rematerialise every document a replayed feed removal names - Plan ACL pages from an unlocked read and lock only the page that will be written - Lock fill documents FOR KEY SHARE SKIP LOCKED so a concurrent delete cannot fail the mark's foreign key - Report an unfinished projection fill as remaining work - Record the database failure class on failed sync-log rows and count only those toward the database retry streak (expand-only migration 0379) - Heartbeat the lease between revokeDocumentAcls transactions - Extract isTriggerAvailable to lib/core/config/trigger-availability and use it for every dispatch decision - Exercise a concurrently committed deletion in the processing lock-scope test * fix(knowledge): constrain the recorded database failure class
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.