WIP: Explain RDI CDC offsets and source retention

[ZdravkoDonev-redis]

What changes

After an interruption, RDI can retain its saved offset while the source can no longer provide the history or capture state needed to resume. Add Understand CDC offsets and source retention next to Architecture, with plain-language explanations and 11 Redis-style SVG diagrams.

Each source section starts with a plain-language summary of its retention controls: time, disk size, cleanup jobs, or a combination. The guide also separates normal resume from manual replay, explains why old changes need matching table definitions, and describes MongoDB full-document lookup behavior.

The diagrams distinguish source-specific mechanisms instead of repeating a single expiry timeline:

Source	Cases explained
Oracle	Missing or inaccessible redo, old restart positions held by long transactions, matching table definitions, snapshot undo, and XStream state
MySQL and MariaDB	Log rotation, file purge/reset, source-switch compatibility, and schema history needed for replay
PostgreSQL	WAL storage growth, size limits, idle timeout, slot continuity, and the confirmed-position limit on rewind
SQL Server	Source capture versus RDI reading, CDC cleanup, and capture-instance removal
MongoDB	Write-volume-dependent oplog history, separate pre-image cleanup, stream invalidation, and current values returned by full-document lookup
Spanner	Separate snapshot and streaming windows, matching RDI and database retention settings, stream lifecycle changes, and changes never captured
Snowflake private preview	Saved table timestamp, stream recreation, Time Travel retention, replacement-table history, and qualified empty-stream checks

Link the guide from the RDI index, Architecture, and Prepare source databases. Correct PostgreSQL minimum-retention wording and replace its unconditional slot-retention guarantee with version-specific guidance for PostgreSQL 10–18. Clarify that MongoDB change streams are used for all ongoing replication, while custom-key and set-output jobs additionally require pre-images and a matching capture mode. Preserve the previous setup-section anchor. Archived docs and product APIs are unchanged.

Validation

• Seven independent database-family reviews against official database and upstream connector documentation; each reviewer checked the revised explanation and SVG labels again.
• Hugo 0.143.0 build passed (19,865 pages). Existing unrelated warnings remain; none concern the edited article or PostgreSQL setup page.
• Internal links and anchors, SVG XML/accessibility metadata, and desktop/mobile rendering checked. All 11 diagrams load without horizontal overflow at 1440 px and 390 px; SVG text bounds also checked.
• Earlier database-native tests passed on PostgreSQL 12.22, 13.23, and 18.6: retained-history decoding, minimum-retention behavior, size-limit invalidation on 13/18, and disabled/enabled idle timeout on 18. Temporary test resources were removed.
• Retention controls and replay limits checked against official database documentation and Debezium 3.5 documentation. Snowflake behavior checked against RIOT-X v1.15.0 (the RDI 2.0.0 chart default): saved table timestamp, stream recreation on batch fetch, and empty-stream checks. Spanner configuration checks verified in the RDI 2.0.0 source. These are code/documentation checks, not new live database reproductions.
• Orhan's offset tests, using collector 3.5.0.Final-rdi.3, provide separate evidence for replay while history remains available. The PostgreSQL RDI rewind replayed 20 of 100 changes under default settings; all 100 replayed with manual acknowledgement only in standalone Debezium. These are not tests of recovery after history expiry. MongoDB token expiry was not reproduced; Spanner and Snowflake were outside that report. This evidence does not resolve the RDI delivery blocker described below.

Why WIP

End-to-end RDI recovery qualification remains incomplete. With RDI 2.0.0 and Debezium 3.5.0.Final-rdi.3, initial snapshots worked, but the first live insert did not reach Redis within the test timeout on all three PostgreSQL versions. Dependent recovery checks remain blocked; the root cause is not established. Database-native tests must not be treated as proof of RDI-to-Redis recovery.

Preview

Understand CDC offsets and source retention (updated after the staging build deploys).

---

Note

Low Risk
Documentation and static assets only; no product code or API changes, though operational guidance should stay aligned with connector behavior.

Overview
Adds Understand CDC offsets and source retention, explaining that a saved capture position is not the same as retained source history and walking through planning, per-source retention behavior, and recovery when history is gone.

The guide is wired in from the RDI overview, Architecture (collector bookmark vs processor checkpoint), and Prepare source databases. PostgreSQL setup gains a Manage WAL retention section with version-specific controls, monitoring queries, and slot rewind limits, replacing blanket slot-retention wording. MongoDB setup now states change streams are used for all ongoing replication, broadens when pre-images are required (custom keys and set output types), and documents matching capture.mode settings.

Eleven new SVG diagrams under static/images/rdi/cdc-retention/ illustrate resume vs missing history for Oracle, MySQL/MariaDB, PostgreSQL, SQL Server, MongoDB, Spanner, and Snowflake.

^{Reviewed by Cursor Bugbot for commit e3bb66e. Bugbot is set up for automated code reviews on this repo. Configure here.}

[github-actions]

Staging links:
https://redis.io/docs/staging/rdi-cdc-offsets-retention/integrate/redis-data-integration/
https://redis.io/docs/staging/rdi-cdc-offsets-retention/integrate/redis-data-integration/architecture/
https://redis.io/docs/staging/rdi-cdc-offsets-retention/integrate/redis-data-integration/cdc-offsets-and-retention
https://redis.io/docs/staging/rdi-cdc-offsets-retention/integrate/redis-data-integration/cdc-offsets-and-retention/
https://redis.io/docs/staging/rdi-cdc-offsets-retention/integrate/redis-data-integration/data-pipelines/prepare-dbs/
https://redis.io/docs/staging/rdi-cdc-offsets-retention/integrate/redis-data-integration/data-pipelines/prepare-dbs/mongodb/
https://redis.io/docs/staging/rdi-cdc-offsets-retention/integrate/redis-data-integration/data-pipelines/prepare-dbs/postgresql/

[github-actions]

🧠 Redis Memory

Found 8 related items from repository history:

• PR — Clarify RDI snapshot modes and missed deletes

  When a saved source log position expires, when_needed can take a new snapshot and resume collection while leaving deleted rows in Redis. T

• PR — DOC-7142 Update RDI when-to-use limits for the Flink default; reunify Cloud and on-prem lists

  DOC-7142

• PR — RDSC-4929: Clarify SQL Server schema migration and capture history limits

  SQL Server's existing capture instance can continue emitting the old column structure after a schema change, so an added column can be omitt

• PR — RDSC-6038: Add RDI 2.0.0 release notes

  Adds the September 2026 RDI 2.0.0 release notes and changes rdi_current_version from 1.19.1 to 2.0.0. The site currently has no 2.0.0

• PR — RDSC-6038: Update Redis Cloud RDI documentation for 2.0.0

  Updates the Cloud RDI guides for the RDI 2.0.0 console. The current guides describe one source and the older five-step setup. The updated gu

• PR — DOC-6878 Expand RDI FAQ CDC section with all supported databases

  What

• PR — Document Supabase support for RDI

  What changed

• Commit — Explain RDI CDC offsets and source retention

  Explain RDI CDC offsets and source retention

Memory updated at e3bb66e