Skip to content

Clarify RDI snapshot modes and missed deletes - #4238

Open
ZdravkoDonev-redis wants to merge 7 commits into
mainfrom
docs/rdi-snapshot-modes-and-missed-deletes
Open

ZdravkoDonev-redis wants to merge 7 commits into
mainfrom
docs/rdi-snapshot-modes-and-missed-deletes

Conversation

@ZdravkoDonev-redis

@ZdravkoDonev-redis ZdravkoDonev-redis commented Oct 9, 2026 •

Copy link
Copy Markdown
Collaborator

When a saved source log position expires, when_needed can take a new snapshot and resume collection while leaving deleted rows in Redis. These docs explain that gap and the connector differences between snapshot modes.

Compare all six listed modes for Debezium sources in one shared explanation used by latest and v1.19.1. Include a shared Mermaid missed-delete timeline and a readable ordered sequence in the Cloud FAQ. Update configuration references, deployment/reset guidance, troubleshooting, multiple-source guidance, and Cloud pages. The embed shortcode enables Mermaid rendering in the parent page.

Both Architecture pages retain the delivery guarantee and add a visible disclaimer about events that cannot be captured from expired source history. Reset and target cleanup remain separate.

Document the Kubernetes restart loop after initial_only completes. Explain where to set Snowflake clearOffset, how INITIAL versus NEVER determines whether existing rows reload, and why the flag should be disabled after recovery. Warn that missed deletes can leave stale records even after a recovery snapshot finishes.

Validation:

  • Connector semantics checked against the bundled Debezium 3.5 baseline, including PostgreSQL no_data; release mapping checked for RDI 1.19.1 and 2.0.0.
  • Focused Hugo 0.143.1 build passed without warnings.
  • YAML examples validated for all six modes; added links and anchors checked.
  • Latest and v1.19.1 sequence diagrams previously verified at desktop and 390 px widths.
  • New initial_only note and Snowflake recovery guidance verified at 1280 and 390 px without horizontal overflow; the collector-property editor anchor resolves.
  • Snowflake recovery wording checked against the Cloud property mapping and RIOT-X offset/stream initialization code.

Companion schema-description PR: https://github.com/RedisLabs/redis-data-integration/pull/2886

Context: Slack discussion.


Note

Low Risk
Documentation-only changes that clarify existing CDC behavior; no runtime or configuration schema changes in this repo.

Overview
Documents Debezium snapshot.mode for RDI CDC sources and the stale-target risk when source log history expires before the collector resumes.

Adds shared embeds (rdi-snapshot-modes.md, rdi-snapshot-gap.md) with a mode comparison table, Debezium 3.5 connector notes, and a Mermaid missed-delete timeline. Pipeline config gains a Choose a snapshot mode section; architecture adds a note that at-least-once delivery does not cover replay of expired history.

Reset / start / stop wording is corrected: redis-di reset clears saved positions without changing snapshot.mode or flushing Redis; recovery behavior depends on the configured mode. New troubleshooting covers unavailable source log history; multiple-source and Redis Cloud (define, FAQ, view-edit) pages are aligned.

The embed-md shortcode now propagates hasMermaid so embedded diagrams render on parent pages. Config reference text for snapshot.mode is expanded and linked to the new guide.

Reviewed by Cursor Bugbot for commit d87ebb8. Bugbot is set up for automated code reviews on this repo. Configure here.

@github-actions

github-actions Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Staging links:
https://redis.io/docs/staging/docs/rdi-snapshot-modes-and-missed-deletes/embeds/rdi-snapshot-modes
https://redis.io/docs/staging/docs/rdi-snapshot-modes-and-missed-deletes/integrate/redis-data-integration/1.19.1/architecture/
https://redis.io/docs/staging/docs/rdi-snapshot-modes-and-missed-deletes/integrate/redis-data-integration/1.19.1/data-pipelines/deploy/
https://redis.io/docs/staging/docs/rdi-snapshot-modes-and-missed-deletes/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config
https://redis.io/docs/staging/docs/rdi-snapshot-modes-and-missed-deletes/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config/
https://redis.io/docs/staging/docs/rdi-snapshot-modes-and-missed-deletes/integrate/redis-data-integration/1.19.1/reference/cli/redis-di-reset/
https://redis.io/docs/staging/docs/rdi-snapshot-modes-and-missed-deletes/integrate/redis-data-integration/1.19.1/reference/config-yaml-reference/
https://redis.io/docs/staging/docs/rdi-snapshot-modes-and-missed-deletes/integrate/redis-data-integration/1.19.1/troubleshooting/
https://redis.io/docs/staging/docs/rdi-snapshot-modes-and-missed-deletes/integrate/redis-data-integration/architecture/
https://redis.io/docs/staging/docs/rdi-snapshot-modes-and-missed-deletes/integrate/redis-data-integration/data-pipelines/deploy/
https://redis.io/docs/staging/docs/rdi-snapshot-modes-and-missed-deletes/integrate/redis-data-integration/data-pipelines/multiple-sources/
https://redis.io/docs/staging/docs/rdi-snapshot-modes-and-missed-deletes/integrate/redis-data-integration/data-pipelines/pipeline-config
https://redis.io/docs/staging/docs/rdi-snapshot-modes-and-missed-deletes/integrate/redis-data-integration/data-pipelines/pipeline-config/
https://redis.io/docs/staging/docs/rdi-snapshot-modes-and-missed-deletes/integrate/redis-data-integration/reference/cli/redis-di-reset/
https://redis.io/docs/staging/docs/rdi-snapshot-modes-and-missed-deletes/integrate/redis-data-integration/reference/config-yaml-reference/
https://redis.io/docs/staging/docs/rdi-snapshot-modes-and-missed-deletes/integrate/redis-data-integration/troubleshooting/
https://redis.io/docs/staging/docs/rdi-snapshot-modes-and-missed-deletes/operate/rc/rdi/define/
https://redis.io/docs/staging/docs/rdi-snapshot-modes-and-missed-deletes/operate/rc/rdi/faq/
https://redis.io/docs/staging/docs/rdi-snapshot-modes-and-missed-deletes/operate/rc/rdi/view-edit/

@github-actions

github-actions Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

🧠 Redis Memory

Found 10 related items from repository history (1 new this commit):

Memory updated at d4cdff7

Comment thread content/integrate/redis-data-integration/1.19.1/architecture/_index.md Outdated
Comment thread content/integrate/redis-data-integration/1.19.1/data-pipelines/pipeline-config.md Outdated
Comment thread content/integrate/redis-data-integration/architecture/_index.md Outdated
Comment thread content/integrate/redis-data-integration/reference/config-yaml-reference.md Outdated

@andy-stark-redis andy-stark-redis left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some suggestions (mainly adding the word "mode" after always, initial, etc) and I'd prefer not to use that new shortcode unless you really feel it's necessary. Otherwise looks good, so I'll approve to avoid holding you up.

Comment thread content/embeds/rdi-snapshot-modes.md Outdated
Comment thread content/embeds/rdi-snapshot-modes.md Outdated
Comment thread content/integrate/redis-data-integration/1.19.1/troubleshooting.md Outdated
Comment thread content/integrate/redis-data-integration/data-pipelines/deploy.md Outdated
Comment thread content/integrate/redis-data-integration/data-pipelines/multiple-sources.md Outdated
Comment thread content/operate/rc/rdi/define.md Outdated
Comment thread content/operate/rc/rdi/define.md Outdated
Comment thread content/operate/rc/rdi/faq.md Outdated
Comment thread content/operate/rc/rdi/faq.md Outdated
Comment thread content/operate/rc/rdi/view-edit.md Outdated
Comment thread content/operate/rc/rdi/view-edit.md Outdated
Comment thread content/embeds/rdi-snapshot-modes.md Outdated
Comment thread content/embeds/rdi-snapshot-modes.md
Comment thread content/embeds/rdi-snapshot-modes.md
Comment thread content/embeds/rdi-snapshot-modes.md
Comment thread content/embeds/rdi-snapshot-modes.md
Comment thread content/integrate/redis-data-integration/1.19.1/troubleshooting.md
Comment thread content/integrate/redis-data-integration/troubleshooting.md Outdated
Comment thread content/operate/rc/rdi/view-edit.md Outdated
Comment thread content/embeds/rdi-snapshot-modes.md
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants