Repository navigation
DAOS-19820 dtx: resync old DTX after quick engine restart - #19361
Open
liuxuezhao wants to merge 2 commits into
Open
liuxuezhao wants to merge 2 commits into
liuxuezhao wants to merge 2 commits into
Conversation
If the DTX leader engine is killed and restarted quickly without pool map change, the DTX entries that have been replied to the client as committable (only in leader's DRAM CoS cache) but not yet batched committed are never committed: - the DTX resync triggered by pool map update on the restarted engine skips the entries whose version equals the resync version (to avoid racing with in-flight IO handlers), but there is no such handler for the entries generated before the engine start; - the stand-alone read on the leader ignores the non-committed leader DTX, and DTX refresh from non-leader gets PREPARED from the leader, so all the replicas return the stale data. Fix: - dtx_resync: do not skip the entry with the same version as resync version if its epoch is older than the engine start epoch; - vos_dtx_check_availability: for stand-alone read on the leader, if the non-committed DTX version is newer than the container's resync version (resync not yet done), return -DER_INPROGRESS to make the client retry instead of ignoring it. Verified by daos_test DTX43: daos_test -n <daos_control.yml> -T -s 3 -u 43 Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com>
Add daos_test DTX43 to reproduce data loss when the DTX leader engine is killed and restarted quickly without any pool map change: - disable batched commit on the leader (DAOS_DTX_NO_BATCHED_CMT), so the replied updates are only PREPARED on all replicas and their committable status exists only in the leader's DRAM CoS cache; - kill the leader with "dmg system stop --force" and restart it with "dmg system start" before it is excluded (pool self_heal=rebuild); - the pool map version stays unchanged, the restarted leader skips those DTX entries in resync (same version as resync version), so the subsequent reads (from any replica) return the stale values. The test also covers the case that a non-leader replica is restarted quickly while the DTXs are only committable in the leader's CoS cache, the reads from the restarted replica must get the new values via DTX refresh with the leader. Also add the dmg_system_query_rank_state() helper to wait for the rank state via "dmg system query". Run only this test with: daos_test -n <daos_control.yml> -T -s 3 -u 43 Signed-off-by: Xuezhao Liu <xuezhao.liu@hpe.com>
|
Ticket title is 'DTX: data loss when leader engine restarts quickly without pool map change ' |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
If the DTX leader engine is killed and restarted quickly without pool map change, the DTX entries that have been replied to the client as committable (only in leader's DRAM CoS cache) but not yet batched committed are never committed:
Fix:
Verified by daos_test DTX43:
daos_test -n <daos_control.yml> -T -s 3 -u 43
Steps for the author:
After all prior steps are complete: