DAOS-19721 vos: fix stale iterator re-probe in vos_iter_validate by mchaarawi · Pull Request #19182 · daos-stack/daos

mchaarawi · 2026-09-30T22:50:16Z

When the scrubber yields mid-extent, the object or key it is scanning may be punched and aggregated away. vos_iter_validate_internal() re-probes parents with GE semantics, so it silently lands on the next entry and child iterators then probe freed tree roots, triggering "Embedded root is incompatible with integer keys" in btr_probe_embedded.

Restart from the object level if the held object was evicted.
Treat an OBJ/DKEY/AKEY re-probe that lands on a different entry as "entry gone".
Use ia_ev (not ia_sv) as the RECX anchor so corruption is marked on the correct extent.

Steps for the author:

Commit message follows the guidelines.
Appropriate Features or Test-tag pragmas were used.
Appropriate Functional Test Stages were run.
At least two positive code reviews including at least one code owner from each category referenced in the PR.
Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

Gatekeeper requested (daos-gatekeeper added as a reviewer).

When the scrubber yields mid-extent, the object or key it is scanning may be punched and aggregated away. vos_iter_validate_internal() re-probes parents with GE semantics, so it silently lands on the next entry and child iterators then probe freed tree roots, triggering "Embedded root is incompatible with integer keys" in btr_probe_embedded. - Restart from the object level if the held object was evicted. - Treat an OBJ/DKEY/AKEY re-probe that lands on a different entry as "entry gone". - Use ia_ev (not ia_sv) as the RECX anchor so corruption is marked on the correct extent. Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@hpe.com>

github-actions · 2026-09-30T23:05:28Z

Ticket title is 'btr_probe_embedded() Assertion '!btr_is_int_key(tcx)' failed: Embedded root is incompatible with integer keys '
Status is 'In Review'
Labels: '2.8.1rc2'
https://daosio.atlassian.net/browse/DAOS-19721

Add object_deleted_by_aggregation_and_gc to the VOS scrubbing test suite. It reproduces the DAOS-19721 use-after-free: while the scrubber yields mid-extent, the object it is scrubbing is fully punched, aggregated away, and garbage collected. A decoy object with a higher OID is kept alive so vos_iter_validate()'s OBJ-level BTR_PROBE_GE re-probe lands on it instead of correctly reporting the deleted object as gone, letting the DKEY-level re-probe run against the freed/reused dkey tree root. Pre-fix, this aborts the process with the btr_probe_embedded() assertion from DAOS-19721. Post-fix (see PR #19182), vos_scrub_pool() completes cleanly, skipping the deleted object. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>

grom72 · 2026-10-13T14:38:19Z

The fix does not seem to solve the problem as it is confirmed in https://daosio.atlassian.net/browse/DAOS-19769 (https://github.com/daos-stack/daos/pull/19205/changes#diff-7bfac1f5c3abd14bc39ecbd989eaaceeef15f87acbdb7921de922c72ed329da6).

mchaarawi · 2026-10-13T21:12:35Z

The fix does not seem to solve the problem as it is confirmed in https://daosio.atlassian.net/browse/DAOS-19769 (https://github.com/daos-stack/daos/pull/19205/changes#diff-7bfac1f5c3abd14bc39ecbd989eaaceeef15f87acbdb7921de922c72ed329da6).

looked into your pr and correct me if im wrong that CSUM_SCRUBBING_15 links the standalone libvos, where vos_sched_seq() is hard-wired to 0.
That means vos_iter_sched_check() is never true, and vos_iter_validate_internal() / vos_iter_cb() return before any of this PR's logic runs. correct?
The test also frees the object synchronously inside the callback with no yield, which can't happen for the scrubber in the engine. Aggregation and GC run in other ULTs, and every ULT switch bumps si_cur_seq.
from what i understand, the situation you target is real, but the way it gets there can't happen in the engine.

to make it valid per my understanding, you need to add a test hook that advances the scheduler counter, and do the delete through the scrubber's yield callback, like CSUM_SCRUBBING_08–10, not inline. then use two or more objects and delete one that isn't last. That reproduces the production sequence and would fail on master but should pass with this PR.
happy to do that if you want me to and you agree about the test itself; let me know.

NiuYawei · 2026-10-19T02:10:13Z

 	}

+	/* Object punched/deleted while yielding; cached obj_df and tree roots may be freed */
+	if (iter->it_type != VOS_ITER_OBJ && vos_obj_is_evicted(vos_iter2oiter(iter)->it_obj)) {


I don't follow the 'vos_obj_evicted()' checking, why it's necessary here?

as i said in the ticket, this patch is ai generated. but here is the reasoning:

Guards the case where the object is deleted and recreated with the same oid while we're yielded. The OBJ re-probe then matches the anchor, but the child iterators are still bound to the freed tree. Eviction is the only signal at that level, same as the check vos_io.c uses after a yield.

I don't follow this.

No matter if the object is deleted or not (or recreated with same oid or not). This vos_iter_validate_internal() will always re-probe from the object level, that would re-open subtrees and recreate btree context for every level.

NiuYawei · 2026-10-19T02:14:48Z

 	rc = iter->it_ops->iop_probe(iter, anchor, VOS_ITER_PROBE_AGAIN);
 	vos_dth_set(old, is_sysdb);

+	/* PROBE_AGAIN is GE: a different entry means ours is gone and children hold stale roots */


I'm totally lost. If the entry is deleted, it'll iterate to next entry, I don't see why children hold stale roots?

Child iterators are opened in place on the parent record's root (dbtree_open_inplace_ex(ii_btr) / evt_open(&krec->kr_evt)). When validation re-probes the parent and it lands on the next entry, validation returns 0, so the child re-probes its own handle, still bound to the deleted entry's freed root. That's the btr_probe_embedded assertion in the backtrace.

NiuYawei · 2026-10-19T02:14:54Z

 		break;
 	case VOS_ITER_RECX:
-		anchor = &iter->it_anchors->ia_sv;
+		anchor = &iter->it_anchors->ia_ev;


good catch.

NiuYawei · 2026-10-19T02:21:38Z

BTW, as for DAOS-19721, the sc_verify_recx() looks suspicious to me, it iterates csums in a loop, but I'm not quite sure if the csum is still valid after sc_verify_finish() (called in the loop) yielding.

mchaarawi · 2026-10-19T13:56:47Z

BTW, as for DAOS-19721, the sc_verify_recx() looks suspicious to me, it iterates csums in a loop, but I'm not quite sure if the csum is still valid after sc_verify_finish() (called in the loop) yielding.

yea probably should snapshot before the loop.

sc_verify_recx() yields between chunks, but the stored csums it compares against point into the evtree record. If aggregation frees that record while the scrubber yields, the remaining chunks are compared against freed memory and reported as corruption, which then triggers vos_iter_validate() on a stale iterator chain. Copy the stored csums into a local buffer before the loop, matching the data that is already read into DRAM up front. Features: checksum Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@hpe.com>

daosbuild3 · 2026-10-19T15:08:20Z

Test stage Unit Test completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-19182/2/testReport/

grom72 · 2026-10-19T15:59:11Z

The fix does not seem to solve the problem as it is confirmed in https://daosio.atlassian.net/browse/DAOS-19769 (https://github.com/daos-stack/daos/pull/19205/changes#diff-7bfac1f5c3abd14bc39ecbd989eaaceeef15f87acbdb7921de922c72ed329da6).

looked into your pr and correct me if im wrong that CSUM_SCRUBBING_15 links the standalone libvos, where vos_sched_seq() is hard-wired to 0. That means vos_iter_sched_check() is never true, and vos_iter_validate_internal() / vos_iter_cb() return before any of this PR's logic runs. correct? The test also frees the object synchronously inside the callback with no yield, which can't happen for the scrubber in the engine. Aggregation and GC run in other ULTs, and every ULT switch bumps si_cur_seq. from what i understand, the situation you target is real, but the way it gets there can't happen in the engine.

to make it valid per my understanding, you need to add a test hook that advances the scheduler counter, and do the delete through the scrubber's yield callback, like CSUM_SCRUBBING_08–10, not inline. then use two or more objects and delete one that isn't last. That reproduces the production sequence and would fail on master but should pass with this PR. happy to do that if you want me to and you agree about the test itself; let me know.

I have updated #19205 based on your feedback and now can confirm that the fix works.

NiuYawei

Mark as -1 until my concern is addressed.

NiuYawei · 2026-10-20T02:01:18Z

 	}

+	/* Object punched/deleted while yielding; cached obj_df and tree roots may be freed */
+	if (iter->it_type != VOS_ITER_OBJ && vos_obj_is_evicted(vos_iter2oiter(iter)->it_obj)) {


I don't follow this.

No matter if the object is deleted or not (or recreated with same oid or not). This vos_iter_validate_internal() will always re-probe from the object level, that would re-open subtrees and recreate btree context for every level.

NiuYawei · 2026-10-20T02:14:37Z

+	    (iter->it_type == VOS_ITER_OBJ || iter->it_type == VOS_ITER_DKEY ||
+	     iter->it_type == VOS_ITER_AKEY) &&
+	    memcmp(saved.da_buf, anchor->da_buf, sizeof(saved.da_buf)) != 0)
+		rc = -DER_NONEXIST;


Looks AI is talking to Ghost. This looks not correct to me. (see my comment above)

ok im checking things myself now, and correct my understanding if im wrong as im not familiar with this part of the code.

The scrubber is walking the data in a nested way: object → dkey → akey → extents. It keeps a cursor open at each level. while it is checking an extent, it can yield. but during that yield/pause, aggregation might happen and delete the object it is looking at. vos_iter_validate() is what now validates that we are still working with soemthing real.
as i understand the assertion we saw on aurora from the trace:

Each one of the cursors under the object (dkey, akey, extent) holds a pointer into the associated object's memory.

When validation asks the object level to find its position again and the object is gone, it lands on the next object and reports valid / 0.

the lower level carry on using their old pointers into the deleted object's freed memory.

as a result we see the crash / assertion.
This is my interpretation of the issue @NiuYawei. if you disagree, can you please investigate the trace yourself and let me know what did i miss?

i believe the pushed fix is just returning >0 instead of 0 in the above.
you mentioned that validation reopens the lower cursors fresh every time. are you sure?? can you check again please. as i understand, It only reopens them when a level reports > 0 and before this fix the object level never reported > 0 in this case. please check. My understanding is that validation doesn't re-open the child trees.

in @grom72 PR (#19205), i believe he updated his test in to go through a real yield and confirmed it fails without this patch and passes with it.

Ok. I overlooked that iop_probe() doesn't use BTR_PROBE_EQ, so looks this vos_iter_validate() interface doesn't work at all from very beginning.

And you are right that if parent is deleted but iop_probe() returns 0, probing in a deleted subtree won't work.

#19205 has been updated to use real yield and the test pass with this PR + fix from #19182 (comment)

daosbuild3 · 2026-10-20T11:50:57Z

Test stage Functional Hardware Medium MD on SSD completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-19182/2/testReport/

NiuYawei · 2026-10-20T15:53:42Z

+	if (iter->it_type != VOS_ITER_OBJ && vos_obj_is_evicted(vos_iter2oiter(iter)->it_obj)) {
+		iter->it_anchors->ia_probe_level = VOS_ITER_OBJ;
+		return VOS_ITER_OBJ;
+	}


The comparison between saved anchor and anchor below is sufficient. This fix could be removed, right?

ok, ill remove

NiuYawei · 2026-10-20T15:56:36Z

+	/* Children are bound to the old entry's subtree; a different entry means it's gone */
+	if (rc == 0 &&
+	    (iter->it_type == VOS_ITER_OBJ || iter->it_type == VOS_ITER_DKEY ||
+	     iter->it_type == VOS_ITER_AKEY) &&


Why the checking for VOS_ITER_SINGLE and VOS_ITER_RECX are skipped? They'll move to next when the anchor is deleted, right?

BTW, should we skip comparison when vos_anchor_is_zero(saved)?

there is nothing below them right? the crash happens as i understand only when a parent level moves. those are at the leaf so there are no child pointers left on a deleted subtree.
for recx, probe is a exact match find that returns DER_AGAIN if the extent is gone.

BTW, should we skip comparison when vos_anchor_is_zero(saved)?
yes good point

there is nothing below them right? the crash happens as i understand only when a parent level moves. those are at the leaf so there are no child pointers left on a deleted subtree.
for recx, probe is a exact match find that returns DER_AGAIN if the extent is gone.

But the point of this vos_iter_validate_internal() is to return positive value when the EV or SV is deleted, returning 0 won't cause corruption but it's wrong, isn't it?

Yeah, I didn't realize scrubber doesn't use sorted iterator for recx. For evtree non-sorted iterator, that'll be an exact match.

ok, i guess returning 0 there is wrong even if it doesn't corrupt anything. will leave RECX as is since the unsorted evtree probe is exact.

The interface vos_iter_validate() is a general interface supposed being used for by any kind of iterators (unsorted evtree iterator is used by scrubber only). We can get it landed as-is though, since it can solve the existing corruption issue.

Co-authored-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com> Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@hpe.com>

Drop the object eviction check in vos_iter_validate_internal(). Objects are evicted on every punch and DTX abort without their trees being freed, so it forced needless restarts; the anchor comparison covers the deleted-entry case. Skip the anchor comparison when the saved anchor is zero, since a zero anchor probes from the first entry and would always mismatch. Features: checksum Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@hpe.com>

daosbuild3 · 2026-10-21T12:01:42Z

Test stage Functional Hardware Medium MD on SSD completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-19182/4/testReport/

The SV re-probe moves past a deleted entry and returns 0 without updating the caller's anchor, so validation reported the value as still present. Fetch the current SV anchor after the probe and compare it like the key levels. Unsorted evtree probes are already exact. Features: checksum Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@hpe.com>

daosbuild3 · 2026-10-22T07:13:03Z

Test stage Functional Hardware Medium MD on SSD completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-19182/5/testReport/

mchaarawi · 2026-10-22T14:05:38Z

the failed test is already known with ticket https://daosio.atlassian.net/browse/DAOS-19589

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

Add an injectable scheduler-sequence callback to standalone VOS TLS and build the scrubber test with VOS_STANDALONE. Add CSUM_SCRUBBING_15 test to delete and aggregate the current object during a yield between checksum chunks, then run GC before iteration resumes. Poison freed PMEM allocations through DAOS fault injection to expose use-after-free, and verify the scrubber skips the deleted object and continues processing later objects. Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

Include the affected OID in checksum corruption logs and the RAS event to make it easier to identify the corrupted object. skip-nlt: true skip-test: true skip-test-hardware: true Test-tag: pr,-unittest Priority: 2 Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

grom72 · 2026-10-22T21:37:15Z

Based on the latest test version the fix does not work exactly as expected.
There is no assert, but scrubber skip the first object after the one just deleted.
For details see my comment in Jira ticket.

daosbuild3 · 2026-10-22T22:22:17Z

Test stage Unit Test completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-19182/6/testReport/

daosbuild3 · 2026-10-22T22:41:56Z

Test stage Unit Test with memcheck completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-19182/6/testReport/

daosbuild3 · 2026-10-23T17:04:43Z

Test stage Functional on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19182/6/execution/node/1372/log

When vos_iter_validate() finds the current entry gone, it sets ia_probe_level and iteration unwinds to that level, which then re-probed with PROBE_AGAIN in its current stage (e.g. POST). The GE re-probe lands on the next entry, which was then treated as already visited and skipped, so the scrubber missed the next object. Re-probe that level from the FILTER stage instead, so the entry it lands on is processed normally. CSUM_SCRUBBING_15 covers this. Features: checksum Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@hpe.com>

mchaarawi · 2026-10-23T19:10:36Z

Based on the latest test version the fix does not work exactly as expected. There is no assert, but scrubber skip the first object after the one just deleted. For details see my comment in Jira ticket.

good find. fixed by re-probes as a fresh entry when validation reports that the entry is gone, so the next object is scrubbed normally

daosbuild3 · 2026-10-23T19:24:18Z

Test stage Functional Hardware Medium Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-19182/6/display/redirect

daosbuild3 · 2026-10-23T19:24:33Z

Test stage Functional Cluster Box Medium Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19182/6/execution/node/1706/log

daosbuild3 · 2026-10-23T21:22:38Z

Test stage Functional on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19182/7/execution/node/1367/log

mchaarawi requested review from Nasf-Fan, NiuYawei and gnailzenh October 1, 2026 12:26

grom72 mentioned this pull request Oct 5, 2026

DAOS-19721 vos: add unit test reproducing scrubber UAF assert #19205

Closed

6 tasks

Nasf-Fan previously approved these changes Oct 7, 2026

View reviewed changes

mchaarawi marked this pull request as ready for review October 14, 2026 14:44

mchaarawi requested review from a team as code owners October 14, 2026 14:44

NiuYawei requested changes Oct 19, 2026

View reviewed changes

mchaarawi dismissed Nasf-Fan’s stale review via 5625078 October 19, 2026 13:58

NiuYawei requested changes Oct 20, 2026

View reviewed changes

grom72 reviewed Oct 20, 2026

View reviewed changes

Comment thread src/vos/vos_pool_scrub.c

NiuYawei requested changes Oct 20, 2026

View reviewed changes

mchaarawi and others added 2 commits October 20, 2026 13:58

Update src/vos/vos_pool_scrub.c

5a4f845

Co-authored-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com> Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@hpe.com>

NiuYawei previously approved these changes Oct 21, 2026

View reviewed changes

mchaarawi requested a review from Nasf-Fan October 22, 2026 14:04

grom72 added 4 commits October 22, 2026 21:06

DAOS-19721 vos: add memory poisoning for some tests

a8f880b

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

Merge branch 'master' into grom72/DAOS-19721

556f416

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

grom72 dismissed NiuYawei’s stale review via 556f416 October 22, 2026 21:33

grom72 requested review from a team as code owners October 22, 2026 21:33

Merge branch 'master' into mschaara/19721

0555b3d

Conversation

mchaarawi commented Sep 30, 2026

Steps for the author:

After all prior steps are complete:

Uh oh!

github-actions Bot commented Sep 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

grom72 commented Oct 13, 2026

Uh oh!

mchaarawi commented Oct 13, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

NiuYawei commented Oct 19, 2026

Uh oh!

mchaarawi commented Oct 19, 2026

Uh oh!

daosbuild3 commented Oct 19, 2026

Uh oh!

grom72 commented Oct 19, 2026

Uh oh!

NiuYawei left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

daosbuild3 commented Oct 20, 2026

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

daosbuild3 commented Oct 21, 2026

Uh oh!

daosbuild3 commented Oct 22, 2026

Uh oh!

mchaarawi commented Oct 22, 2026

Uh oh!

grom72 commented Oct 22, 2026

Uh oh!

daosbuild3 commented Oct 22, 2026

Uh oh!

daosbuild3 commented Oct 22, 2026

Uh oh!

daosbuild3 commented Oct 23, 2026

Uh oh!

mchaarawi commented Oct 23, 2026

Uh oh!

daosbuild3 commented Oct 23, 2026

Uh oh!

github-actions Bot commented Sep 30, 2026 •

edited

Loading