DAOS-19721 vos: add unit test reproducing scrubber UAF assert by grom72 · Pull Request #19205 · daos-stack/daos

grom72 · 2026-10-05T15:11:40Z

Add CSUM_SCRUBBING_15 to reproduce the checksum-scrubber race when aggregation and GC delete an object during yielding.
The iterator fix is intentionally out of scope and will be delivered in a separate PR. See: #19182.

Steps for the author:

Commit message follows the guidelines.
Appropriate Features or Test-tag pragmas were used.
Appropriate Functional Test Stages were run.
At least two positive code reviews including at least one code owner from each category referenced in the PR.
Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

Gatekeeper requested (daos-gatekeeper added as a reviewer).

When the scrubber yields mid-extent, the object or key it is scanning may be punched and aggregated away. vos_iter_validate_internal() re-probes parents with GE semantics, so it silently lands on the next entry and child iterators then probe freed tree roots, triggering "Embedded root is incompatible with integer keys" in btr_probe_embedded. - Restart from the object level if the held object was evicted. - Treat an OBJ/DKEY/AKEY re-probe that lands on a different entry as "entry gone". - Use ia_ev (not ia_sv) as the RECX anchor so corruption is marked on the correct extent. Signed-off-by: Mohamad Chaarawi <mohamad.chaarawi@hpe.com>

github-actions · 2026-10-05T15:13:57Z

Ticket title is 'btr_probe_embedded() Assertion '!btr_is_int_key(tcx)' failed: Embedded root is incompatible with integer keys '
Status is 'In Review'
Labels: '2.8.1rc2'
https://daosio.atlassian.net/browse/DAOS-19721

daosbuild3 · 2026-10-05T16:44:00Z

Test stage Functional on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19205/1/execution/node/1344/log

daosbuild3 · 2026-10-06T16:03:50Z

Test stage Unit Test completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-19205/4/testReport/

daosbuild3 · 2026-10-06T16:24:26Z

Test stage Unit Test with memcheck completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-19205/4/testReport/

daosbuild3 · 2026-10-06T18:23:46Z

Test stage Functional on EL 9 with Valgrind completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19205/4/execution/node/1119/log

daosbuild3 · 2026-10-06T19:04:28Z

Test stage Unit Test completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-19205/5/testReport/

daosbuild3 · 2026-10-06T19:07:21Z

Test stage Unit Test with memcheck completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-19205/5/testReport/

daosbuild3 · 2026-10-06T19:27:28Z

Test stage Unit Test completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-19205/6/testReport/

daosbuild3 · 2026-10-06T19:35:31Z

Test stage Unit Test completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-19205/7/testReport/

daosbuild3 · 2026-10-06T19:52:37Z

Test stage Unit Test with memcheck completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-19205/7/testReport/

daosbuild3 · 2026-10-06T20:04:42Z

Test stage Functional on Leap 15 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-19205/4/testReport/

daosbuild3 · 2026-10-06T20:13:07Z

Test stage Functional on SLES 15 completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-19205/4/testReport/

daosbuild3 · 2026-10-06T21:09:26Z

Test stage Functional on EL 9 with Valgrind completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19205/5/execution/node/1167/log

daosbuild3 · 2026-10-07T11:10:11Z

Test stage Unit Test completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-19205/8/testReport/

daosbuild3 · 2026-10-07T11:23:37Z

Test stage Unit Test with memcheck completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-19205/8/testReport/

daosbuild3 · 2026-10-07T15:58:34Z

Test stage Unit Test completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-19205/9/testReport/

daosbuild3 · 2026-10-07T16:11:40Z

Test stage Unit Test with memcheck completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-19205/9/testReport/

daosbuild3 · 2026-10-12T16:04:05Z

Test stage Unit Test completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-19205/10/testReport/

daosbuild3 · 2026-10-12T16:17:51Z

Test stage Unit Test with memcheck completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-19205/10/testReport/

daosbuild3 · 2026-10-12T18:21:28Z

Test stage Unit Test completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-19205/11/testReport/

daosbuild3 · 2026-10-12T18:40:36Z

Test stage Unit Test with memcheck completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-19205/11/testReport/

daosbuild3 · 2026-10-12T20:29:35Z

Test stage Unit Test completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-19205/12/testReport/

daosbuild3 · 2026-10-12T20:52:20Z

Test stage Unit Test with memcheck completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-19205/12/testReport/

daosbuild3 · 2026-10-12T21:19:11Z

Test stage Unit Test completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-19205/13/testReport/

daosbuild3 · 2026-10-22T07:53:45Z

Test stage Functional on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19205/34/execution/node/1379/log

janekmi

Additionally, this comment is still not resolved:

#19205 (comment) ✅

daosbuild3 · 2026-10-22T15:15:15Z

Test stage Unit Test completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-19205/37/testReport/

daosbuild3 · 2026-10-22T15:36:25Z

Test stage Unit Test with memcheck completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-19205/37/testReport/

daosbuild3 · 2026-10-22T16:22:23Z

Test stage Functional Hardware Medium Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19205/35/execution/node/1621/log

daosbuild3 · 2026-10-22T16:22:24Z

Test stage Functional on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19205/36/execution/node/1363/log

daosbuild3 · 2026-10-22T17:03:46Z

Test stage Unit Test completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-19205/38/testReport/

daosbuild3 · 2026-10-22T17:17:23Z

Test stage NLT completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-19205/38/display/redirect

daosbuild3 · 2026-10-22T17:50:07Z

Test stage Unit Test completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-19205/39/testReport/

daosbuild3 · 2026-10-22T18:23:48Z

Test stage Unit Test with memcheck completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-19205/39/testReport/

daosbuild3 · 2026-10-22T20:38:12Z

Test stage Functional on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-19205/39/execution/node/1379/log

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

Add an injectable scheduler-sequence callback to standalone VOS TLS and build the scrubber test with VOS_STANDALONE. Add CSUM_SCRUBBING_15 test to delete and aggregate the current object during a yield between checksum chunks, then run GC before iteration resumes. Poison freed PMEM allocations through DAOS fault injection to expose use-after-free, and verify the scrubber skips the deleted object and continues processing later objects. Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

Include the affected OID in checksum corruption logs and the RAS event to make it easier to identify the corrupted object. skip-nlt: true skip-test: true skip-test-hardware: true Test-tag: pr,-unittest Priority: 2 Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

daosbuild3 · 2026-10-22T21:59:56Z

Test stage Unit Test completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-19205/43/testReport/

daosbuild3 · 2026-10-22T22:18:40Z

Test stage Unit Test with memcheck completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-19205/43/testReport/

grom72 · 2026-10-23T06:36:57Z

Merged into #19182

grom72 force-pushed the grom72/DAOS-19721 branch from 4356175 to 0cfb094 Compare October 6, 2026 15:27

grom72 force-pushed the grom72/DAOS-19721 branch from 0cfb094 to bddc4f7 Compare October 6, 2026 18:11

grom72 force-pushed the grom72/DAOS-19721 branch from f127f3b to d3a0a57 Compare October 6, 2026 19:08

grom72 force-pushed the grom72/DAOS-19721 branch from a29569a to 9543aa7 Compare October 7, 2026 15:23

grom72 force-pushed the grom72/DAOS-19721 branch from 9543aa7 to 1cec1da Compare October 12, 2026 15:20

grom72 requested a review from janekmi October 22, 2026 08:06

janekmi requested changes Oct 22, 2026

View reviewed changes

Comment thread src/vos/tests/pool_scrubbing_tests.c Outdated

Comment thread src/vos/tests/pool_scrubbing_tests.c Outdated

grom72 requested a review from janekmi October 22, 2026 08:52

janekmi requested changes Oct 22, 2026

View reviewed changes

Comment thread src/vos/tests/pool_scrubbing_tests.c Outdated

grom72 requested a review from janekmi October 22, 2026 14:31

janekmi requested changes Oct 22, 2026

View reviewed changes

grom72 requested a review from janekmi October 22, 2026 16:20

janekmi previously approved these changes Oct 22, 2026

View reviewed changes

grom72 dismissed janekmi’s stale review via 88c1cb7 October 22, 2026 17:15

grom72 force-pushed the grom72/DAOS-19721 branch from 88c1cb7 to d5c0a32 Compare October 22, 2026 20:36

DAOS-19721 vos: add memory poisoning for some tests

a8f880b

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

grom72 force-pushed the grom72/DAOS-19721 branch from d5c0a32 to e0e0cdc Compare October 22, 2026 21:11

grom72 added 2 commits October 22, 2026 21:15

grom72 force-pushed the grom72/DAOS-19721 branch from e0e0cdc to dd8ef79 Compare October 22, 2026 21:16

Merge branch 'master' into grom72/DAOS-19721

556f416

Signed-off-by: Tomasz Gromadzki <tomasz.gromadzki@hpe.com>

grom72 closed this Oct 23, 2026

Conversation

grom72 commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Steps for the author:

After all prior steps are complete:

Uh oh!

github-actions Bot commented Oct 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

daosbuild3 commented Oct 5, 2026

Uh oh!

daosbuild3 commented Oct 6, 2026

Uh oh!

daosbuild3 commented Oct 6, 2026

Uh oh!

daosbuild3 commented Oct 6, 2026

Uh oh!

daosbuild3 commented Oct 6, 2026

Uh oh!

daosbuild3 commented Oct 6, 2026

Uh oh!

daosbuild3 commented Oct 6, 2026

Uh oh!

daosbuild3 commented Oct 6, 2026

Uh oh!

daosbuild3 commented Oct 6, 2026

Uh oh!

daosbuild3 commented Oct 6, 2026

Uh oh!

daosbuild3 commented Oct 6, 2026

Uh oh!

daosbuild3 commented Oct 6, 2026

Uh oh!

daosbuild3 commented Oct 7, 2026

Uh oh!

daosbuild3 commented Oct 7, 2026

Uh oh!

daosbuild3 commented Oct 7, 2026

Uh oh!

daosbuild3 commented Oct 7, 2026

Uh oh!

daosbuild3 commented Oct 12, 2026

Uh oh!

daosbuild3 commented Oct 12, 2026

Uh oh!

daosbuild3 commented Oct 12, 2026

Uh oh!

daosbuild3 commented Oct 12, 2026

Uh oh!

daosbuild3 commented Oct 12, 2026

Uh oh!

daosbuild3 commented Oct 12, 2026

Uh oh!

daosbuild3 commented Oct 12, 2026

Uh oh!

daosbuild3 commented Oct 22, 2026

Uh oh!

janekmi left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

daosbuild3 commented Oct 22, 2026

Uh oh!

daosbuild3 commented Oct 22, 2026

Uh oh!

daosbuild3 commented Oct 22, 2026

Uh oh!

daosbuild3 commented Oct 22, 2026

Uh oh!

daosbuild3 commented Oct 22, 2026

Uh oh!

grom72 commented Oct 5, 2026 •

edited

Loading

github-actions Bot commented Oct 5, 2026 •

edited

Loading

janekmi left a comment •

edited

Loading