Skip to content

nvidia-gpu-tests: optionally fail the play on a failed DCGM diagnostic - #1436

Open
saqibkh wants to merge 1 commit into
NVIDIA:masterfrom
saqibkh:gpu-tests-fail-on-dcgm-error
Open

saqibkh wants to merge 1 commit into
NVIDIA:masterfrom
saqibkh:gpu-tests-fail-on-dcgm-error

Conversation

@saqibkh

@saqibkh saqibkh commented Oct 23, 2026 •

Copy link
Copy Markdown

As suggested in #1434. Every test in roles/nvidia-gpu-tests runs with ignore_errors, so a failing GPU never fails the play. This adds gpu_test_dcgm_fail_on_error, off by default. When it is set, a non-zero exit from dcgmi diag fails the play. The default behaviour doesn't change, and the other tests keep ignore_errors. The README line for the DCGM step mentions the option.

I tested it with ansible-core 2.21, running the role's DCGM task with a stub dcgmi on the PATH: with the option off a failing diagnostic is ignored and the play continues; with it on and a passing diagnostic the play continues; with it on and a failing one (the stub exits 226) the play fails. ansible-lint --offline roles/nvidia-gpu-tests passes.

Signed off with git commit -s.

Every test in the role runs with ignore_errors, so a failing GPU never
fails the play. Add gpu_test_dcgm_fail_on_error (off by default); when
set, a non-zero exit from dcgmi diag fails the play.

Signed-off-by: Saqib Khan <11544614+saqibkh@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant