[saqibkh]

Every test in roles/nvidia-gpu-tests runs with ignore_errors: yes, so the play finishes green even if a GPU fails them. This adds one opt-in step that can fail the play.

With gpu_test_pantheon: yes the role runs pantheon --test quick --gpu all (Pantheon, pip install pantheon-gpu) and prints each GPU's verdict. Pantheon exits non-zero when it detects a fault, and the task is not marked ignore_errors, so the play fails. The option is off by default, so nothing changes for current users. The suite, the seconds per workload and the output directory are variables, and the README lists the new requirement.

I wrote Pantheon, so I'm not neutral about it. If you'd rather not carry a third-party tool in this role, I understand.

What I ran, with ansible-core 2.21 on a machine with two RTX 3060s, using only the new tasks (I did not run the role's older steps, which need DCGM, Docker and a TensorFlow image):

• default settings: all three tasks skipped.
• gpu_test_pantheon: true: both GPUs reported HEALTHY after about 70 seconds and the play passed.
• an invalid suite name (Pantheon exits 1): the play fails on the Pantheon task.

I did not test on DGX or on A100/H100 nodes, and I did not test with AMD GPUs.

Signed off with git commit -s.

[saqibkh]

A note on testing, since the description only lists what I didn't do.

On real hardware I ran the new tasks with ansible-core on two RTX 3060s: both GPUs reported HEALTHY in about 70 seconds and the play passed. An invalid suite name (Pantheon exits 1) failed the play, and with the option off all three tasks were skipped. I did not run the role's older steps (DCGM, Docker, the TensorFlow image).

I also tried the tasks on simulated A100, H100 and MI300X cards on hosted runners (PantheonSim). That tells us little. On the simulated A100 and H100, Pantheon reported a fault and exited 1 and the play failed as it should, but only because Pantheon's kernels link the CUDA runtime statically, which the simulator can't host, so there was no healthy run. The simulated MI300X job exhausted the runner and was killed. I have not run it on real DGX, A100/H100 or AMD hardware.

[dholt]

Thanks for this, and for being upfront that you wrote Pantheon. We're going to pass on adding it to this role: we'd rather not take on a new third-party dependency here, and the role already runs the DCGM diagnostics for GPU health.

You've found a real gap, though. Every test in this role runs with ignore_errors, so a failing GPU never fails the play. We'd welcome a small PR that adds an opt-in variable (off by default) so a failed DCGM diagnostic fails the play. Thanks again for the careful testing notes.

[saqibkh]

Understood, and thanks for the clear answer. I've opened the DCGM option as a separate PR, as you suggested.