Skip to content

Fail fast on Flux MPI launches outside an allocation - #1972

Closed
sbryngelson wants to merge 3 commits into
MFlowCode:masterfrom
sbryngelson:fix-flux-test-login
Closed

sbryngelson wants to merge 3 commits into
MFlowCode:masterfrom
sbryngelson:fix-flux-test-login

Conversation

@sbryngelson

@sbryngelson sbryngelson commented Oct 9, 2026 •

Copy link
Copy Markdown
Member

Description

Fixes #1967

On Flux systems, MPI launches outside an allocation (login node) submit a separate pbatch job per launch (default 12 h), so ./mfc.sh test queues one job per case and appears to hang. The templates now check FLUX_ENCLOSING_ID (set inside flux alloc/flux batch, unset on the login node) and stop with a clear message:

Not inside a Flux allocation: each launch would be submitted as its own batch job. Run inside 'flux alloc ...', use '-e batch', or pass --no-mpi.

  • toolchain/templates/include/helpers.mako: new require_flux_allocation helper.
  • tuo.mako: check for interactive MPI runs.
  • default.mako: same check when the selected launcher is srun and flux exists (on Tuolumne srun is the Flux wrapper; mfc.sh test uses the default template, which is what produced the pbatch jobs in the issue).

Testing

  • Login node, before (master): ./mfc.sh test -j 4 --no-gpu --only Cylinder launched pbatch jobs (named 1) via the flux srun wrapper (cancelled/failed immediately).
  • Login node, after: same command fails each case immediately with the message above; flux jobs shows no pending/running jobs.
  • Inside flux batch on pdebug: ./mfc.sh test -j 4 --no-gpu --only 6ED50013 passes.
  • Login node: ./mfc.sh test --no-mpi --no-gpu -j 4 --only 6ED50013 passes (unaffected).
  • ./mfc.sh format and ./mfc.sh precheck pass.

Not verified: mfc.sh run -c tuo interactive path directly (same helper), batch engine, non-Flux systems. With mfc.sh test each selected case reports the failure rather than aborting on the first (the harness's existing early-abort applies after 20 cases). Interactive run fails fast rather than only warning.

Contribution Policy

All contributions are expected to demonstrate:

  • A clear understanding of the codebase
  • Alignment with product direction
  • Thoughtful reasoning behind changes
  • Evidence of real-world usage or hands-on experience with the problem

If these expectations are not met, we would prefer to implement the changes ourselves rather than spend time reviewing low-effort submissions.

@sbryngelson

Copy link
Copy Markdown
Member Author

Closing this for now to bring the number of open PRs from me down to four, so the ones that stay open get real review attention rather than being lost in a queue of 35.

No work is lost and nothing needs re-doing. The branch fix-flux-test-login remains on the fork at 95dea44b38, this PR's diff and discussion stay here, and the head commit is additionally archived in a local verified bundle. Reopening takes no force-push and no rebase:

gh pr reopen 1972 -R MFlowCode/MFC

I intend to reopen these in small batches as the open slots free up. Apologies for the notification noise.

Opened and closed with Claude Code.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

mfc.sh test on a Tuolumne login node submits every test as a 12 h pbatch job

1 participant