Repository navigation
Conversation
41-lastuserjob-ssh used 'grep -w' for the localusers.backup exemption and an
unanchored 'sed "/${SLURM_JOB_USER}/d"' on /etc/localusers. Both match
substrings and regular expressions: a job user 'al' removed the SSH allow entry
of 'alice' and 'halley', a dot matched any character, and 'je' was treated as
exempt when 'je.kim' was listed. Match the exact line, literally, in both
places (the same -x -F form scripts 40 and 42 already use).
Adds six cases to tests/slurm-epilog/run-tests.sh.
Signed-off-by: Doug Holt <dholt@nvidia.com>
…e hatch (#1407) The *-lastuserjob-* epilog scripts decided what to remove from squeue alone and killed every process the user owned, so a job of the same user placed on the node just after squeue answered (CompleteWait=0), or one the controller was slow to report, could lose its processes. New role variable slurm_epilog_cleanup_scope, rendered at deploy time: job (default) The dispatcher also scans /proc/<pid>/cgroup for the user's processes in any job_<id> cgroup other than the ending job's (cgroup v1 and v2 layouts) and skips the last-user-job scripts when it finds one. 40-lastuserjob-processes replaces killall -9 -u with a PID list classified by cgroup: the ending job's processes are killed, another job's never, orphans only when the user has no other job. 42 skips when job_container/tmpfs is configured. Anything that cannot be read or recognised is treated as 'another job may be here': the cleanup is deferred, never guessed, and the epilog never fails. user DEPRECATED, removed in 27.01. Renders the 26.09 scripts byte-for-byte. The playbook validates the value and warns when it is user. Signed-off-by: Doug Holt <dholt@nvidia.com>
…hot (#1407) Extends tests/slurm-epilog/run-tests.sh with a fake /proc and a recording kill so the job-scoped paths run unprivileged: dispatcher guard (cgroup v1 and v2, other user, orphan, unrecognised, empty, missing library, no uid), script 40 per-PID classification, 42 job_container/tmpfs skip, 50 behind the guard, and a snapshot check that scope=user renders the 26.09 scripts byte-for-byte. render.py takes the scope and matches Ansible's trim_blocks. Signed-off-by: Doug Holt <dholt@nvidia.com>
…_container/tmpfs (#1407) Signed-off-by: Doug Holt <dholt@nvidia.com>
Signed-off-by: Doug Holt <dholt@nvidia.com>
…ed outside Slurm (#1407)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the cleanup half of #1407 (does not claim the whole issue is fixed; see "Limits").
What changes
The Slurm epilog cleanup scripts now work per job. When one job of a user ends, the processes,
/tmpfiles and enroot containers of that user's other running jobs are no longer removed. Leftovers are still removed after the user's last job ends. Processes are classified through/proc/<pid>/cgroup(cgroup v1 and v2): ending job = kill, other job = never, unassigned = only when it is the user's last job. Anything unreadable defers the cleanup (logged) and the epilog never fails.slurm_epilog_cleanup_scope, defaultjob.userrestores the 26.09 scripts and is deprecated, to be removed in 27.01.job_container/tmpfsis documented, not rebuilt; script 42 skips when it is configured.alalso removedalice,.matched any character). It now matches whole lines. Applies in both scopes.docs/slurm-cluster/epilog-job-scoped-cleanup.md.scope=useris byte-identical to 26.09Rendered with the real Ansible
templatemodule for both trees;diff -ris empty forepilog.sh,run-parts.sh,40,42,50(script 41 is the intended exception, separate commit):The suite pins this with snapshots under
tests/slurm-epilog/snapshots/user-scope/.Testing
tests/slurm-epilog/run-tests.sh: 68 passed, 0 failed (was 20). Deliberate breakages of the templates turn it red.Limits
proctrack/cgroup(the DeepOps default). With another tracker every process counts as unassigned and the cleanup falls back tosqueue.