Repository navigation
Conversation
… in a container shell right away, or with hooks that drop you at the start of the loop over easystack files, or after an eb command has finished with non-zero exit. Also support debugging of accelerator builds by selecting a single accelerator - even if the initial build was multi-accelerator. Finally, make sure that we can resume from a build tarball, so that if a lot of dependencies have already been built, we don't have to redo it.
… to reproduce the failure
…r that the preferred way of debugging is to start at the bot-build.slurm level
…rget or the tarball to resume from up front
Co-authored-by: Caspar van Leeuwen <33718780+casparvl@users.noreply.github.com>
Co-authored-by: Caspar van Leeuwen <33718780+casparvl@users.noreply.github.com>
…n interactive shell. Then, use read -p, and since this prints to stdout, do the whole selection conversation on stdout to avoid spreading it out
…arget / tarball Simplify the selection of the accelerator target and the tarball to resume from in interactive builds (EESSI_INTERACTIVE_BUILD): don't infer anything. - If EESSI_INTERACTIVE_BUILD_ACCELERATOR / EESSI_INTERACTIVE_BUILD_RESUME_FROM is set, use it as given. - If not, ask the user to choose from the available options if stdin is a terminal, and stop with an error naming the variable to set if it is not. - Only choices with a single option are not asked: a single accelerator target, or no tarballs to resume from. We still ask whether to resume from a tarball or start from scratch if there is at least one tarball. This removes the pairing of tarballs with accelerator targets, the suggested defaults and the warnings about assumptions, position_of() and the canonicalization of the tarball paths that was only needed for lookups. The options are now only listed when we need to ask for them. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…TOR_TARGET_OVERRIDE EESSI_INTERACTIVE_BUILD_ACCELERATOR now takes the full name (e.g. accel/nvidia/cc80), consistent with EESSI_ACCELERATOR_TARGET_OVERRIDE and with the names listed when asking. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
See how much we save by simplifying...
|
I tried this out using the RUG Zen5 bot that builds for RTX6000 acel builds and it worked fine. Note: I didn't easily find a failed build, so I tried the One thing I noticed, after interrupting a build ( |
|
Testing this now for LAMMPS. It works from the bot account like a charm. But if you try running it from another account you run into trouble. |
|
Oh this is anoying when you exit |
|
Is it a bug, or is it a feature? Yes, the build continues after you exit the debug shell. It is essentially a breakpoint that enables you to play around, then continue the build process. Maybe we can make that behaviour dependent on (another) environment variable, I don't know how EESSI that will be. For now, I think this is better than nothing - you might actually want to continue the build process, because you want to know if you encounter more bugs, or if you fixed anything by whatever you did interactively. I agree though that it is extremely annoying that a ctrl+C doesn't work. I'm not sure why that is happening, and if it can easily be resolved. I'll check with my AI friend.... |
The debug shell has job control, so it makes its own process group the terminal's foreground group. That group was left behind as the foreground group after the shell exited, so Ctrl+C no longer reached the build (observed: TPGID pointing at an empty process group). Explicitly set our own process group as the foreground group again. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
Interactive build: give the terminal back after the debug shell exits
|
Tried to fix it, it didn't work. The AI couldn't replicate it outside our container+startprefix environment anyway, so it was a bit of guess work in debugging. I'll look further next week. |
3 participants
Problem
We often have issues reproducing build failures interactively. Currently, we have about three options:
EESSI-extend. This is the easiest, but also the most 'different' from the bot's build environment. It uses EESSI for dependencies, but it doesn't run in a build container.bot/inspect.shfrom the job dir of the failed build job. This script was designed to mimicbot/build.shand drop you into an interactive shell in the failed build environment. However, it is very much outdated, and keeping it in sync currently requires a lot of code duplication frombot/build.sh. An example of how out-of-sync it is: it does not have any GPU support (no NVIDIA GPU support either).bot/build.sh. It is also very time-consuming.Proposed solution
The solution presented here is to tightly integrate an option for interactive debugging into the
bot/build.sh, that will drop you into an interactive debugging shell ifEESSI_INTERACTIVE_BUILDis set. To get an environment that is as similar as possible, one should call the job script that the bot submits interactively. I.e. if a bot is configured to use the default, that would beEESSI_INTERACTIVE_BUILD=<mode> eessi-bot-software-layer/scripts/bot-build.slurm.modesupports three options:The idea is that this is ran from the job directory left behind by the bot, with the failing build. By default, it will resume from the latest tarball of the build step (this is selectable). If accelerators are used, it will run for the last accelerator in the list by default (this is also selectable).
Limitations
shared_fs_directorycan not be updated (what's there can be used as long as your user has read permissions to that dir).bot/build.shfromsoftware-layerhard-clones and links thesoftware-layer-scripts/bot/build.sh. That results in an exit, because the script runs withset -e. My current hack is just to replace thebot/build.shin the job dir with a one-liner:software-layer-scripts/bot/build.sh. We should fix this more properly, and maybe make the whole clone-business conditional onEESSI_INTERACTIVE_BUILDnot being set.