Massively Parallel Job Submission on Supercomputers
Who is this for?
Users who execute many independent small jobs simultaneously and want to speed up their work and use the hardware resource efficiently.
Examples include:
It is not possible to combine multiple runs/tasks into a single application.
Large parameter sweep based on the same application or workflow with dependent pipeline steps.
Workflow involves runs that are depending on several independent software which do not scale beyond a certain number of cores.
What will you learn?
You will learn how to easily and efficiently overcome the issues mentioned above. More specifically, you will learn about two strategies:
- SLURM arrays
- QCG-PilotJob
Through hands-on exercises you will see how to work with both options and when you should use them.
Why is this important?
Researchers often have to conduct parameter sweeps or Monte-Carlo type of computational research.
Doing it in an efficient manner such as is suggested here will speed up your work and will make better use of the computational resources.
How is the session structured?
Mixture of presenter led explanations and hands-on exercises. Users will have access to the National Supercomputer Snellius, therefore gaining experience with a state-of-the-art computing system.
Prerequisite
Attending (or knowledge of the content) in: