Large-Scale Data Version Control for HPC with Git and DataLad

Published

October 6, 2026

Session lecturer

  • Dr. Andreas Knüpfer
  • Head of Scientific Computing Core CASUS Center for Advanced Systems Understanding, Görlitz Helmholtz-Zentrum Dresden-Rossendorf e.V. (HZDR)
  • Website

Who is this for?

  • Computational scientists with small/medium/large data volumes of input and output data on HPC clusters.
  • Scientists who are familiar with git and collaborative software development and want to use the same tools for research data (or any kind of data) including large, binary files.

What will you learn?

  • Learn how to version control research data (or any kind of data) with git including large, binary files
  • Learn how to work with data repositories in a collaborative fashion with a git forge
  • Learn how to add machine-actionable reproducibility to a data repository
  • This means how to track how the code (under git version control) is generating/modifying data (under the same git version control) and provide an fully automatic way to re-execute it.
  • Learn how to bring the three previous skills to an HPC environment with Slurm, because it requires some adaptations.
  • There are some added benefits for the typical life-cycle of HPC projects.

Why is this important?

  • This is part of F.A.I.R. data management (especially the Accessible and Reusable aspects), see https://www.go-fair.org/fair-principles/.
  • It adds full version control for data which is beyond the F.A.I.R. requirements. This allows to adopt the collaborative git / git forge work style for data repositories and workloads executed on HPC environments.
  • Reproducibility of scientific results is still a challenge. The presented approach allows to combined code and data repositories with full reproducibility instructions embedded in the git log. Such a repository can be an entirely self-contained reproducibility appendix for your next paper.

How is the session structured?

  • Mix of presentation and hands-on parts, some on the own laptop and some on the HPC cluster

Prerequisite

Attending (or knowledge of the content) in: