The GADGET-4 (GAlaxies with Dark matter and Gas intEracT - 4) is the fourth iteration of the gravitational N-body and smoothed-particle hydrodynamics (SPH) simulation code written by Volker Springel and was released in 2020. While GADGET-3 never made it to have a public release, GADGET-4 is the next publicly available version of the suite after GADGET-2 from 2005. This tutorial covers how to install GADGET-4 and run a simulation with it on a Linux system.
GADGET-4 is (unsurprisingly) similar to GADGET-2 in many aspects, which I already had a tutorial on how to install and run. Its main purpose, software design and structure, as well as its usage are very similar to its predecessor. GADGET-4 realizes a hybrid cosmological N-body and hydrodynamics simulation that can be used to study the formation and evolution of galaxies and large-scale structures in the Universe. The code is written in ISO C++, although inherits many code segments and patterns from earlier GADGET versions written in C.
In this guide I will show how to correctly install GADGET-4 on a Linux system, and how to run a simple example simulation with it. As simulations depend on the set of compile-time options and the initial conditions, this tutorial alone cannot serve as a comprehensive guide to all possible simulation setups as-is. However, I will cover the common steps that you will encounter during the installation and running of GADGET-4.
GADGET-4 is built on top of the same non-standard software libraries and tools as GADGET-2 does. To successfully compile GADGET-4, you need to obtain, compile and then link these libraries appropriately during the building process. The following list contains all necessary software that you will need to download and install before trying to install GADGET-4 itself:
You can download all of these files by clicking on the links above, or by using e.g. the wget or curl commands in the terminal. (Or you can just search for them yourself on the internet, of course.) When choosing an MPI library, it should not really matter which one you go with. Moreover, on computation servers, usually one or both of them are already installed on the system. In this tutorial I am going to use OpenMPI, but the steps should be identical in case of using MPICH too. Using wget you can download all of them at once:
$ wget https://download.open-mpi.org/release/open-mpi/v4.1/openmpi-4.1.5.tar.gz
$ wget https://ftp.gnu.org/gnu/gsl/gsl-2.8.tar.gz
$ wget http://www.fftw.org/fftw-3.3.10.tar.gz
$ wget https://github.com/HDFGroup/hdf5/releases/download/hdf5_1.14.4.3/hdf5-1.14.4-3.tar.gz
$ wget https://download.open-mpi.org/release/hwloc/v2.10/hwloc-2.10.0.tar.gz
If you downloaded all tarballs from the URLs above, now you can extract, compile and install them. The installation of these libraries is a standard procedure that you can find in this tutorial.
--enable-mpi --enable-float --enable-threads flags during the configuration step! More info about them and FFTW in the tutorial.
Alternatively, on Debian-based systems like Ubuntu, if you have root access to the machine, you can install the dependencies using the apt package manager. However, the versions of these packages might be outdated, as well as elevated permissions on a machine are not available to an end user in most circumstances, so it is generally recommended to install them from source. If you still want to install them using apt, you can do it with the following command:
$ sudo apt install openmpi-bin libopenmpi-dev libhdf5-dev hwloc libgsl-dev libfftw3-dev
After you installed all dependencies, you can proceed to the next section.
The installation of GADGET-4 is somewhat more complex that GADGET-2, but it is still a straightforward process. The main steps are the configuration of the compile-time flags in Config.sh, correctly linking the necessary dependencies in multiple other configuration files, and finally building GADGET-4 with make. Download GADGET-4 from the official repository to e.g. a dedicated apps folder in your home directory, but you can download it to wherever you want it:
$ git clone https://gitlab.mpcdf.mpg.de/vrs/gadget4.git ~/apps/GADGET-4
$ cd ~/apps/GADGET-4
For convenience and clarity, somewhere between the release of GADGET-2 and GADGET-4, the Makefile of GADGET-2 was separated and organized into multiple files, which I will showcase in the following sections. The very first thing you have to do is copying the Template-Config.sh and Template-Makefile.systype files and renaming these copies in the root directory of GADGET-4:
$ cp Template-Config.sh Config.sh # although we will overwrite this in this guide
$ cp Template-Makefile.systype Makefile.systype
The Config.sh file contains all the compile-time options of GADGET-4 that can tune the behavior of the code for all simulations performed with it. On the other hand, the Makefile.systype file should be used in tandem with the contents of the buildsystem directory and the Makefile in the root directory of GADGET-4 to define the appropriate compiler and linker flags for your system.
First focus on the system-specific setups as those only depend on your system, and generally independent of the simulations you want to run. The goal is to define the necessary build flags and link the dependencies correctly for the Makefile of GADGET-4. There are four files you have to pay attention to: Makefile.systype, buildsystem/Makefile.comp.*, buildsystem/Makefile.path.* and the actual Makefile.
Makefile.systypeThe Makefile.systype file is a simple file, where the only thing you have to do is to define the system type of your machine. This is a short string that is used to indicate for the Makefile, which system-specific configuration file to use from the buildsystem directory. There is already a number of system types defined in it (along with their configuration files and setting in the buildsystem folder and in the Makefile) that corresponds to some well-known supercomputers in the world, where GADGET was commonly used. To create a custom system type, just define a new string (e.g mycomputer as in this example) and comment out the others in the Makefile.systype file:
# Select Target Computer
#
# Please copy this file to Makefile.systype and uncomment your
# system. Don't commit changes to this file unless you add support for
# a new system.
SYSTYPE="mycomputer"
#SYSTYPE="Generic-gcc"
#SYSTYPE="Generic-intel"
#[... other entries commented out ...]
buildsystem filesThe buildsystem directory contains all the system-specific configuration files and some related scripts. For supported systems in the Makefile.systype file, there is also a corresponding Makefile.comp.* and Makefile.path.* file in the buildsystem directory. The Makefile.comp.* files contain the compiler flags that are used to build GADGET-4, while the Makefile.path.* files contain the linker flags with the correct paths to the necessary libraries (GSL, FFTW, etc.) and include files that are used during the build process.
Because we defined a custom system in the Makefile.systype file in the previous step, now we have to define new or use existing Makefile.comp.* and Makefile.path.* files in the buildsystem directory.
While the default Makefile.comp.gcc file is a good starting point for most cases, a new Makefile.path.* file should created to correctly link the dependencies installed on the machine. In this guide I will use the Makefile.comp.gcc file as-is, but I will create a new Makefile.path.mycomputer file in the buildsystem directory to link the dependencies correctly.
If you followed the tutorial on the installation of dependencies from source and installed all of them to their own separate folders in e.g. the directory /home/username/opt, then the setup of the Makefile.path.mycomputer file should look like this:
GSL_INCL = -I/home/username/opt/gsl/include
GSL_LIBS = -L/home/username/opt/gsl/lib
FFTW_INCL = -I/home/username/opt/fftw/include
FFTW_LIBS = -L/home/username/opt/fftw/lib -lfftw3
HDF5_INCL = -I/home/username/opt/hdf5/include
HDF5_LIBS = -L/home/username/opt/hdf5/lib -lhdf5
HWLOC_INCL = -I/home/username/opt/hwloc/include
HWLOC_LIBS = -L/home/username/opt/hwloc/lib
Evidently, replace username with your own username on the machine. Variables ending in _INCL define the include paths of the libraries, while variables with the _LIBS suffix define the linker flags and the paths to the libraries themselves. The -lfftw3 and -lhdf5 flags are necessary to link the FFTW and HDF5 libraries, respectively.
If you have root access to the machine you are using and you installed the dependencies using apt on Debian-based system (like Ubuntu) and the dependencies are installed to the default system directories, then the Makefile.path.mycomputer file should look like this:
GSL_INCL = -I/usr/include/gsl
GSL_LIBS = -L/usr/lib/x86_64-linux-gnu
FFTW_INCL = -I/usr/include
FFTW_LIBS = -L/usr/lib/x86_64-linux-gnu -lfftw3
HDF5_INCL = -I/usr/include/hdf5/serial
HDF5_LIBS = -L/usr/lib/x86_64-linux-gnu/hdf5/serial -lhdf5
HWLOC_INCL = -I/usr/include/hwloc
HWLOC_LIBS = -L/usr/lib/x86_64-linux-gnu/hwloc
MakefileThe Makefile in the root directory of GADGET-4 is the main file that is used to build the code. It is the place, where every system-specific setup from the other files are linked together. There are two sections in this file that requires your attention: the system definitions, referenced by the SYSTYPE variable from Makefile.systype and the Python interpreter.
Latter is just a simple variable that defines the Python executable used during the compilation of GADGET-4. The default value of this variable is python. Although the executable of Python 3.x versions is called python3, this is mapped to python on many Linux distributions and systems. However, this is not always the case and python3 should be explicitly called here. You can change this in the Makefile file on line 97, if necessary:
PYTHON = python # Could be python3 on some systems
In the other part, the corresponding buildsystem files (one with the compiler flags and one with the linker flags for the dependencies) are linked to the predefined system from Makefile.systype. Since we defined an entirely new system called mycomputer in Makefile.systype, we have to tell the Makefile which buildsystem files correspond to this. Remember, we only created a new file for the linker flags (Makefile.path.mycomputer) and used the default one for the compiler flags (Makefile.comp.gcc). To include the new system in the Makefile, you have to add the following to the list of system definitions:
##########################
#define available Systems#
##########################
ifeq ($(SYSTYPE),"mycomputer")
include buildsystem/Makefile.path.mycomputer
include buildsystem/Makefile.comp.gcc
endif
ifeq ($(SYSTYPE),"Generic-gcc")
# [... other system definitions ...]
The Config.sh file contains all the compile-time options of GADGET-4 that can tune the general behavior of the code for all simulations performed with it. These include options like defining the boundary conditions, selecting which physical effects to simulate or consider, miscellaneous technical and debug options, etc. The file is well-documented, and you can find all possible options and their descriptions in it.
GADGET-4 offers a plethora of options that can be freely combined to create a custom simulation setup for a wide range of astrophysical problems. One always has to consider what kind of simulation is going to be run with GADGET-4, and set these options accordingly. If a new type of simulation have to be run, then the options need to be adjusted and GADGET-4 has to be recompiled!
In this guide, I will show how to set up one of the example simulations included in the examples directory with GADGET-4. For convenience, I selected the legacy galaxy collision simulation (examples/G2-galaxy), as it is much less resource intensive than the more robust example (CollidingGalaxiesSFR) included in GADGET-4. All example simulations have their own already prepared Config.sh file in their respective directories, and you can copy them to the root directory of GADGET-4 to compile the code for that specific simulation. Here, just copy the Config.sh file from the examples/G2-galaxy directory to the root directory of GADGET-4:
$ cd ~/apps/GADGET-4
$ cp ./examples/G2-galaxy/Config.sh ./
With the correct buildsystem files, Makefile.systype, Makefile and Config.sh in place, GADGET-4 can be built with make command from the same directory where the Makefile is located. The output of the make command can be redirected to a file, so you can check for any errors or warnings that might occur during the build process. It can be also sped up by utilizing more CPU cores with the -jN option, where N is the desired number of cores. All of these can be done with the following command:
$ cd ~/apps/GADGET-4
$ make -j4 2>&1 | tee m.out
If everything went as well, then this command will automatically create an executable called Gadget4 in this same directory, which can be finally used to run simulations.
GADGET simulations require at least three basic components:
Makefile and the corresponding Config.sh file.$ mkdir -p ~/data/simulations/galaxy
Just for convenience, I copy the compiled executable to this directory. Besides that, I will also copy the parameter file into the same directory; we will need to edit this later, so we can preserve the content of the original file this way.
$ cp ~/apps/GADGET-4/Gadget4 ~/data/simulations/galaxy/
$ cp ~/apps/GADGET-4/examples/G2-galaxy/param.txt ~/data/simulations/galaxy/
Creating physically meaningful and correct initial conditions for an astrophysical simulation is a complex task that requires its own set of methodologies and thus usually separate software tools. IC creation is out of the scope of this tutorial, however, all examples in GADGET-4 come with their own IC files that can be used to run simulations. They can be downloaded from the official repository of GADGET-4 by clicking this link or by using the commands below. For the sake of this tutorial I will put these files into the ~/data directory created earlier, but of course, you can put these files anywhere you want:
$ cd ~/data
$ curl -O https://wwwmpa.mpa-garching.mpg.de/gadget4/example_ics.tar
$ tar -xvf example_ics.tar && rm example_ics.tar
This will result us in a new directory called ExampleICs in the ~/data folder with all the IC files of the example simulations. The IC file for the galaxy collision simulation is called galaxy_littleendian.dat, which we will use later.
The parameter file is the single input file of the Gadget4 executable that contains all the necessary run-time parameters of the simulation, like the location of the IC file and the output directory, values of main cosmological parameters, integration time step and smoothing lengths, etc. Since in the GADGET-2 tutorial I covered the setup and execution of the same example simulation, this section will be close to identical to that.
Before running the simulation, there are some parameters which needs to be altered first in the copied simulations/galaxy/params.txt file. The first two lines after the initial comment line should be changed to contain the absolute paths of the IC input file (InitCondFile) and the output directory (OutputDir). In this example case these are the following:
%---- Relevant files
InitCondFile /home/username/data/ExampleICs/galaxy_littleendian.dat
OutputDir /home/username/data/simulations/galaxy/output
Note that I defined an additional output folder in the ~/data/simulations/galaxy directory to store the output of the simulation. This is a good practice, as the simulation will create a lot of files during the run, and it is better to keep them in a separate folder.
As any other computer simulation, GADGET-4 works by reading in a set of ICs and evolving this state in time. During a run at given time intervals, GADGET-4 saves a ‘snapshot’ of the current state of the system that contains the positions, velocities and other attributes of the simulated particles. GADGET-4 supports multiple different file formats, name its own Gadget-I and II formats and the more modern HDF5 format.
I recommend using HDF5 in general, as it is a much more modern, versatile and most importantly maintained file format and, which is much easier to handle with e.g. Python. To change the output format, you will need to find the %---- File formats block in the parameter file and change the SnapFormat value to 3. This will tell GADGET-4 to save the snapshot files in HDF5 format. Since the prepared example IC is in Gadget format, the ICFormat value should be set to 1 to indicate this. The final %---- File formats block in the parameter file should look like this:
%---- File formats
ICFormat 1
SnapFormat 3
The last parameters you might want to change are the ones that control the length and the output frequency of the simulation. The meaning of space and time in GADGET and its derivatives is not physical, at least not directly. When we mention the unit of distance, velocity, mass or time in this context, we actually mean simulation units. When an IC is created for a simulation, one chooses some unit length, unit velocity and unit mass to define the physical meaning of the numerical values in the IC file. In GADGET, the unit time is derived from these units as UnitTime_in_s = UnitLength_in_cm / UnitVelocity_in_cm_per_s. Unit length, velocity and mass is controlled in the parameter file under the System of units section. Considering the galaxy collision example, the unit length UnitLength_in_cm is set to 3.085678e21, which corresponds to $1 \mathrm{kpc}$. Similarly, the unit velocity UnitVelocity_in_cm_per_s is set to 1.0e5, which corresponds to $1 \mathrm{km/s}$. This gives us a unit time UnitTime_in_s of 3.085678e16, which corresponds to $\approx 0.9785 \mathrm{Gyr}$.
GADGET-4 will create a snapshot at every time interval defined by the TimeBetSnapshot parameter. The shorter this interval is, the more frequently the current state of the simulation will be saved to the disk, resulting in a more detailed “timeline” at the cost of using more disk space. This can be edited in the Output freqency block in the line
%---- Output frequency and output paramaters
...
TimeBetSnapshot 0.01
...
which I set to $0.01$ in case of the simulation you can see at the bottom of the page. Given that a unit time is $\approx 0.9785 \mathrm{Gyr}$, this means that the simulation will be saved every $\approx 9.785 \mathrm{Myr}$!
Besides that, I also increased the TimeMax value (which defines the end time of the simulation) from 3.0 to 8.0 to make the simulation cover a longer time period:
%---- Caracteristics of run
TimeBegin 0.0 % Begin of the simulation
TimeMax 8.0 % End of the simulation
This means that the simulation will run for $\approx 8 \mathrm{Gyr}$. Please note that with this configuration, the simulation will write (TimeMax - TimeBegin) / TimeBetSnapshot $=800$ snapshot files to the disk, so first make sure that you have enough space to store them.
The only remaining step we have to do is to run the simulation. For that, we need to invoke the Gadget4 executable with the parameter file as its only argument. Since we installed OpenMPI, we can run the simulation in parallel on multiple CPU cores using the
$ mpirun -np N <command>
command, where N is the number of threads we want to use. In this example, I am using 16 CPU cores to run the simulation (a number which might not be accessible to you, given your setup). Assuming that the executable is named as Gadget4 and is located in the ~/data/simulations/galaxy directory, as well as the parameter file named param.txt is located in the same directory, the command to run the simulation on e.g. $16$ CPU threads should look like this:
$ mpirun -np 16 ~/data/simulations/galaxy/Gadget4 ~/data/simulations/galaxy/param.txt
Of course, you can name and place these files however and wherever you like, as long as you provide the correct paths, when they are needed (e.g. in the parameter file or in the command above).
After the simulation is finished, you should see a bunch of new files in the output directory. The most important ones are the snapshot files. They will be named as snapshot_XXX.hdf5, where XXX is the index of each snapshot, arranged in a temporal order. Depending on the specific simulation, these files will contain parameters and the state of various ‘particle types’ (e.g. gas, dark matter, stars, etc.) at different times during the simulation. GADGET and similar software utilizes $6$ different particle types. In this particular example simulation, we have dark matter (the galactic halo) particles and particles in the galactic disk, which corresponds to PartType1 and PartType2 in the snapshot files, respectively. These files can be read and analyzed with the help of e.g. the h5py package using Python. You can open these particular galaxy collision snapshots with the following Python code:
import h5py
import numpy as np
def extract_parttype(f, parttype):
'''
Extracts the particle IDs, coordinates and velocities of a given
particle type from a snapshot file.
Parameters:
-----------
f : h5py.File
The snapshot file opened with h5py.
parttype : int
The particle type to extract. 1 for dark matter (halo), 2 for
disk particles.
Returns:
--------
np.ndarray of shape (N, 7)
A 2D numpy array with the particle IDs, coordinates and velocities.
'''
I = f[f'PartType{parttype}/ParticleIDs'][:] # Particle IDs
C = f[f'PartType{parttype}/Coordinates'][:] # Particle Coordinates
V = f[f'PartType{parttype}/Velocities'][:] # Particle Velocities
return np.hstack((I[:, np.newaxis], C, V))
def open_snapshot(s):
'''
Opens a GADGET-4 snapshot file and extracts the dark matter and disk
particle data from it.
Parameters:
-----------
s : str
The path to the snapshot file.
'''
with h5py.File(s, 'r') as f:
G_halo = extract_parttype(f, parttype=1)
G_disk = extract_parttype(f, parttype=2)
return G_halo, G_disk
You should not forget that coordinates and velocities are in the simulation units, so you have to convert them to physical units using the unit values defined in the parameter file! After that, the data can be visualized with any library or tool one prefers.
The final animation I created with matplotlib from the simulation I ran is shown below. The animation shows the evolution of the galaxy collision simulation over time, where the dark matter halo is shown in grey and the disk particles are shown in black. The second section of the video zooms into the center of the collision, where the particles are shown in red and blue, representing the two galaxies that are colliding. The animation is created by plotting the particle positions in the $X$-$Y$, $X$-$Z$ and $Y$-$Z$ planes at each time step. The animation can be found on YouTube: