The GADGET-2 (GAlaxies with Dark matter and Gas intEracT - 2) is the second iteration of the gravitational N-body and smoothed-particle hydrodynamics (SPH) simulation code written by Volker Springel. This code was used to create the famous Millenium simulation in 2005, which was the biggest N-body simulation at the time. In this guide I will summarize the steps needed to install GADGET-2 on a Linux computer, and show how to run an example simulation with it.
GADGET-2 is a highly versatile and powerful simulation code that can be used to simulate a wide array of astrophysical phenomena. It is capable of simulating the evolution of dark matter, gas and stars in a cosmological context, and can be used to study the formation and evolution of galaxies, galaxy clusters, and large-scale structure in the universe. The code is written in C and is highly optimized for parallel computing, making it suitable for running on a wide range of computer architectures, from laptops to supercomputers.
GADGET-2 is built on a number of non-standard software libraries that you will need to install first before trying to compile and run GADGET-2. The following list contains all necessary software that you will need to download and install first:
You can download all of these files by clicking on the links above, or by using e.g. the wget or curl commands in the terminal. (Or you can just search for them yourself on the internet, of course.) When choosing an MPI library, it should not really matter which one you go with. Moreover, on computation servers, usually one or both of them are already installed on the system. In this tutorial I am going to use OpenMPI, but the steps should be identical in case of using MPICH too. Using wget you can download all of them at once:
$ wget https://download.open-mpi.org/release/open-mpi/v4.1/openmpi-4.1.5.tar.gz
$ wget https://ftp.gnu.org/gnu/gsl/gsl-2.7.tar.gz
$ wget http://www.fftw.org/fftw-2.1.5.tar.gz
$ wget https://github.com/HDFGroup/hdf5/releases/download/hdf5_1.14.4.3/hdf5-1.14.4-3.tar.gz
If you downloaded all tarballs from the URLs above, now you can extract, compile and install them. The installation of these libraries is a standard procedure that you can find in this tutorial.
--enable-mpi --enable-float --enable-type-prefix flags during the configuration step. More info about them and FFTW in the tutorial.
After you installed all dependencies, you can proceed to the next section.
The installation of GADGET-2 consists of editing the GADGET-2 Makefile by configuring the compile-time flags and linking the necessary dependencies. and then building the software with make. That is all. Download Gadget 2.0.7 from the official website and extract it to a directory of your choice. Assuming that the tar.gz file was downloaded to the ~/Downloads/ directory, you can extract it by running the following command:
~/Downloads/$ tar -xzvf gadget-2.0.7.tar.gz
The Makefile, which is our center of attention can be found in the Gadget-2.0.7/Gadget2 directory. In case of GADGET-2, this file can be divided into two parts. The first part contains the compiler flags the software uses to configure the available features in the code, and the second part contains the information about the dependencies and the target computer we are building and running on.
Now it is the best time to move the Gadget-2.0.7 directory to some more appropriate location. For brevity, assume that you created a directory named ~/GADGET-2 in your home directory, and moved contents of the Gadget-2.0.7 directory there using
$ mv ~/Downloads/Gadget-2.0.7 ~/GADGET-2
First, focus on the flags in the first section (see below). You can activate or deactivate certain flags by commenting or uncommenting them using the # character. Depending on the flags you set, the code will be compiled with different features and functionalities. Different kinds of simulations will require different flags to be set. For example, if you want to run a simulation with a periodic boundary condition, you need to activate the -DPERIODIC flag. Or, if you want to run a simulation with a tree-based gravity solver, you need to activate the -DPMGRID=SOME_INTEGER flag with an appropriate value. The full list of the available flags can be found in the Makefile itself, with their detailed explanations provided in the user guide of GADGET-2.
In this tutorial I assume that you want to run one of the example simulations of GADGET-2, namely the galaxy collision example. This simulation requires the following flags to be set:
#----------------------------------------------------------------------
# From the list below, please activate/deactivate the options that
# apply to your run. If you modify any of these options, make sure
# that you recompile the whole code by typing "make clean; make".
#
# Look at end of file for a brief guide to the compile-time options.
#----------------------------------------------------------------------
#--------------------------------------- Basic operation mode of code
#OPT += -DPERIODIC
OPT += -DUNEQUALSOFTENINGS
#--------------------------------------- Things that are always recommended
OPT += -DPEANOHILBERT
OPT += -DWALLCLOCK
#--------------------------------------- TreePM Options
#OPT += -DPMGRID=128
#OPT += -DPLACEHIGHRESREGION=3
#OPT += -DENLARGEREGION=1.2
#OPT += -DASMTH=1.25
#OPT += -DRCUT=4.5
#--------------------------------------- Single/Double Precision
#OPT += -DDOUBLEPRECISION
#OPT += -DDOUBLEPRECISION_FFTW
#--------------------------------------- Time integration options
OPT += -DSYNCHRONIZATION
#OPT += -DFLEXSTEPS
#OPT += -DPSEUDOSYMMETRIC
#OPT += -DNOSTOP_WHEN_BELOW_MINTIMESTEP
#OPT += -DNOPMSTEPADJUSTMENT
#--------------------------------------- Output
OPT += -DHAVE_HDF5
OPT += -DH5_USE_16_API
#OPT += -DOUTPUTPOTENTIAL
#OPT += -DOUTPUTACCELERATION
#OPT += -DOUTPUTCHANGEOFENTROPY
#OPT += -DOUTPUTTIMESTEP
#--------------------------------------- Things for special behaviour
#OPT += -DNOGRAVITY
#OPT += -DNOTREERND
#OPT += -DNOTYPEPREFIX_FFTW
#OPT += -DLONG_X=60
#OPT += -DLONG_Y=5
#OPT += -DLONG_Z=0.2
#OPT += -DTWODIMS
#OPT += -DSPH_BND_PARTICLES
#OPT += -DNOVISCOSITYLIMITER
#OPT += -DCOMPUTE_POTENTIAL_ENERGY
#OPT += -DLONGIDS
#OPT += -DISOTHERM_EQS
#OPT += -DADAPTIVE_GRAVSOFT_FORGAS
#OPT += -DSELECTIVE_NO_GRAVITY=2+4+8+16
#--------------------------------------- Testing and Debugging options
#OPT += -DFORCETEST=0.1
#--------------------------------------- Glass making
#OPT += -DMAKEGLASS=262144
Important note that here I added -DH5_USE_16_API a new flag in the # -- Output section that does not exist in the original Makefile of GADGET-2. It enables backward compatibility for GADGET-2 with newer versions of HDF5 (e.g. with version 1.10.6 that we installed above). This flag is necessary to compile GADGET-2 with HDF5 support, and without it the compilation will fail with an error message.
The next section of the Makefile is dedicated to specifying the C-compiler flags and the locations of the dependencies we installed in the first section of the tutorial. Here you need to tell the compiler where the GSL, FFTW, and HDF5 libraries (as well as lib files of OpenMPI) are located on your computer. Below, you find an example layout that demonstrates how the table should look when properly configured.
In the section labeled # -- Select target computer, you have the option to define a new machine by adding a new entry under an arbitrary name of your choice. Make sure to deactivate all other existing SYSTYPE variables. In my example, I used the identifier:
SYSTYPE="mycomputer"
However, you are free to name it whatever you would like. The only requirement is to use the same name everywhere else, where it is needed.
Moving on to the section # -- Adjust settings for target computer, you should create a new entry that corresponds to the SYSTYPE you have chosen – mycomputer in this case. You can simply use the existing entries as templates for setting up your new configuration. Your entry should look something like this:
ifeq ($(SYSTYPE),"mycomputer")
[...compiler flags and includes...]
endif
Within this section, it is crucial to provide the compiler with the absolute paths to all the required dependencies. This ensures that the compiler incorporates these libraries correctly. An appropriate configuration for the example above would look like the example below. The /path/to/... and /path/to/..._install parts of the paths should be replaced with the actual paths where you installed the libraries.
#----------------------------------------------------------------------
# Here, select compile environment for the target machine. This may need
# adjustment, depending on your local system. Follow the examples to add
# additional target platforms, and to get things properly compiled.
#----------------------------------------------------------------------
#--------------------------------------- Select some defaults
CC = mpicc # sets the C-compiler
OPTIMIZE = -O2 -Wall -g # sets optimization and warning flags
MPICHLIB = -lmpich
#--------------------------------------- Select target computer
SYSTYPE="mycomputer"
#SYSTYPE="MPA"
#SYSTYPE="Mako"
#SYSTYPE="Regatta"
#SYSTYPE="RZG_LinuxCluster"
#SYSTYPE="RZG_LinuxCluster-gcc"
#SYSTYPE="OpteronMPA"
#SYSTYPE="OPA-Cluster32"
#SYSTYPE="OPA-Cluster64"
#--------------------------------------- Adjust settings for target computer
ifeq ($(SYSTYPE),"mycomputer")
CC = mpicc
OPTIMIZE = -O3 -Wall -g
GSL_INCL = -I/path/to/gsl_install/include
GSL_LIBS = -L/path/to/gsl_install/lib
FFTW_INCL= -I/path/to/fftw_install/include
FFTW_LIBS= -L/path/to/fftw_install/lib
MPICHLIB = -L/path/to/openmpi_install/lib
HDF5INCL = -I/path/to/hdf5_install/include
HDF5LIB = -L/path/to/hdf5_install/lib -lhdf5 -lz
endif
# ---- The other `SYSTYPE` definitions start here, but ----
# ---- you don't need to change them. ----
ifeq ($(SYSTYPE),"MPA")
[...]
If you made sure that you set all the parameters correctly in the Makefile, you can save and close it. Now GADGET-2 could be built by changing the current working directory to ~/GADGET-2/Gadget2, and running make, same as we did above:
~/Downloads/Gadget-2.0.7/Gadget2/$ make 2>&1 | tee m.out
If everything went as well, then this command will automatically create an executable called Gadget2 in this same directory, which can be finally used to run simulations.
There are three components you need for a standard GADGET-2 simulation. The first one is the executable compiled above. The second one is a so-called “parameter file”, which stores the necessary run-time parameters of a simulation. The third one is the initial conditions (IC) file (or files), an appropriately formatted file, which contains the initial positions, velocities and other relevant attributes of the simulated particles. Selecting the appropriate parameters and creating the IC file is always an important and complex step to any computer simulation. Detailing the steps of IC generation for gravitational N-body simulations is beyond the scope of this tutorial. Fortunately, there are some pre-made examples shipped with the GADGET-2 download that contain both the IC file and parameter file for specific simulations. They can be launched immediately after building GADGET-2.
As I mentioned already, here I will use the galaxy collision example for demonstration. The IC file and the parameter file for this simulation can be found in the GADGET-2/ICs and GADGET-2/Gadget2/parameterfiles directories, respectively. These files can be distinguished by the galaxy prefix in their names. We need to use the galaxy_littleendian.dat IC file and the galaxy.param parameter file for this example. You will also need a directory to store the output files of the simulation, which can be created anywhere on your computer. Just for the sake of clarity, I created a directory named Simulations/galaxy in the GADGET-2 directory, which will contain all the output files of the simulations:
$ mkdir -p ~/GADGET-2/Simulations/galaxy
Besides that, I will also copy the parameter file into this same directory, so we can edit it in the next step, without modifying the original example file:
$ cp ~/GADGET-2/Gadget2/parameterfiles/galaxy.param ~/GADGET-2/Simulations/galaxy/
You can now edit the galaxy.param file in the Simulations/galaxy directory. Before running the simulation, there are some parameters which needs to be altered first. The first two lines after the initial comment line should be changed to contain the absolute paths of the IC input file and the output directory, which in our case are:
% Relevant files
InitCondFile /home/your_user_name/GADGET-2/ICs/galaxy_littleendian.dat
OutputDir /home/your_user_name/GADGET-2/Simulations/galaxy/
GADGET-2 works by taking in a set of initial conditions from a pre-defined IC file and evolving it in time. During a simulation, GADGET-2 creates a “snapshot” of the current state at given time intervals that contains the positions, velocities and other attributes of the simulated particles, saving it to the disk. GADGET-2 supports two different file formats during these I/O processes: the default GADGET format and a more modern HDF5 format.
I recommend using the HDF5 format, since it is a modern and versatile file format and it can be handled with e.g. Python much more easily. To change the output format, you will need to find the Code options block in the parameter file and change the SnapFormat value to 3. This will tell GADGET-2 to save the snapshot files in HDF5 format. However, since the pre-made example IC is in GADGET format, the ICFormat value should be set to 1 to indicate this. The final Code options block should look like this:
% Code options
ICFormat 1
SnapFormat 3
The last parameters you want to change control the length and the output frequency of the simulation. The meaning of space and time in GADGET-2 is not physical. When we mention “distances”, “velocities”, “masses” or “time” we mean pre-set simulation units. When an IC is created for a simulation, one chooses some unit length, unit velocity and unit mass to define the physical meaning of the numerical values in the IC file. In GADGET-2, the unit time is derived from these units as UnitTime_in_s = UnitLength_in_cm / UnitVelocity_in_cm_per_s. Unit length, velocity and mass is controlled in the parameter file under the System of units section. Considering the galaxy collision example, the unit length UnitLength_in_cm is set to 3.085678e21, which corresponds to $1 \mathrm{kpc}$. Similarly, the unit velocity UnitVelocity_in_cm_per_s is set to 1.0e5, which corresponds to $1 \mathrm{km/s}$. This gives us a unit time UnitTime_in_s of 3.085678e16, which corresponds to $\approx 0.9785 \mathrm{Gyr}$.
GADGET-2 will create a snapshot at every time interval defined by the TimeBetSnapshot parameter. The shorter this interval is, the more frequently the current state of the simulation will be saved to the disk, resulting in a more detailed “timeline” at the cost of using more disk space. This can be edited in the Output freqency block in the line
% Output frequency
TimeBetSnapshot 0.01
which I set to $0.01$ in case of the simulation you can see at the bottom of the page. Given that a unit time is $\approx 0.9785 \mathrm{Gyr}$, this means that the simulation will be saved every $\approx 9.785 \mathrm{Myr}$.
Besides that, I also increased the TimeMax value (which defines the end time of the simulation) from 3.0 to 8.0 to make the simulation cover a longer time period:
TimeBegin 0.0 % Begin of the simulation
TimeMax 8.0 % End of the simulation
This means that the simulation will run for $\approx 8 \mathrm{Gyr}$. Please note that with this configuration, the simulation will write (TimeMax - TimeBegin) / TimeBetSnapshot $=800$ snapshot files to the disk, so first make sure that you have enough space to store them.
The only thing that remains is to run the simulation. To do that, we need to invoke the Gadget2 executable with the parameter file as an argument. Since we installed OpenMPI, we can run the simulation in parallel on multiple CPU cores using the
$ mpirun -np N <command>
command, where N is the number of threads we want to use. In this example, I am using 16 CPU cores to run the simulation (a number which might not be accessible to you, given your setup). Assuming that the executable is named as Gadget2 and is located in the ~/GADGET-2/Gadget2 directory, as well as the parameter file named galaxy.param is located in the ~/GADGET-2/Simulations/galaxy directory, the command to run the simulation on e.g. $16$ CPU threads should look like this:
$ mpirun -np 16 ~/GADGET-2/Gadget2/Gadget2 ~/GADGET-2/Simulations/galaxy/galaxy.param
Take note of the abundance of Gadget2s in this command… :) Of course, you can place and name these files however and wherever you like, as long as you provide the correct paths and names, when it is needed (e.g. in the parameter file or in the command above).
After the simulation is finished, you should see a bunch of new files in the output directory. The most important ones are the snapshot files. They will be named as snapshot_XXX.hdf5, where XXX is the index of each snapshot, arranged in a temporal order. These files then can be loaded in e.g. Python using the h5py library, and then be visualized using whichever visualization library you prefer.
The final animation I created with matplotlib can be seen in this video: