Getting started with Numerical Relativity Simulations

This tutorial will guide you through the inital steps of running large scale binary black hole simulations. First you will copy the AEI Cactus source code onto your personal machine. Then you will set up SimFactory , a service for to assist in submitting and monitoring simulations on various cluster. Next, you will sync your code to our cluster, datura, and remotely submit a job. With the job is submitted you will be able to monitor it using SimFactory . Finally, you will retrieve output from the simulation and visualize it locally using SimulationTools .

Note: this tutorial assumes you already have an account on datura and have set up password-less login with public key encryption.

Download

Download Cactus and all our code into your home directory:

cd $HOME
git clone --recursive et@git.barrywardell.net:AEI/Cactus

This may take a while. Feel free to be productive in the meantime by reading about Cactus. Once the clone is finished, you should have a Cactus/ directory within your home directory.

Configure

SimFactory is a tool that provides a consistent interface for interacting with simulations on different clusters with different hardware, queuing systems, etc. You will call SimFactory often, and you should therefore add an alias for SimFactory to your shell startup file (e.g. .bash_profile, or the like):

alias sim=simfactory/bin/sim

Also run this command so that the alias is available. Note that this alias assumes you will always be calling ‘sim’ from within your Cactus/ directory.

Next, xonfigure SimFactory :

cd Cactus
sim setup

Answer the questions. Use the defaults (hit enter) except for email address

Add the following to the top of simfactory/etc/defs.local.ini, to tell SimFactory which files to sync with the cluster:

sync-sources = <<EOT

flesh
thornlists
parfiles
EOT

and this to the bottom to tell SimFactory your username on datura.

[datura]
user            = <your username on datura>

If you start working on another cluster in the future, you will add another section here for that cluster.

Sync source tree to cluster

Within your Cactus/ directory, run:

sim sync datura

Compile

Compile Cactus on Datura

sim --remote datura build --thornlist thornlists/aeivacuum.th

 

Submit a simulation

sim --remote datura create-submit parfiles/Examples/bbh.rpar 60 24:00:00

Monitor simulation status

sim --remote datura list-simulations
sim --remote datura show-output bbh

Copy lightweight simulation data to local machine

When the simulation is finished, transfer the lightweight simulation output to your local machine.

mkdir ~/Simulations
getsim datura bbh

TODO: tidy up and provide the getsim script somewhere

Visualise the simulation in Mathematica

Install SimulationTools [http://simulationtools.org] (contact Ian or Barry for access to the repository and installation instructions)

Add

$SimulationDirectory = FileNameJoin[{$HomeDirectory,"Simulations}]";

to your Mathematica init.m file (~/Library/Mathematica/Kernel/init.m on Mac OS, or .Mathematica/Kernel/init.m on Linux)

Run the Mathematica front-end and enter:

<<SimulationTools`

psi4 = ReadPsi4["bbh", 2, 2, 100]

PresentationListLinePlot[Re[psi4], PlotRange -> All]

You should see a plot of a waveform.