HackLatt 2008 MILC Code with SciDAC

Objectives

In this tutorial you will first learn how to build the MILC code with the SciDAC/C packages and, time permitting, how to download, build, and install those packages.

The first exercises in this tutorial assume that you already have a precompiled, installed version of the SciDAC packages. In later exercises we will show how to install them.

Building su3_omelyan_rhmc with QMP
Building su3_omelyan_rhmc with QIO
Building su3_omelyan_rhmc with QOP
Obtaining and Installing the SciDAC C code packages

Exercise: Building su3_omelyan_rhmc with QMP

Return to the directory ks_imp_rhmc. In case you changed the Makefile, restore the default version of this file by copying it to this directory from the parent directory.


   cd /opt/localhomes/lab/milc/v7.6.2.beta/ks_imp_rhmc
   cp ../Makefile .

(Mind the dot .) Find the lines in the Makefile beginning with WANT. Edit them so they read

   WANTQOP = 
   WANTQDP = 
   WANTQIO = 
   WANTQMP = true

This is the way to call for the QMP message passing package, but to not use any other of the packages. Compile the code as usual:


   make su3_omelyan_rhmc

For a single processor, this change has essentially no effect on execution, since there is no communication involved, anyway. The code is linked against the single-processor version of the QMP package.

If you had also uncommented MPP = true in the Makefile before running make, the code would instead be linked with the MPP version of the QMP package. The installed version of the QMP package implements message passing with MPI, so there is not much advantage to using QMP by itself this way. But the QCDOC has its own implementation of QMP, so with that package (and the appropriate cross compiler) the compiled MPP code would run on the QCDOC. And you will need QMP if you want any of other SciDAC components.

Exercise: Building su3_omelyan_rhmc with QIO

Next, edit the "wants" to read


   WANTQOP = 
   WANTQDP = 
   WANTQIO = true
   WANTQMP = true

and remake the code su3_omelyan_rhmc as before. This time you are calling for the SciDAC I/O package. The compiled code is then able to read and write SciDAC formatted files.

Note that QIO requires QMP, so if you don't say WANTQMP = true you'll get it anyway without asking.

Copy the sample input parameter file to in.test.su3_omelyan_rhmc.1 and edit the copy so the last line reads

save_serial_scidac lat.test.scidac

and run the code as before

./su3_omleyan_rhmc < in.test.su3_omelyan_rhmc.1 > out.test.su3_omelyan_rhmc.1

The output lattice is in USQCD SciDAC format.

To get an ILDG-compatible file, you need an extra line to specify the logical file name string. For example:

save_serial_ildg lat.test.ildg ILDG_LFN lfn://lat.test.ildg

Try running the code to produce either SciDAC format. Then examine the file contents using the QIO (LIME) utility

~/scidac/qio-single/bin/lime_contents lat.test.ildg

Exercise: Building su3_omelyan_rhmc with QOP (Level 3)

The Level 3 optimized code can be implemented in various ways. We have installed the QDP implementation, which makes the code highly portable. There is also a QCDOC implementation written by Chulwoo Jung, based on the RBC CPS code.

To get the QDP implementation (called QOPQDP), edit the lines to read

WANTQOP = QDP WANTQDP = true WANTQIO = true WANTQMP = true

Since QOPQDP require QDP, and QDP requires, in turn, QLA, QIO, and QMP, you'll get the rest of the packages even if you don't specify them explicitly with "true".

You might have wondered why there isn't a separate WANTQLA line for the SciDAC single-processor-linear-algebra package. The reason is that at the moment we don't implement QLA directly in the MILC code. We use it only through QDP.

Next, let's build the code with global double precision by editing the line beginning with PRECISION in the Makefile to read

PRECISION = 2

Next, be sure the installation directories for the SciDAC package are also correctly specified in the Makefile. For the lab installation they should read

SCIDAC = ${HOME}/scidac # Parallel versions QMPPAR = ${SCIDAC}/qmp QIOPAR = $(SCIDAC)/qio # Single processor versions QMPSNG = ${SCIDAC}/qmp-single QIOSNG = $(SCIDAC)/qio-single QLA = ${SCIDAC}/qla # Either version QDP = ${SCIDAC}/qdp-single QOPQDP = ${SCIDAC}/qopqdp-single

Finally, let's get some performance information about the inverters by editing the line beginning with CTIME to read

CTIME = -DCGTIME -DFFTIME -DLLTIME -DGFTIME -DREMAP

Now build the code as before and run it as usual:

./su3_omelyan_rhmc < in.sample.su3_omelyan_rhmc.1 > out.test.su3_omelyan_rhmc.1

The output log file should have a great many more lines giving performance information. The lines beginning with CONGRAD are for the conjugate gradient solver. They also tell you which precision is being used, which version of the solver is being used, and how many masses are solved simultaneously. The coding is as follows:

fn_qop D QOP double precision Fat/Naik inverter
fn_qop F QOP single precision Fat/Naik inverter

In this way you can tell that some of the inversions are being done in single precision and some in double. They all use the Level 3 QOPQDP solver. You can also see that the code polishes off its multimass inversions with a pass through the single-mass inverter for each mass.

Obtaining and Installing the SciDAC C code packages

All SciDAC packages are available from the USQCD software site http://usqcd.jlab.org/usqcd-software/. You need five C-coded packages: QMP, QIO, QLA, QDP, and QOPQDP. Links from the top level page take you to documentation and information about the most recent versions, as well as the actual tar balls. This site is scheduled for improvement, so please be patient. To find the most recent version number, click on the package name on the sidebar and (in most cases) in the synopsis page and follow the link in the line "Current and past releases can be found here." to get a directory listing of the tarballs.

The most direct way to build the code is to download each package, unpack it into a separate directory, and run configure and make. You need to build the packages in the proper order and need to know a variety of configuration options to do this successfully.

A simpler way to install the code is to use the qinstall installation scripts, currently under development, and soon to be linked to the USQCD software site. They can be obtained from the JLab CVS site as follows

cvs -d :pserver:anonymous@cvs.jlab.org:/group/lattice/cvsroot co -P qinstall

We have provided a version of these scripts with this tutorial. They are intended to automate the configuration process and build the code in the proper order.

The README file provides some instruction. The files with suffix prf are editable profile files that specify the build directory, installation directory, and the rules for building the packages. There is a subdirectory for each package containing the configuration options for various architectures and system environments. These files are used by the qinstall script to download, configure, make, and install the packages. The idea is that developers will contribute to the profile and architecture files for various installations, so that eventually users will be able to find the appropriate set for nearly any machine and compiler.

For practice, let's build and install the Pentium-4 single-processor QMP and QIO packages with the qinstall script.

Create a top-level directory for building the code (usually in some large scratch space) and a top-level directory for installing the packages (usually under your home directory). For example

mkdir ~/scidac-build-test mkdir ~/scidac-test

You also need these subdirectories

mkdir ~/scidac-build-test/download mkdir ~/scidac-build-test/src mkdir ~/scidac-build-test/build

Then copy the entire qinstall tree to this location

cp -pR ~/scidac/build/qinstall ~/scidac-build-test

Then edit the first few lines in the file p4single.prf to get the directory definitions right. Note that the file syntax is Bourne shell. The two lines you should change are these:

rootdir=$HOME/scidac-build-test INSROOT=$HOME/scidac-test

Then run these commands (the order is important)

./qinstall p4single qmp 2.2.0 ./qinstall p4single qio 2.3.4

to download the QMP and QIO packages with the indicated version numbers, build them with the gcc compiler, and install them.

If you are successful, you should find the following entries in the directory $HOME/scidac-test:

qio-2.3.3-single qio-single->qio-2.3.3-single qmp-2.2.0-single qmp-single->qmp-2.2.0-single

The symbolic links, indicated with an arrow, are good references for building your code. The directories to which they point contain the installed packages. Note that the MILC code Makefile uses the same symbolic links to set the locations of these SciDAC packages with its make macros QMPSNG and QIOSNG. There are corresponding macros for the other packages, including the MPP versions.

To build a recent version of the full MPP code using a generic version of MPI, you would edit the p4.prf file just as you did p4single.prf to get the directories right and then run these commands (again, the order is important)

./qinstall p4 qmp 2.2.0 ./qinstall p4 qio 2.3.4 ./qinstall p4 qla 1.6.2 ./qinstall p4 qdp 1.7.1 ./qinstall p4 qopqdp 0.10.1

Don't try this during the tutorial session, since the QLA and QDP packages are rather large and take a while to build, and the MPP versions of QMP and QIO require MPI, which is not installed in the NESC lab.