Profiling GEOS-Chem with the TAU performance system

From Geos-chem
Revision as of 17:48, 25 June 2019 by Bmy (Talk | contribs) (Installing TAU)

Jump to: navigation, search

Previous | Guide to GEOS-Chem performance | Getting Started with GEOS-Chem | GEOS-Chem Main Page

  1. GEOS-Chem 7-day timing tests
  2. GEOS-Chem scalability
  3. GEOS-Chem 1-month benchmark timing results
  4. Profiling GEOS-Chem with the TAU performance system


Overview

The TAU Performance System is a profiling tool for performance analysis of parallel programs in Fortran, C, C++, Java, and Python. TAU uses a visualization tool, ParaProf, to create graphical displays of the performance analysis results.

Installing TAU

The best way to install TAU is to use Spack.

(1) Download Spack as described HERE.

(2) Define environment variables for Spack as described HERE.

export SPACK_ROOT=/path/to/spack
export PATH=$SPACK_ROOT/bin:$PATH

(3) Check if Spack can find your compiler as described HERE.

(4) Install TAU with the following Spack command:

spack install tau openmp=True

NOTE: You can also refine this by forcing a specific compiler version e.g.

spack install tau %gcc@8.2.0 openmp=True

(5) Define environment variables for TAU as shown below.

# TAU home location
export TAU_HOME=$(spack location -i tau)

# Put the location of tau_f90.sh in the search path
export PATH=$PATH:$TAU_HOME/bin

# Define options for TAU
export TAU_OPTIONS="-optRevert -optVerbose -optPreProcess -optContinueBeforeOMP  -optPdtGnuFortranParser"

# Define location of TAU makefile
export TAU_MAKEFILE=$TAU_HOME/x86_64/lib/Makefile.tau-papi-pthread-pdt-openmp

You can place these into one of your system environment files so that they are executed on startup.

Setting up TAU compiler

To install TAU on your system, follow the instructions on the TAU Software Download page. On some systems, you can load pre-compiled software packages with the module command. check with your system administrator or IT staff to see if a TAU module is already installed on your system. If it is, we recommend that you module to load TAU into your computing environment. For example:

 module load tau/2.24.1-fasrc01   # Load TAU

You can also create a TAU_OPTIONS environment variable for your system startup file:

 If you use bash, add this to your ~/.bashrc:

    # Options for TAU profiler
    export TAU_OPTIONS="-optRevert -optVerbose -optPreProcess -optContinueBeforeOMP"

 If you use csh or tcsh, add this to your ~/.cshrc:
 
    # Options for TAU profiler  
    setenv TAU_OPTIONS "-optRevert -optVerbose -optPreProcess -optContinueBeforeOMP"

A list of options available to the TAU compiler scripts can be found by typing man tau_compiler.sh. For more detailed information on TAU, see the TAU User's Guide.

Compiling and running GEOS-Chem with TAU

To profile GEOS-Chem with TAU, you must first compile with the TAU_PROF=y Makefile option, e.g.:

 # Remove files from a previous compilation with TAU (if necessary)
 make tauclean

 # Compile with TAU profiling
 make TAU_PROF=y ...etc. other makefile options ...

It is important that you compile on a single processor (i.e. don't pass -j4 or -j8) to allow TAU to properly instrument the code.

The TAU_PROF=y option will set COMPILE_CMD :=tau_f90.sh instead of COMPILE_CMD :=$(FC) where FC is ifort, gfortran, or pgfortran.

Once you have compiled GEOS-Chem with TAU_PROF=y, you can run GEOS-Chem as you normally would. GEOS-Chem will create profile.* files containing the profiling information.

Using ParaProf to create plots from the profiling data

In your run directory, there should be one or more profile.* files. The number of profile.* files will depend on the number of CPUs that you use for your GEOS-Chem simulation. To pack all of the profiling data into a single file, type:

paraprof --pack GEOS-Chem_Profile_Results.ppk

Then run paraprof on the packed format (.ppk) file using:

paraprof GEOS-Chem_Profile_Results.ppk

This will open two windows, the ParaProf Manager window and the Main Data window. For more information on how to interpret the profiling data, see the ParaProf User's Manual.

If you click on the the bar labeled "thread0" in the ParaProf manager window, you can generate a plot that looks like this:

V11-01-thread0.png

The value displayed, the units, and the sort order can be changed from the Options menu. The time that each subroutine spent on the master thread is displayed as a histogram. By examining the histogram you can see which routines are taking the longest to execute. For example, the above plot shows that the COMPUTE_F routine (highlighted with the red box) is spending 444 seconds on the master thread, which is longer than the Rosenbrock solver takes to run. This is a computational bottleneck, which was ultimately caused by an unparallelized DO loop.

To save the plot, select "Save as Bitmap Image" from the File menu. In the Save Image File window, you may select the output type (JPEG File or PNG file) and specify the file name and location.

--Melissa Sulprizio (talk) 22:48, 8 February 2017 (UTC)
--Bob Yantosca (talk) 16:55, 10 February 2017 (UTC)