Difference between revisions of "Profiling GEOS-Chem with the TAU performance system"

From Geos-chem
Jump to: navigation, search
(Created page with "== Overview == The [http://www.cs.uoregon.edu/research/tau/home.php TAU Performance System] is a profiling tool for performance analysis of parallel programs in Fortran, C, C...")
 
(Compiling and running GEOS-Chem with TAU)
(28 intermediate revisions by the same user not shown)
Line 1: Line 1:
 +
__FORCETOC__
 +
'''''[[GEOS-Chem 1-month benchmark timing results|Previous]] | [[Speeding up GEOS-Chem|Next]] | [[Guide to GEOS-Chem performance]]'''''
 +
 +
#[[Parallelizing GEOS-Chem]]
 +
#[[GEOS-Chem 7-day timing tests]]
 +
#[[GEOS-Chem scalability]]
 +
#[[GEOS-Chem 1-month benchmark timing results]]
 +
#<span style="color:blue">'''Profiling GEOS-Chem with the TAU performance system'''</span>
 +
#[[Speeding up GEOS-Chem]]
 +
 +
 +
<span style="color:red">'''''NOTE: This page is still under development.  The Spack installation guide is still being validated.'''''</span>
 +
 
== Overview ==
 
== Overview ==
  
Line 5: Line 18:
 
== Installing TAU ==
 
== Installing TAU ==
  
TBD
+
The best way to build TAU is with Spack.  Please see these instructions at [https://github.com/geoschem/geos-chem/issues/637 '''Github issue geoschem/geos-chem #637'''].
  
== Setting up TAU compiler ==
+
== Compiling and running GEOS-Chem with TAU ==
 
+
To install TAU on your system, follow the instructions on the [https://www.cs.uoregon.edu/research/tau/downloads.php TAU Software Download] page. On some systems, you can load pre-compiled software packages with the <tt>module</tt> command. check with your system administrator or IT staff to see if a TAU module is already installed on your system. If it is, we recommend that you <tt>module</tt> to load TAU into your computing environment. For example:
+
  
  module load tau/2.24.1-fasrc01  # Load TAU
+
You can use the <tt>tau_exec</tt> command to profile GEOS-Chem in order to reveal computational bottlenecks. Below is a job script that you can use.
  
You can also create a <tt>TAU_OPTIONS</tt> environment variable for your system startup file:
+
NOTE: The <tt>gcc.env</tt> file is the [https://github.com/geoschem/geos-chem/issues/637#issuecomment-788968697 script that loads your Spack modules].
  
  If you use bash, add this to your ~/.bashrc:
+
#!/bin/bash
 
   
 
   
    # Options for TAU profiler
+
#SBATCH -c 24
    export TAU_OPTIONS="-optRevert -optVerbose -optPreProcess -optContinueBeforeOMP"
+
#SBATCH -n 1
 +
#SBATCH -p YOUR-QUEUE-NAME
 +
#SBATCH --mem=15000
 
   
 
   
  If you use csh or tcsh, add this to your ~/.cshrc:
+
### geoschem.profile.run
 +
### Job script to run GEOS-Chem with TAU profiling
 
    
 
    
    # Options for TAU profiler  
+
  source ~/.bashrc  # Load bash settings
    setenv TAU_OPTIONS "-optRevert -optVerbose -optPreProcess -optContinueBeforeOMP"
+
source ~/gcc.env  # Environment file that loads TAU module from Spack
 
+
A list of options available to the TAU compiler scripts can be found by typing <code>man tau_compiler.sh</code>. For more detailed information on TAU, see the [https://www.cs.uoregon.edu/research/tau/tau-usersguide.pdf TAU User's Guide].
+
# Set the # of OpenMP threads to the # of cores we requested
 
+
export OMP_NUM_THREADS=24
== Compiling and running GEOS-Chem with TAU ==
+
 
+
To profile GEOS-Chem with TAU, you must first compile with the <tt>TAU_PROF=y</tt> Makefile option, e.g.:
+
 
+
  # Remove files from a previous compilation with TAU (if necessary)
+
  make tauclean
+
 
   
 
   
  # Compile with TAU profiling
+
# Run GEOS-Chem Classic and have TAU profile it
  make TAU_PROF=y ...etc. other makefile options ...
+
srun -c $OMP_NUM_THREADS tau_exec -T serial,openmp -ebs -ebs_resolution=function ./gcclassic > log 2>&1
  
It is important that you compile on a single processor (i.e. don't pass <tt>-j4</tt> or <tt>-j8</tt>) to allow TAU to properly instrument the code.
+
If you use SLURM, you can submit this with
  
The <tt>TAU_PROF=y</tt> option will set <tt>COMPILE_CMD :=tau_f90.sh</tt> instead of <tt>COMPILE_CMD :=$(FC)</tt> where <tt>FC</tt> is <tt>ifort</tt>, <tt>gfortran</tt>, or <tt>pgfortran</tt>.
+
sbatch geoschem.profile.run
  
Once you have compiled GEOS-Chem with <tt>TAU_PROF=y</tt>, you can run GEOS-Chem as you normally would. GEOS-Chem will create <tt>profile.*</tt> files containing the profiling information.
+
Otherwise you can just run it interactively:
 +
 
 +
./geoschem.profile.run
  
 
== Using ParaProf to create plots from the profiling data ==
 
== Using ParaProf to create plots from the profiling data ==
Line 64: Line 74:
  
 
--[[User:Melissa Payer|Melissa Sulprizio]] ([[User talk:Melissa Payer|talk]]) 22:48, 8 February 2017 (UTC)<br>--[[User:Bmy|Bob Yantosca]] ([[User talk:Bmy|talk]]) 16:55, 10 February 2017 (UTC)
 
--[[User:Melissa Payer|Melissa Sulprizio]] ([[User talk:Melissa Payer|talk]]) 22:48, 8 February 2017 (UTC)<br>--[[User:Bmy|Bob Yantosca]] ([[User talk:Bmy|talk]]) 16:55, 10 February 2017 (UTC)
 +
 +
 +
----
 +
'''''[[GEOS-Chem 1-month benchmark timing results|Previous]] | [[Speeding up GEOS-Chem|Next]] | [[Guide to GEOS-Chem performance]]'''''

Revision as of 23:21, 2 March 2021

Previous | Next | Guide to GEOS-Chem performance

  1. Parallelizing GEOS-Chem
  2. GEOS-Chem 7-day timing tests
  3. GEOS-Chem scalability
  4. GEOS-Chem 1-month benchmark timing results
  5. Profiling GEOS-Chem with the TAU performance system
  6. Speeding up GEOS-Chem


NOTE: This page is still under development. The Spack installation guide is still being validated.

Overview

The TAU Performance System is a profiling tool for performance analysis of parallel programs in Fortran, C, C++, Java, and Python. TAU uses a visualization tool, ParaProf, to create graphical displays of the performance analysis results.

Installing TAU

The best way to build TAU is with Spack. Please see these instructions at Github issue geoschem/geos-chem #637.

Compiling and running GEOS-Chem with TAU

You can use the tau_exec command to profile GEOS-Chem in order to reveal computational bottlenecks. Below is a job script that you can use.

NOTE: The gcc.env file is the script that loads your Spack modules.

#!/bin/bash

#SBATCH -c 24
#SBATCH -n 1
#SBATCH -p YOUR-QUEUE-NAME
#SBATCH --mem=15000

### geoschem.profile.run
### Job script to run GEOS-Chem with TAU profiling
 
source ~/.bashrc  # Load bash settings
source ~/gcc.env  # Environment file that loads TAU module from Spack

# Set the # of OpenMP threads to the # of cores we requested
export OMP_NUM_THREADS=24

# Run GEOS-Chem Classic and have TAU profile it
srun -c $OMP_NUM_THREADS tau_exec -T serial,openmp -ebs -ebs_resolution=function ./gcclassic > log 2>&1

If you use SLURM, you can submit this with

sbatch geoschem.profile.run

Otherwise you can just run it interactively:

./geoschem.profile.run

Using ParaProf to create plots from the profiling data

In your run directory, there should be one or more profile.* files. The number of profile.* files will depend on the number of CPUs that you use for your GEOS-Chem simulation. To pack all of the profiling data into a single file, type:

paraprof --pack GEOS-Chem_Profile_Results.ppk

Then run paraprof on the packed format (.ppk) file using:

paraprof GEOS-Chem_Profile_Results.ppk

This will open two windows, the ParaProf Manager window and the Main Data window. For more information on how to interpret the profiling data, see the ParaProf User's Manual.

If you click on the the bar labeled "thread0" in the ParaProf manager window, you can generate a plot that looks like this:

V11-01-thread0.png

The value displayed, the units, and the sort order can be changed from the Options menu. The time that each subroutine spent on the master thread is displayed as a histogram. By examining the histogram you can see which routines are taking the longest to execute. For example, the above plot shows that the COMPUTE_F routine (highlighted with the red box) is spending 444 seconds on the master thread, which is longer than the Rosenbrock solver takes to run. This is a computational bottleneck, which was ultimately caused by an unparallelized DO loop.

To save the plot, select "Save as Bitmap Image" from the File menu. In the Save Image File window, you may select the output type (JPEG File or PNG file) and specify the file name and location.

--Melissa Sulprizio (talk) 22:48, 8 February 2017 (UTC)
--Bob Yantosca (talk) 16:55, 10 February 2017 (UTC)



Previous | Next | Guide to GEOS-Chem performance