View source for Timing tests with GEOS-Chem v11-01

On this page we show the results of timing tests done with [[GEOS-Chem v11-01]].

== Overview ==

The GEOS-Chem Support Team has created a timing test package that you can use to determine the performance of GEOS-Chem on your system.  The time test runs the [[GEOS-Chem v11-01#v11-01 provisional release|GEOS-Chem v11-01 provisional release code]] for 7 model days with the "Standard" chemistry mechanism.  Our experience has shown that a 7-day simulation will give a more accurate timing result than a 1-day simulation.  This is because much of the file I/O (i.e. HEMCO reading annual or monthly-mean emissions fields) occurs on the first day of a run.

=== Installation ===

If you haven't already, download the GEOS-Chem v11-01 source code using:

 git clone https://bitbucket.org/gcst/geos-chem Code.v11-01

Create your run directory using the GEOS-Chem UnitTest by following [[Creating_GEOS-Chem_run_directories|these instructions]]. To copy the <tt>gc_timing</tt> run directory, make sure you have the following lines in your <tt>CopyRunDirs.input</tt> file:

 #--------|-----------|------|------------|------------|------------|---------|
 # MET    | GRID      | NEST | SIMULATION | START DATE | END DATE   | EXTRA?  |
 #--------|-----------|------|------------|------------|------------|---------|
  geosfp   4x5         -      gc_timing    2013070100   2013070800   -

=== Compilation ===

To build the code, follow these steps:
   
  cd geosfp_4x5_gc_timing
  make realclean
  make -j4 mpbuild 2>&1 log.build

Information about the options used for the compilation (as well as the compiler version) will be printed to the file <tt>lastbuild.mp</tt>.

=== Performing the timing test ===

To run the code, follow the instructions in the 

  geosfp_4x5_gc_timing/README 

file.  We have provided sample run scripts that you can use to submit jobs:

  geosfp_4x5_gc_timing/doTimeTest          # Submit job directly
  geosfp_4x5_gc_timing/doTimeTest.slurm    # Submit job using the SLURM scheduler  

The regular GEOS-Chem output as well as timing information will be sent to a log file named:

  doTimeTest.log.ID

where ID is either the SLURM job ID # or the process ID.

=== Displaying the test results ===

You can print out the timing results with the <tt>printTime</tt> script:

  cd geosfp_4x5_gc_timing
  ./printTime doTimeTest.log.ID

which will display results similar to this:

 GEOS-Chem 7-Model-Day Time Test Results
 ===============================================================================
 
 Machine information
 -------------------------------------------------------------------------------
 Machine or node name: : holyjacob03.rc.fas.harvard.edu
 CPU vendor            : GenuineIntel
 CPU model name        : Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz
 CPU speed [MHz]       : 2494.097 
 
 GEOS-Chem information
 -------------------------------------------------------------------------------
 GEOS-Chem Version     : v11-01
 Last commit           : Now use correct species DB FullName for TOMAS DUST* species 
 Commit date           : Tue Dec 13 18:05:28 2016 -0500 
 Compiler version      : ifort 11.1
 Compilation options   : geosfp 4x5 standard no_reduced UCX timers
 
 Simulation information
 -------------------------------------------------------------------------------
 Simulation start date : 20130701 000000
 Simulation end date   : 20130708 000000
 Number of CPUs used   : 4
 Total CPU time  [s]   : 42926.897
 Wall clock time [s]   : 11810.496
 CPU / Wall ratio      : 3.6346
 % of ideal performace : 90.87

You can then use these results to fill in [[#Table of 7-model-day run times|the table below]].

--[[User:Bmy|Bob Yantosca]] ([[User talk:Bmy|talk]]) 18:12, 19 December 2016 (UTC)

== Table of 7-model-day run times ==

The following timing test results were done with the '''"out-of-the-box" [[GEOS-Chem v11-01]] provisional release''' code configuration.

*All jobs used [[GEOS-FP]] meteorology at 4&deg; x 5&deg; resolution.  
*Jobs started on model date <tt>2013/07/01 00:00 GMT</tt> and finished on <tt>2013/07/08 00:00 GMT</tt>.
*The code was compiled from the run directory (<tt>geosfp_4x5_gc_timing</tt>) with the the standard option <tt>make -j4 mpbuild</tt>.  This sets the following compilation variables:
**<tt>MET=geosfp GRID=4x5 CHEM=benchmark UCX=y NO_REDUCED=n TRACEBACK=n BOUNDS=n FPE=n DEBUG=n NO_ISO=n NEST=n</tt>
*Wall clock times are listed from fastest to slowest, for the same number of CPUs.  (Bands of white and cyan in the table indicate different number of CPUs.)
*It's OK to round CPU and wall clock times to the nearest second, for clarity.

{| border=1 cellspacing=0 cellpadding=5 
|-valign="top" bgcolor="#CCCCCC"
!width="150px"|Submitter
!width="200px"|Machine or Node<br>and Compiler
!width="100px"|CPU vendor
!width="150px"|CPU model
!width="75px"|Speed [MHz]
!width="50px"|# of<br>CPUs
!width="85px"|CPU time
!width="85px"|Wall time
!width="75px|CPU / Wall<br>ratio
!width="75px"|% of ideal

|-valign="top" bgcolor="#CCFFFF"
!colspan="10"|Results from 7-model-day time tests using 16 CPUs

|-valign="top" 
|Jenny Fisher<br>(NCI)
|r1808<br>ifort 12.1.3
|GenuineIntel
|Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz
|2601.000
|16
|73955.44 s<br>'''20:32:35'''
|6129.04 s<br>'''01:42:09'''
|12.0664
|75.41

|-valign="top" bgcolor="#CCFFFF"
!colspan="10"|Results from 7-model-day time tests using 12 CPUs

|-valign="top" 
|Bob Yantosca<br>([[GCST]])
|holyjacob03.rc.fas.harvard.edu<br>gfortran 4.8.2
|GenuineIntel
|Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz
|2494.097
|12
|47826.4 s<br>'''13:17:06'''
|6205.87 s<br>'''01:43:26'''
|7.7066
|64.22

|-valign="top" 
|Bob Yantosca<br>([[GCST]])
|holyjacob06.rc.fas.harvard.edu<br>ifort 11.1
|GenuineIntel
|Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz
|2494.120
|12
|53075.95 s<br>'''14:26:37'''
|5612.76 s<br>'''01:33:33'''
|9.4563
|78.8

|-valign="top" 
|Bob Yantosca<br>([[GCST]])
|holyjacob02.rc.fas.harvard.edu<br>ifort 11.1
|GenuineIntel
|Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz
|2494.444
|12
|52928.482 s<br>'''14:33:49'''
|5600.497 s<br>'''01:33:20'''
|9.4507
|78.76

|-valign="top" bgcolor="#CCFFFF"
!colspan="10"|Results from 7-model-day time tests using 8 CPUs

|-valign="top" 
|Jenny Fisher<br>(NCI)
|r52<br>ifort 12.1.3
|GenuineIntel
|Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz
|2601.000
|8
|57825.7 s<br>'''16:03:46'''
|8812.05 s<br>'''02:26:52'''
|6.5621
|82.03

|-valign="top" 
|Bob Yantosca<br>([[GCST]])
|holyjacob04.rc.fas.harvard.edu<br>gfortran 4.8.2
|GenuineIntel
|Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz
|2494.196
|8
|47629.5 s<br>'''13:13:50'''
|8290.15 s<br>'''02:18:10'''
|5.7453
|71.82

|-valign="top" 
|Bob Yantosca<br>([[GCST]])
|holyjacob03.rc.fas.harvard.edu<br>ifort 11.1
|GenuineIntel
|Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz
|2494.097
|8
|51955.61 s<br>'''14:25:56'''
|7917.931 s<br>'''02:11:58'''
|6.6188
|82.74

|-valign="top" 
|Melissa Sulprizio<br>([[GCST]])
|holyjacob05.rc.fas.harvard.edu<br>ifort 11.1
|GenuineIntel
|Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz
|2494.177
|8
|50711.89 s<br>'''14:05:11'''
|7673.93 s<br>'''02:07:56'''
|6.6083
|82.6

|-valign="top" bgcolor="#CCFFFF"
!colspan="10"|Results from 7-model-day time tests using 6 CPUs

|-valign="top" 
|Bob Yantosca<br>([[GCST]])
|holyjacob06.rc.fas.harvard.edu
|GenuineIntel
|Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz
|2494.120
|6
|43571.73<br>'''12:06:12'''
|8351.52<br>'''02:19:12'''
|5.2172
|86.95

|-valign="top" bgcolor="#CCFFFF"
!colspan="10"|Results from 7-model-day time tests using 4 CPUs

|-valign="top" 
|Bob Yantosca<br>([[GCST]])
|holyjacob03.rc.fas.harvard.edu<br>ifort 11.1
|GenuineIntel
|Intel(R) Xeon(R) CPU E5-2680 v3 @ 2.50GHz
|2494.097
|4
|42667.423 s<br>'''11:51:07'''
|11810.496 s<br>'''03:16:50'''
|3.6346
|90.87

|-valign="top" bgcolor="#CCFFFF"
!colspan="10"|Results from 7-model-day time tests using 2 CPUs

|-valign="top" 
|TBD
|
|
|
|
|
|
|
|
|

|-valign="top" bgcolor="#CCFFFF"
!colspan="10"|Results from 7-model-day time tests using 1 CPU

|-valign="top" 
|TBD
|
|
|
|
|
|
|
|
|

|}

--[[User:Bmy|Bob Yantosca]] ([[User talk:Bmy|talk]]) 15:46, 22 December 2016 (UTC)

== Graph of 7-model-day run times ==

We will post this when we have added many more entries to the table above.

--[[User:Bmy|Bob Yantosca]] ([[User talk:Bmy|talk]]) 17:23, 22 December 2016 (UTC)

== Time spent in each GEOS-Chem operation ==

You can view the amount of time spent in each GEOS-Chem operation by enabling [[GEOS-Chem timers]]. Adding <tt>TIMERS=1</tt> to your make command will tell GEOS-Chem to enable timers and print a timer summary at the end of your log file. 

=== Comparison of ifort and gfortran ===

'''''[[User:Bmy|Bob Yantosca]] wrote:'''''

<blockquote>For reference, I was able to compare how long each GC operation takes from 2 timetest runs on the Intel Xeon nodes of <tt>odyssey.rc.fas.harvard.edu</tt>.  The colored lines indicate where [[GNU Fortran compiler|gfortran]] takes significantly longer than [[Intel Fortran Compiler|ifort]]:</blockquote>

                                ifort 11           gfortran 4.8.2
                                12 CPUs            12 CPUs
                                holyjacob06        holyjacob03 
                                SLURM: 77647117    SLURM: 77631200 
 
       Timer name               hh:mm:ss.SSS       hh:mm:ss.SSS
     ----------------------------------------------------------------
       <span style="color:red">GEOS-Chem             :  01:33:30.437       01:43:22.906</span>
       Initialization        :  00:00:07.390       00:00:04.062       
       Timesteps             :  01:33:21.906       01:43:18.062
       HEMCO                 :  00:17:03.859       00:15:21.625
       All chemistry         :  00:37:55.312       00:38:45.875
       => Strat chem         :  00:00:39.140       00:00:38.781
       => Gas-phase chem     :  00:33:06.609       00:30:21.562
       <span style="color:blue">=> All aerosol chem   :  00:03:40.921       00:07:07.343</span>
       <span style="color:purple">Transport             :  00:07:58.062       00:12:25.000</span>
       <span style="color:green">Convection            :  00:21:48.296       00:26:19.812</span>
       Dry deposition        :  00:00:25.062       00:00:30.375       
       Wet deposition        :  00:03:41.156       00:03:38.343       
       <span style="color:orange">Diagnostics           :  00:02:23.421       00:03:51.343</span>
       Reading met fields    :  00:00:06.265       00:00:08.437       
       Reading restart file  :  00:00:00.312       00:00:00.343       
       <span style="color:brown">Writing restart file  :  00:01:13.703       00:01:32.843</span>

<blockquote>I wonder if some of this comes down to either OpenMP or math optimizations.  In any case, we know that gfortran is slower in general but it’s interesting to see that for some operations it’s very comparable to ifort.  

Note that the results above are only from 2 individual tests; we can improve the statistics by averaging the results of several tests.  We will do this as time allows.</blockquote>

--[[User:Bmy|Bob Yantosca]] ([[User talk:Bmy|talk]]) 17:22, 22 December 2016 (UTC)

== For more information ==

We invite you to consult the following resources for more information about GEOS-Chem's performance:

*[[GEOS-Chem performance]]
*[[GEOS-Chem performance#GEOS-Chem scalability|GEOS-Chem scalability]]
*[[Parallelizing GEOS-Chem]]
*[[Profiling GEOS-Chem]]

--[[User:Bmy|Bob Yantosca]] ([[User talk:Bmy|talk]]) 21:36, 15 December 2016 (UTC)