GEOS-Chem unit tester for v11-02

From Geos-chem
(Redirected from GEOS-Chem Unit Tester)
Jump to: navigation, search


GEOS-Chem v11-02-final will also carry the designation GEOS-Chem 12.0.0. We are migrating to a purely numeric versioning system in order to adhere more closely to software development best practices. For a complete description of the new versioning system, please see our GEOS-Chem version numbering system wiki page.




This page describes the GEOS-Chem Unit Tester package that corresponds to GEOS-Chem v11-02 (aka 12.0.0).

NOTE: If you are still using GEOS-Chem v11-01, then please see our GEOS-Chem unit tester for v11-01 wiki page.

Overview

The GEOS-Chem Unit Tester contains the various Makefiles, scripts, and run directories that you will need to run GEOS-Chem. With the Unit Tester, you can:

Action Description
Create GEOS-Chem run directories You can use a script to create fresh copies of GEOS-Chem run directories for each supported GEOS-Chem simulation. Each run directory that you create will contain all of the necessary input files with the proper settings. For detailed instructions, please visit our Creating GEOS-Chem run directories wiki page.
Set up GEOS-Chem 7-day timing tests You can create run directories that can be used to perform 7-model-day timing tests. This will allow you to quickly evaluate the performance of GEOS-Chem on your system. For more information, please see our Timing tests with GEOS-Chem v11-02 wiki page.
Run GEOS-Chem unit tests An individual GEOS-Chem Unit Test will look for computational and numerical issues in simulations for a given combination of:
  1. Met field type
  2. Horizontal grid
  3. Simulation type

We recommend that you perform frequent unit tests as you develop code, as this is the best way to find many common types of errors. For more information about how to submit unit tests, please see the Using the Unit Tester section below.

Run GEOS-Chem difference tests A Difference Test validates that structural updates in a given GEOS-Chem "Development" (or "Dev") produce identical results when compared to a "Reference" (or "Ref") version.

You can use a script that will create difference test run directories for any of the supported GEOS-Chem simulations. For more information on how to set up GEOS-Chem difference tests, please see our Performing Difference Tests with GEOS-Chem wiki page.

--Bob Yantosca (talk) 19:12, 20 March 2018 (UTC)

Installing the GEOS-Chem Unit Tester

Requirements

Most GEOS-Chem users will already have the necessary software to be able to run the GEOS-Chem Unit Test package. These are:

Item If you use GEOS-Chem
on a computer cluster
If you use GEOS-Chem
on the Amazon Web Services cloud
A modern Unix operating system
  • CentOS, Ubuntu, Fedora ...
This will be already installed on your cluster. The particular flavor of Unix should not matter. This will be included in the Amazon Machine Image (AMI) that you use to initialize your login environment.
The bash Unix shell This should be pre-installed with your OS. " "
The Perl programming language " " " "
The GNU make utility " " " "

--Bob Yantosca (talk) 19:24, 20 March 2018 (UTC)

Downloading the GEOS-Chem Unit Tester

IMPORTANT NOTE! As of June 2018, we have migrated the GEOS-Chem Unit Tester repository from Bitbucket back to Github. Going forward, please make sure to clone or pull code updates ONLY from the Github repository. Using Github will allow us to define a DOI for each GEOS-Chem major version.

First, make sure that your system has these software packages installed. (Most of these come standard with your Unix-based operating system.)

Next, clone the GEOS-Chem Unit Tester package with these commands:

git clone https://github.com/GCST/geos-chem-unittest UT
git checkout master

This will download a fresh copy of the GEOS-Chem-UnitTest repository into a directory named UT, and place you in the master branch. The master branch always corresponds to the last publicly-released GEOS-Chem version.

If you would like to check out the Unit Tester that corresponds to a specific numbered GEOS-Chem version you can type one of the following commands:

git checkout 12.0.0
git checkout v11-02-rc
git checkout v11-02e

etc.

--Bob Yantosca (talk) 21:12, 21 June 2018 (UTC)

Reverting to an older state

If you would like to use the Unit Tester to validate an older version of GEOS-Chem, then you can use the gitk browser to open a new branch at a past commit as follows:

  1. Open the gitk browser by typing gitk & at the command line.
  2. In the top-left window of gitk, find the commit that you want to revert to. Usually this will be denoted with a yellow tag (e.g. v9-02-Provisional-Release' or v10-01b, etc.).
  3. Right click with the mouse. This will open a context menu. Select Create new branch.
  4. A new dialog box will pop up asking you to name the branch. Type OLDER_BRANCH and press OK.
  5. Close gitk and open the Git GUI by typing git gui &
  6. From the Git GUI dropdown menu, select Branch / Checkout, and then pick OLDER_BRANCH.

That's it! We now have two branches that represent different GEOS-Chem versions.

  1. The master branch represents current state of the unit tester
  2. The OLDER_BRANCH branch represents the state of the unit tester that you checked out (synced to an older GEOS-Chem version)

--Bob Yantosca (talk) 19:48, 24 April 2017 (UTC)

Directory structure

The GEOS-Chem Unit Tester adheres to the following directory structure:

Directory Description
jobs/ Job scripts created by the GEOS-Chem Unit Test driver script gcUnitTest will be sent here.
logs/ Log files containing output from the GEOS-Chem Unit Test simulations will be sent here.
perl/ Contains the Perl scripts that are used to submit GEOS-Chem Unit Test simulations.
  • The driver script is called gcUnitTest.
  • The default input file for gcUnitTest is named UnitTest.input.
  • Sample customized versions of UnitTest.input are stored in the perl/inputs subdirectory.
  • The cleanFiles script is used to remove job scripts, log files, and output files created by GEOS-Chem.
runs/ Contains the various run directories. Each run directory is named according to the met field, horizontal grid, and type of simulation (4x5_standard, 2x25_RnPbBe, etc).

--Bob Yantosca (talk) 15:08, 20 December 2016 (UTC)

Using the GEOS-Chem Unit Tester

What the GEOS-Chem Unit Tester does

For each individual unit test (met field + horizontal grid + simulation type) that you specify in the RUNS section of the input file (more on this below), the GEOS-Chem Unit Tester will do the following:

  1. Run GEOS-Chem with strict debugging flags with OpenMP parallelization turned OFF
    • This is called the single processor or sp test phase.
  2. Run GEOS-Chem with strict debugging flags with OpenMP parallelization turned ON
    • This is called the multi processor or mp test phase.

The strict debugging flags will look for common computational and numerical issues, such as

  1. Floating-point math exceptions: div-by-zero, invalid computations (i.e. SQRT(-1), LOG(-1), etc.)
  2. Array-out-of-bounds errors
  3. Creation of array temporaries

Furthermore, the Unit Tester compare the output of the sp and mp test phases. If these outputs differ, then this indicates a parallelization error.

--Bob Y. 10:08, 25 April 2014 (EDT)

Specifying unit test options with an input file

Before you can start the GEOS-Chem Unit Tester, you must first specify the options for the unit test simulations in an input file. You can specify the directory paths for your system, the location of the GEOS-Chem code directory, the submit command for your system, and the types of simulations that you want the GEOS-Chem Unit Tester to validate. The default input file is named UnitTest.input, but you can cut-n-paste this file to create as many customized input files as you need.

The UnitTest.input file looks like this. All lines beginning with a # character are treated as comments and will be ignored.

#------------------------------------------------------------------------------
#                  GEOS-Chem Global Chemical Transport Model                  !
#------------------------------------------------------------------------------
#BOP
#
# !INCLUDE: UnitTest.input
#
# !DESCRIPTION: Input file that specifies debugging options for the 
#  GEOS-Chem unit tester.
#\\
#\\
# !REMARKS:  
#  To omit individual unit tests (or input settings), place a # comment
#  character in the first column.
#
#  For a complete explanation of how to customize the settings below for
#  your installation, please see these wiki posts:
#
#    http://wiki.geos-chem.org/GEOS-Chem_Unit_Tester#The_INPUTS_section
#    http://wiki.geos-chem.org/GEOS-Chem_Unit_Tester#The_RUNS_section
#
# !REVISION HISTORY: 
# Type 'gitk' at the prompt to browse the revision history.
#EOP
#------------------------------------------------------------------------------
#
# !INPUTS:
#
# %%% ID tags %%%
#
   VERSION          : v11-02
   DESCRIPTION      : Tests GEOS-Chem v11-02
#
# %%% Data path and HEMCO settings %%%
#
   DATA_ROOT        : /n/holylfs/EXTERNAL_REPOS/GEOS-CHEM/gcgrid/data/ExtData
   HEMCO_ROOT       : {DATAROOT}/HEMCO
   VERBOSE          : 3
   WARNINGS         : 3
#
# %%% Code and queue settings %%%
#
   CODE_DIR         : {HOME}/GC/Code.v11-02
   MAKE_CMD         : make -j4 BOUNDS=y DEBUG=y FPEX=y NO_ISO=y BPCH_DIAG=y NC_DIAG=n
   SUBMIT           : sbatch
#
# %%% Unit tester path names %%%
#
   UNIT_TEST_ROOT   : {HOME}/UT
   RUN_ROOT         : {UTROOT}/runs
   RUN_DIR          : {RUNROOT}/{RUNDIR}
   JOB_DIR          : {UTROOT}/jobs
   LOG_DIR          : {UTROOT}/logs/{VERSION}
   PERL_DIR         : {UTROOT}/perl
#
# %%% Web and text display options %%%
#
   TEMPLATE         : {PERLDIR}/ut_template.html
   TXT_GRID         : {LOGDIR}/{VERSION}.results.txt
   WEB_GRID         : {LOGDIR}/{VERSION}.results.html
   WEB_PUSH         : NONE
#
#  %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
#  %%%  OPTIONAL COMMANDS:                                          %%%
#  %%%                                                              %%%
#  %%%  If your system uses the SLURM scheduler, then you can       %%%
#  %%%  define several #SBATCH tags that will be added to the top   %%%
#  %%%  of the job script.  Otherwise you can comment these lines   %%%
#  %%%  out with the # comment character.                           %%%
#  %%%                                                              %%%
#  %%%     NOTE: If you do use these SLURM commands, then also be   %%%
#  %%%     sure to set the SUBMIT command above to "sbatch".        %%%
#  %%%                                                              %%%
#  %%%  The INIT_COMMANDS tag will let you specify any optional     %%%
#  %%%  initialization commands that will be placed at the top      %%%
#  %%%  of the job script.  For example, if your system requires    %%%
#  %%%  that modules need to be loaded from within the job script,  %%%
#  %%%  you can specify those instructions here.                    %%%
#  %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
#
   SLURM_CPUS       : 4
   SLURM_NODES      : 1
   SLURM_TIME       : 1-12:00
   SLURM_MEMORY     : 35000
   SLURM_PARTITION  : huce_intel
   SLURM_CONSTRAINT : broadwell
   SLURM_MAILTYPE   : END

   # Add initialization commands for your system here if necessary
   # Otherwise, leave these commented out
   #INIT_COMMANDS    : . ~/.bashrc
#
# !RUNS:
#  Specify the debugging runs that you want to perform below.
#  You can deactivate runs by commenting them out with "#".
#
#--------|-----------|------|----------------|------------|--------------|-----|
# MET    | GRID      | NEST | SIMULATION     | START DATE | END DATE     |EXTRA|
#--------|-----------|------|----------------|------------|--------------|-----|
# ======= GEOS-Chem benchmark ================================================
  geosfp   4x5         -      benchmark        2016070100   201607010020   -
# ======= Radon-Lead-Beryllium ===============================================
  geosfp   4x5         -      RnPbBe           2016010100   201601010020   -
  merra2   4x5         -      RnPbBe           2016010100   201601010020   -
  geosfp   2x25        -      RnPbBe           2016010100   201601010020   -
  merra2   2x25        -      RnPbBe           2016010100   201601010020   -
# ======= Mercury ============================================================
  geosfp   4x5         -      Hg               2016010100   201601010020   -
  merra2   4x5         -      Hg               2016010100   201601010020   -
  geosfp   2x25        -      Hg               2016010100   201601010020   -
  merra2   2x25        -      Hg               2016010100   201601010020   -
# ======= Tagged Mercury =====================================================
  geosfp   4x5         -      tagHg            2016010100   201601010020   -
  merra2   4x5         -      tagHg            2016010100   201601010020   -
# ======= POPs ===============================================================
  geosfp   4x5         -      POPs             2016070100   201607010020   -
  merra2   4x5         -      POPs             2016070100   201607010020   -
  geosfp   2x25        -      POPs             2016070100   201607010020   -
  merra2   2x25        -      POPs             2016070100   201607010020   -
# ======= Methane ============================================================
  geosfp   4x5         -      CH4              2016070100   201607010020   -
  merra2   4x5         -      CH4              2016070100   201607010020   -
  geosfp   2x25        -      CH4              2016070100   201607010020   -
  merra2   2x25        -      CH4              2016070100   201607010020   -
  geosfp   025x03125   na     CH4              2016070100   201607010010   -
  merra2   05x0625     na     CH4              2016070100   201607010020   -
# ======= Tagged O3 ==========================================================
  geosfp   4x5         -      tagO3            2016070100   201607010020   -
  merra2   4x5         -      tagO3            2016070100   201607010020   -
  geosfp   2x25        -      tagO3            2016070100   201607010020   -
  merra2   2x25        -      tagO3            2016070100   201607010020   -
# ======= Tagged CO ==========================================================
  geosfp   4x5         -      tagCO            2016070100   201607010020   -
  merra2   4x5         -      tagCO            2016070100   201607010020   -
  geosfp   2x25        -      tagCO            2016070100   201607010020   -
  merra2   2x25        -      tagCO            2016070100   201607010020   -
# ======= Carbon Dioxide =====================================================
  geosfp   2x25        -      CO2              2016070100   201607010020   -
  merra2   2x25        -      CO2              2016070100   201607010020   -
# ======= Offline Aerosols ===================================================
  geosfp   4x5         -      aerosol          2016070100   201607010020   - 
  merra2   4x5         -      aerosol          2016070100   201607010020   - 
  geosfp   2x25        -      aerosol          2016070100   201607010020   -
  merra2   2x25        -      aerosol          2016070100   201607010020   -
# ======= Standard ===========================================================
  geosfp   4x5         -      standard         2016070100   201607010020   -
  merra2   4x5         -      standard         2016070100   201607010020   -
  geosfp   2x25        -      standard         2016070100   201607010020   -
  merra2   2x25        -      standard         2016070100   201607010020   -
# ======= Tropchem ===========================================================
  geosfp   4x5         -      tropchem         2016070100   201607010020   -
  merra2   4x5         -      tropchem         2016070100   201607010020   -
  geosfp   2x25        -      tropchem         2016070100   201607010020   -
  merra2   2x25        -      tropchem         2016070100   201607010020   -
# ======= Complex SOA (w/o and w/ SVPOA) =====================================
  geosfp   4x5         -      complexSOA       2016070100   201607010020   -
  merra2   4x5         -      complexSOA       2016070100   201607010020   -
  geosfp   2x25        -      complexSOA       2016070100   201607010020   -
  merra2   2x25        -      complexSOA       2016070100   201607010020   -
  geosfp   4x5         -      complexSOA_SVPOA 2016070100   201607010020   -
  merra2   4x5         -      complexSOA_SVPOA 2016070100   201607010020   -
  geosfp   2x25        -      complexSOA_SVPOA 2016070100   201607010020   -
  merra2   2x25        -      complexSOA_SVPOA 2016070100   201607010020   -
# ======= Acid uptake on dust ================================================
  geosfp   4x5         -      aciduptake       2016070100   201607010020   -
  geosfp   2x25        -      aciduptake       2016070100   201607010020   -
# ======= Marine POA =========================================================
  geosfp   4x5         -      marinePOA        2016070100   201607010020   -
  geosfp   2x25        -      marinePOA        2016070100   201607010020   -
# ======= TOMAS aerosol microphysics =========================================
  geosfp   4x5         -      TOMAS15          2016070100   201607010030   -
  geosfp   4x5         -      TOMAS40          2016070100   201607010030   -
# ======= RRTMG online radiative transfer ====================================
  geosfp   4x5         -      RRTMG            2016070100   201607010040   -
  merra2   4x5         -      RRTMG            2016070100   201607010040   -
# geosfp   2x25        -      RRTMG            2016070100   201607010040   -
# merra2   2x25        -      RRTMG            2016070100   201607010040   -
# ======= Nested model runs ==================================================
  merra2   05x0625     as     tropchem         2016070100   201607010020   -
  merra2   05x0625     na     tropchem         2016070100   201607010020   -
  geosfp   025x03125   ch     tropchem         2016070100   201607010010   -
  geosfp   025x03125   na     tropchem         2016070100   201607010010   -
 !END OF RUNS:
#EOP
#------------------------------------------------------------------------------

The INPUTS section

Under the !INPUTS: section, you can customize the directory paths and other options for your system. As seen above, most of these inputs are self-explanatory.

Option DescriptionMet field type
VERSION An ID tag that will be added to all log files and output files.
DESCRIPTION A short (1-sentence) description of what features are being validated by this unit test.
DATA_ROOT Specifies the path for your root-level data directory.
HEMCO_ROOT Specifies the top-level path for the HEMCO data directory tree.
  • The {DATAROOT} token in HEMCO_ROOT will be replaced with the value you specify for RUN_ROOT option. Leave this as-is.
VERBOSE Specifies the level of debug output that will be sent to the HEMCO log file. (0=no debug output; 3=max debug output)
  • Recommended setting: 0
WARNINGS Specifies the level of warning messages that will be sent to the HEMCO log file. (0=no warnings; 3=max warnings)
  • Recommended setting: 1
CODE_DIR Specifies the path of the source code directory.
MAKE_CMD Specifies the make command for your system. Usually this is make, but on some systems this may be gmake. Recommended options are:
  • BOUNDS=y: Turns on array-out-of-bounds checking
  • DEBUG=y: Turns off optimization and enables features that will facilitate running GEOS-Chem in a debugger (if need be). Will cause GEOS-Chem to check for array temporaries. Also enables program traceback.
  • NO_ISO=y: Disables the ISORROPIA aerosol thermodynamical equilibrium module, which is known to produce random numerical noise.
  • FPEX=y: Turns on checks for floating-point issues (div-by-zero, floating invalid, underflow, overflow, etc).
  • BPCH_DIAG=y: Turns on the bpch diagnostics. This option is turned on by default in GEOS-Chem v11-02.
  • NC_DIAG=y: Turns on netCDF diagnostics. By default, this will prevent the bpch diagnostic code from being compiled into the GEOS-Chem executable.
    • NOTE: we recommend compiling with either BPCH_DIAG=y or NC_DIAG=y, but not both.
SUBMIT Specifies the command that will send the Unit Test job to your queue system.
  • If you don't have a queue system, leave this blank, and the Unit Tester will run interactively.
UNIT_TEST_ROOT Specifies the path to the GEOS-Chem Unit Tester.
RUN_ROOT Specifies the top-level unit test run directories.

Leave this as-is.

RUN_DIR Specifies the run directory subdirectory.
  • The {RUNDIR} token in RUN_DIR will be replaced with the name of a subdirectory.

Leave this as-is.

JOB_DIR Specifies the directory where the unit test job script will be created.

Leave this as-is.

LOG_DIR Specifies the directory where log files from this unit test simulation will be sent.
  • The {VERSION} token in LOG_DIR will be replaced with the value of VERSION that you specify above. This allows the Unit Tester to send log files from different unit tests to subdirectories of LOG_DIR. This will let you preserve older log files.
PERL_DIR Specifies the directory where the unit test Perl scripts are found.
  • Leave this as-is.
TEMPLATE Specifies the template HTML file that will be used to create a web page with unit test results.
  • Leave this as-is.
TXT_GRID Specifies the path of this summary text file
  • The {LOGDIR} token in TXT_GRID will be replaced with the value of LOG_DIR. Leave this as-is.
  • The {VERSION} token in TXT_GRID will be replaced with the value of VERSION that you specify above. Leave this as-is.
WEB_GRID Specifies the name of the web page that will contain the unit test results.
  • This web page is created from the template HTML file specified with the TEMPLATE option.
  • The {LOGDIR} token in WEB_GRID will be replaced with the value of LOG_DIR. Leave this as-is.
  • The {VERSION} token in WEB_GRID will be replaced with the value of VERSION that you specify above. Leave this as-is.
WEB_PUSH Specifies the remote server to which the WEB_GRID web page will be uploaded. Options are:
  • NONE: Do not upload the page
  • user@host|directory/: Specifies the remote directory. (Example: geoschem@fas.harvard.edu|public_html/ut/)
The following commands are optional.
Most of these options are used to schedule the unit test jobs with the SLURM scheduler.
If your computer cluster does not use SLURM, you can comment all of these settings out in the UnitTest.input file.
SLURM_CPUS Specifies the number of CPUs that will be used for the unit test (with #SBATCH -c)
SLURM_NODES Specifies the number of nodes that will be used for the unit test (with #SBATCH -n)
  • NOTE: For GEOS-Chem "Classic" simulations, this should always = 1.
SLURM_TIME Specifies the requested time for the run. Recommended format: DD-HH:MM
SLURM_MEM Specifies the total amount of memory (in MB) needed to run the unit tests.
  • NOTE: If you are going to run 2° x 2.5° and/or nested-grid unit tests, then pick a high number like 30000 MB = 30 GB.
SLURM_PARTITION Specifies the partition (aka queue) in which the unit tests will run (with #SBATCH -p).
SLURM_CONSTRAINT Allows you to restrict the unit test job to CPUs of a specific type (with #SBATCH --constraint=. For example
  • intel will restrict the unit test job to Intel CPUs
  • haswell will restrict the unit test job to Intel CPUs of type "haswell"
  • broadwell will restrict the unit test job to Intel CPUs of type "broadwell"
SLURM_MAILTYPE Specifies when you would like to receive updates from the SLURM scheduler (with #SBATCH --mail-type=).
  • The most common option is END, which notifies you when the unit test job ends. You can also use ALL which will send you an email when the job starts, ends, and if it dies prematurely.
SLURM_MAILUSER Specifies the email address (with #SBATCH --mail-user=) where notifications from SLURM will be sent.
SLURM_STDOUT Specifies the file that contains the redirected "stdout" stream (i.e. echo-back of commands to the screen).
  • Normally this is: {LOGDIR}/{VERSION}.stdout.log
SLURM_STDERR Specifies the file that contains the redirected "stderr" stream (i.e. echo-back of error messages to the screen).
  • NOTE: If the unit test fails, you should check this file first to see what the specific error messages were.
INIT_COMMANDS Specify can specify any additional commands that are specific to your computer system. For example, you might need to source a .bashrc file, or to issue a command to load modules. You can leave this commented out otherwise.

Note that you can use a symbolic link (i.e. /home/{USER}/UT/ in the RUN_ROOT, RUN_DIR, JOB_DIR, LOG_DIR and PERL_DIR variables. This can help you shorten the file paths.

--Bob Yantosca (talk) 14:16, 21 March 2018 (UTC)

The RUNS section

Under the !RUNS: section, you will list each of the combinations that you want the GEOS-Chem Unit Tester to validate. Simply list the following information under the proper column heading. Lines starting with # will be ignored.

# !RUNS:
#  Specify the debugging runs that you want to perform below.
#  You can deactivate runs by commenting them out with "#".
#
#--------|-----------|------|----------------|------------|--------------|-----|
# MET    | GRID      | NEST | SIMULATION     | START DATE | END DATE     |EXTRA|
#--------|-----------|------|----------------|------------|--------------|-----|
  geosfp   4x5         -      benchmark        2016070100   201607010020   -
  geosfp   4x5         -      RnPbBe           2016010100   201601010020   -
  geosfp   4x5         -      Hg               2016010100   201601010020   -
  geosfp   4x5         -      tagHg            2016010100   201601010020   -
  geosfp   4x5         -      POPs             2016070100   201607010020   -
  geosfp   4x5         -      CH4              2016070100   201607010020   -
  geosfp   025x03125   na     CH4              2016070100   201607010010   -
  geosfp   4x5         -      tagO3            2016070100   201607010020   -
  geosfp   4x5         -      tagCO            2016070100   201607010020   -
  geosfp   2x25        -      CO2              2016070100   201607010020   -
  geosfp   4x5         -      aerosol          2016070100   201607010020   - 
  geosfp   4x5         -      standard         2016070100   201607010020   -
  geosfp   4x5         -      tropchem         2016070100   201607010020   -
  geosfp   4x5         -      complexSOA       2016070100   201607010020   -
  geosfp   4x5         -      complexSOA_SVPOA 2016070100   201607010020   -
  geosfp   4x5         -      aciduptake       2016070100   201607010020   -
  geosfp   4x5         -      marinePOA        2016070100   201607010020   -
  geosfp   4x5         -      TOMAS15          2016070100   201607010030   -
  geosfp   4x5         -      TOMAS40          2016070100   201607010030   -
  geosfp   4x5         -      RRTMG            2016070100   201607010040   -
  geosfp   025x03125   ch     tropchem         2016070100   201607010010   -
  geosfp   025x03125   na     tropchem         2016070100   201607010010   -
 !END OF RUNS:
#EOP
#------------------------------------------------------------------------------

NOTE: For clarity, only simulations using GEOS-FP meteorology have been shown above. You can also schedule simulations using MERRA-2 meteorology.

Options:

MET
Specifies the met field type. Allowable values are:
Value Met field type
geosfp Selects the GEOS-FP met fields.
merra2 Selects the MERRA-2 met fields.
GRID
Specifies the horizontal grid. Allowable values are:
Value Grid type
4x5 Selects the GMAO 4° x 5° horizontal grid.
2x25 Selects the GMAO 2° x 2.5° horizontal grid.
05x0666 Selects the GMAO 0.5° x 0.666° grid for use with GEOS-5 met fields only.
(You must also specify a value for NEST.)
05x0625 Selects the GMAO 0.5° x 0.625° grid for use with MERRA-2 met fields only.
(You must also specify a value for NEST.)
025x03125 Selects the GMAO 0.25° x 0.3125° grid for use with GEOS-FP met fields only.
(You must also specify a value for NEST.)
NEST
Specifies the nested grid. May only be used if the value for GRID is 05x0625 or 025x03125. Allowable values are:
Value Nested grid option
- Skips the nested grid option (default). Use this for global simulations.
as Selects the Asia (AS) nested grid option for use with MERRA-2 met fields only.
ch Selects the China (CH) nested grid option for use with GEOS-FP met fields only.
eu Selects the Europe (EU) nested grid option.
na Selects the North American (NA) nested grid option.
SIMULATION
Specifies the simulation type. Allowable values are:
Value Simulation
Full-chemistry simulations
benchmark Selects the Benchmark chemistry mechanism used in the GEOS-Chem benchmark simulations. This is the Standard chemistry mechanism, plus both complex SOA and simple SOA.
standard Selects the Standard chemistry mechanism.
tropchem Selects the The NOx-Ox-HC-Aer-Br tropospheric chemistry simulation.
complexSOA Selects the Secondary organic aerosols simulation, without semi-volatile POA.
complexSOA_SVPOAa Selects the Secondary organic aerosols simulation, with semi-volatile POA.
aciduptake Selects the Standard chemistry simulation, with acid uptake on dust aerosols turned on.
marinePOA Selects the Standard chemistry simulation, with marine primary organic aerosols turned on.
RRTMG Selects the The NOx-Ox-HC-Aer-Br tropospheric chemistry simulation, with RRTMG turned on.
Specialty simulations
RnPbBe Selects the Radon-Lead-Beryllium simulation. A passive species is also included in this simulation by default.
Hg Selects the Mercury simulation.
tagHg Selects the Tagged mercury simulation.
POPs Selects the Persistent Organic Pollutants (POPs) simulation.
CH4 Selects the CH4 simulation.
tagCO Selects the Tagged CO simulation.
tagO3 Selects the Simple tagged O3 simulation (w/ 2 tracers: Total O3 and Stratospheric O3).
CO2 Selects the CO2 simulation.
TOMAS15 Selects the TOMAS aerosol microphysics simulation with 15 size bins.
TOMAS40 Selects the TOMAS aerosol microphysics simulation with 40 size bins.
START DATE
Specifies the start date (YYYYMMDD) and hour (hh) of the unit test.
END DATE
Specifies the ending date (YYYYMMDD), hour (hh), and minutes (mm) of the unit test.
  • In most cases, a 1-hour simulation is sufficient to detect bugs.
  • For unit tests at 2° x 2.5° resolution or higher, you can end the simulation after one timestep, which is less than an hour.
EXTRA?
Specifies any simulation-specific Makefile options that need to be set at compile time for a particular simulation. Typical values are:

Further notes about the !RUNS section:

  1. We typically run from 2016070100 to 2016070101 for GEOS-FP and MERRA-2 met fields.
  2. The actual year for the met fields does not matter so much for the unit tests. The unit tests runs the same simulation twice (with and without parallelization) and then tests for identical results.

--Bob Yantosca (talk) 14:35, 21 March 2018 (UTC)

Running the GEOS-Chem Unit Tester

Once you have added your desired options to the input file, you may then proceed to start the GEOS-Chem Unit Tester. At the Unix prompt, type:

cd perl 
./gcUnitTest FILENAME

where FILENAME is the name the input file (e.g. UnitTest.input) that you have just edited. The settings in the input file will be used in the Unit Test simulations. If FILENAME is omitted, then the gcUnitTest script will use UnitTest.input by default.

You will usually want the Unit Test simulations to run in a queue on your computational cluster. For example, you can set up a Unit Test job to run overnight. The GEOS-Chem Unit Tester will compile and run the code for each of the types of simulations that you specified in the input file. The Unit Tester will compile and run GEOS-Chem twice for each type of simulation (once with OpenMP parallelizaton turned off, and again with OpenMP parallelization turned on). Then the Unit tester will check to see if the output files from both simulations are identical.

Log-file output generated by the GEOS-Chem Unit Tester is sent to the logs/{VERSION} directory, where {VERSION} denotes the version tag that you assigned in the input file. (This allows you to preserve log file output from several previous Unit Test simulations.)

--Bob Yantosca (talk) 19:05, 24 April 2017 (UTC)

Examining the results

Output files written to each unit test run directory

The GEOS-Chem Unit Tester will run GEOS-Chem simulations in the subdirectories of the runs/ directory. Each individual subdirectory of runs/ corresponds to a particular unit test, or combination of met field + horizontal grid + simulation type. Some example run directory names are:

runs/4x5_tropchem   # Run dir for 4x5 "tropchem" simulations (both GEOS-FP and MERRA-2)
runs/2x25_CO2       # Run dir for 2x25 "CO2" simulations (both GEOS-FP and MERRA-2)
runs/4x5_RnPbBe     # Run dir for 4x5 grid "Rn-Pb-Be" simulations (both GEOS-FP and MERRA-2)
etc.

etc. In each of these run directories, GEOS-Chem will create several output files containing tracer concentrations and diagnostic quantities. These files are:

File Description Notes
GEOSChem_restart.*.sp Restart file containing instantaneous species concentrations at the end of the single-processor test. Includes all species (advected and non-advected).

For a complete list of GEOS-Chem species, please see our Species in GEOS-Chem wiki page.

GEOSChem_restart.*.mp Restart file containing species concentrations at the end of the multi-processor test.
HEMCO_restart.*.sp Restart file generated by HEMCO at the end of the single processor test. The HEMCO restart archive certain emissions quantities (such as soil NO deposition) that need to be preserved between multiple stages of a long GEOS-Chem simulation.
HEMCO_restart.*.mp Restart file generated by HEMCO at the end of the multi-processor test
HEMCO_diagnostics.*.sp netCDF file containing diagnostics archived by HEMCO, from the single-processor test. The HEMCO_Diagn.rc input file (located in each run directory) specifies which quantities from HEMCO will be archived as diagnostic output.

For more information, please see the the Diagnostics section of The HEMCO User's Guide.

HEMCO_diagnostics.*.mp netCDF file containing diagnostics archived by HEMCO, from the multi-processor test.
HEMCO.log.sp HEMCO log file generated by the single-processor test. The HEMCO.log* files contains detailed "echo-back" information about the creation of the HEMCO diagnostic containers and I/O processes.

These files are not scanned by the GEOS-Chem Unit Tester. If a unit test fails, you should look at these HEMCO.log* files to determine if an error occurred in emissions.

HEMCO.log.mp HEMCO log file generated by the multi-processor test.
If you specified bpch diagnostic output (with BPCH_DIAG=y), the following files will be created:
trac.avg.*.sp Binary punch (bpch) format file containing diagnostic output from the the single-processor test. For a list of available diagnostics, please see our List of diagnostics archived to bpch format wiki page.
trac.avg.*.mp Binary punch (bpch) format file containing diagnostic output from the the multi-processor test.
If you specified netCDF diagnostic output (with NC_DIAG=y), the following files will be created:
GEOSChem.*.YYYYMMDD_hhmmz.nc4.sp netCDF files containing diagnostic output from the single-processor test. One file will be created for each diagnostic collection that is specified in the HISTORY.rc input file.

For a list of diagnostic collections, please see our List of diagnostics archived to netCDF format wiki page.

GEOSChem.*.YYYYMMDD_hhmmz.nc4.mp netCDF files containing diagnostic output from the multi-processor test.

--Bob Yantosca (talk) 15:26, 21 March 2018 (UTC)

Log files written to the unit test log directory

The GEOS-Chem Unit Tester will also create a series of log files in the logs/{VERSION} directory, where {VERSION} is specifed in the The !INPUTS section of the unit test input file. These log files are:

File Description Notes
{VERSION}.stderr.log
(SLURM scheduler)

job.{VERSION}.o*
(GridEngine scheduler)
File containing the Unix stderr output. All error and warning messages will get sent to this file. This is created if you run the Unit Tester in the Oracle Grid Engine queue system. You normally won't have to look at this file, unless a particular simulation dies unexpectedly. The output contained can be very useful in determining where the error ocurrred.
{VERSION}.stdout.log File containing the Unix stdout output. In most cases you will not need to look at this file.
{VERSION}.results.log Log file containing the results from each individual unit test simulation that you ran.

More information on this file is contained below.

{VERSION}.results.html Web page (HTML) containing a graphical representation of the unit test results.

More information on this file is contained below.

{VERSION}.results.txt Text file containing a table representation of the unit test results.

More information on this file is contained below.

{VERSION}.{MET}_{GRID}_{SIM}.log.sp
(Global simulations)

{VERSION}.{MET}_{GRID}_{SIM}_{NEST}.log.sp
(Nested-grid simulations)
GEOS-Chem log file output from the single-processor test phase of the unit test for the given met field, grid, simulation type (and, if applicable, nested-grid region).
{VERSION}.{MET}_{GRID}_{SIM}.log.mp
(Global simulations)

{VERSION}.{MET}_{GRID}_{SIM}_{NEST}.log.mp
(Nested-grid simulations)
GEOS-Chem log file output from the multi-processor test phase of the unit test for the given met field, grid, simulation type (and, if applicable, nested-grid region).


--Bob Yantosca (talk) 19:20, 24 April 2017 (UTC)

Unit test results and error messages

The {VERSION}.results.log file will contain printout similar to the following. Looking at this file will tell you if the output from the sp and mp test phases of each unit test simulation were identical to each other, or if they differed.

If you compiled with BPCH_DIAG=y then your results file will look similar to this:

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%
%%%  GEOS-CHEM UNIT TEST RESULTS FOR VERSION: BpchOnly_Mar19
%%%  job sent to queue @ 2018/03/19 14:17:43
%%%
%%%  DESCRIPTION: c40ec4d Restructure calculation of binned dust AOD diag to avoid seg fault in GCHP
%%%
%%%  BUILT WITH: make -j8 BOUNDS=y DEBUG=y FPEX=y NO_ISO=y BPCH_DIAG=y NC_DIAG=n
%%%
%%%  This is the main log file, which shows output from
%%%  each individual phase of the unit test sequence.
%%%
%%%  Log files from individual unit-test runs are also
%%%  stored in this same directory.
%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
###############################################################################
### VALIDATION OF GEOS-CHEM OUTPUT FILES
### Run ID: geosfp_4x5_RnPbBe
##@
### IDENTICAL : GEOSChem_restart.201607010020.nc.{sp,mp}
### IDENTICAL : HEMCO_diagnostics.201607010000.nc.{sp,mp}
### IDENTICAL : HEMCO_restart.201607010020.nc.{sp,mp}
### IDENTICAL : trac_avg.geosfp_4x5_RnPbBe.201607010000.{sp,mp}
##%
###############################################################################
... etc ...

If, on the other hand, you compiled with NC_DIAG=y then your results file will look similar to this:

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%
%%%  GEOS-CHEM UNIT TEST RESULTS FOR VERSION: Nc_Mar19
%%%  job sent to queue @ 2018/03/19 14:17:43
%%%
%%%  DESCRIPTION: c40ec4d Restructure calculation of binned dust AOD diag to avoid seg fault in GCHP
%%%
%%%  BUILT WITH: make -j8 BOUNDS=y DEBUG=y FPEX=y NO_ISO=y BPCH_DIAG=n NC_DIAG=y
%%%
%%%  This is the main log file, which shows output from
%%%  each individual phase of the unit test sequence.
%%%
%%%  Log files from individual unit-test runs are also
%%%  stored in this same directory.
%%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
###############################################################################
### VALIDATION OF GEOS-CHEM OUTPUT FILES
### Run ID: geosfp_4x5_RnPbBe
##@
### IDENTICAL : GEOSChem.CloudConvFlux.20130701_0000z.nc4.{sp,mp}
### IDENTICAL : GEOSChem.DryDep.20130701_0000z.nc4.{sp,mp}
### IDENTICAL : GEOSChem.LevelEdgeDiags.20130701_0000z.nc4.{sp,mp}
### IDENTICAL : GEOSChem.RadioNuclide.20130701_0000z.nc4.{sp,mp}
### IDENTICAL : GEOSChem.SpeciesConc.20130701_0000z.nc4.{sp,mp}
### IDENTICAL : GEOSChem.StateMet.20130701_0000z.nc4.{sp,mp}
### IDENTICAL : GEOSChem.WetLossConv.20130701_0000z.nc4.{sp,mp}
### IDENTICAL : GEOSChem.WetLossLS.20130701_0000z.nc4.{sp,mp}
### IDENTICAL : GEOSChem_restart.201307010020.nc.{sp,mp}
### IDENTICAL : HEMCO_diagnostics.201307010000.nc.{sp,mp}
### IDENTICAL : HEMCO_restart.201307010020.nc.{sp,mp}
##%
###############################################################################

Possible outcomes for each set of files being compared are:

Outcome Description
IDENTICAL This means that the corresponding .sp and *.mp files being compared against each other are 100% bitwise identical.
DIFFERENT This means that the corresponding *.sp. and *.mp files being compared against each other differ in some way.
MISSING This means that one of the corresponding *.sp or *.mp files could not be found in the unit test run directory.

The {VERSION}.stderr.log will contain any detailed error messages (such as error traceback output). Check this if your unit tests have halted unexpectedly.

--Bob Yantosca (talk) 16:03, 21 March 2018 (UTC)

How unit tests are scored

The following criteria are used to score GEOS-Chem unit tests:

Result Criteria
PASS
  • All corresponding pairs of {*.sp, *.mp} restart files are IDENTICAL, and
  • All corresponding pairs of {*.sp, *.mp} diagnostics files are IDENTICAL
PASS WITH WARNINGS
  • All corresponding pairs of {*.sp, *.mp} restart files are IDENTICAL, but
  • One or more corresponding pairs of {*.sp, *.mp} diagnostics files are DIFFERENT
FAIL
  • One or more corresponding pairs of {*.sp, *.mp} restart files are DIFFERENT
  • Any *.sp or *.mp restart file is MISSING
  • Any *.sp or *.mp diagnostics file is MISSING

--Bob Yantosca (talk) 16:41, 21 March 2018 (UTC)

Unit test results displayed as a web page

The GEOS-Chem Unit Tester will also create a web page that summarizes the results in an easy-to-read table. The name of the web page is specified with the WEB_GRID option in the !INPUTS: section of the unit test input file. The web page will contain output similar to this example:

GEOS-Chem Unit Test Results

Version: BpchOnly_Mar19
Date Submitted: 2018/03/19 14:17:43
Description: c40ec4d Restructure calculation of binned dust AOD diag to avoid seg fault in GCHP
Compiled with: make -j8 BOUNDS=y DEBUG=y FPEX=y NO_ISO=y BPCH_DIAG=y NC_DIAG=n


Unit Tests Stan-
dard
Trop
chem
complex
SOA
complex
SOA-SVPOA
Acid
Uptk
Marine
POA
RR
TMG
Rn-
Pb-Be
Hg TagHg POPs TagCO TagO3 CH4 CO2 Aer-
osol
TOMAS
15
TOMAS
40
MERRA-2 @ 4° x 5°                                    
GEOS-FP @ 4° x 5°                                    
MERRA-2 @ 2° x 2.5°                                    
GEOS-FP @ 2° x 2.5°                                    


Benchmark
Unit Tests
Bench
mark
GEOS-FP @ 4° x 5°  

<td width="40"> </td>

Nested-Grid
Unit Tests
Trop
chem
CH4 Hg
GEOS-FP NA @
0.25° x 0.3125°
     
GEOS-FP CH @
0.25° x 0.3125°
     
MERRA-2 NA @
0.5° x 0.625°
     
MERRA-2 AS @
0.5° x 0.625°
     
 

LEGEND

  The unit test was successful.
  Further investigation is necessary (e.g. The restart files were identical but the diagnostic output differed).
  The unit test failed.
  A unit test was not performed for this combination of met field, horizontal grid, and simulation type.


--Bob Yantosca (talk) 16:43, 21 March 2018 (UTC)

Unit test results displayed as a text file

The GEOS-Chem Unit Tester will also send a simple text summary of the unit test results to a file named {VERSION}.results.txt in the log directory. (NOTE: {VERSION} will be overwritten with the version string that you specified in the UnitTest.input file.) The file will contain output similar to this:

GEOS-Chem Unit Test             : RESULT
------------------------------------------
geosfp_4x5_RnPbBe               : GREEN
merra2_4x5_RnPbBe               : GREEN
... etc ...

--Bob Yantosca (talk) 16:47, 21 March 2018 (UTC)

Cleaning old files

To clean all of the files created by the unit tester (job scripts, logs, bpch files), type:

cd perl  
./cleanFiles

--Bob Yantosca (talk) 19:44, 24 April 2017 (UTC)