Difference between revisions of "Performing Difference Tests with GEOS-Chem"

From Geos-chem
Jump to: navigation, search
Line 1: Line 1:
 
On this page we show you how to use the [[GEOS-Chem Unit Tester]] to generate and use difference tests to evaluate modifications to GEOS-Chem.
 
On this page we show you how to use the [[GEOS-Chem Unit Tester]] to generate and use difference tests to evaluate modifications to GEOS-Chem.
 +
 +
== Overview ==
 +
 +
A difference test validates two versions of GEOS-Chem. It compares model output from a version of GEOS-Chem in which you have made updates (aka the development code, or "Dev") against a version of GEOS-Chem with known behavior (aka the reference code, or "Ref").
 +
 +
Typically you can only expect a difference test to pass if you expect the Dev code run to produce results identical to the Ref run code.  This will be true if Dev differs from Ref only by structural changes (i.e. modifications in how data gets passed from one place to another, replacing old Fortran code with more modern equivalent code). If Dev contains scientific changes (new chemistry reactions, addition of tracers, new photolysis reactions, etc.), then you can still do a difference but the DiffTest will not pass. However, there are additional tools to explore the differences that may be useful for validation.
 +
 +
DiffTests are not meant to replace benchmarking. The [[GEOS-Chem Support Team]] will continue to rely on the 1-month and 1-year benchmarks to validate the code.
  
 
== Downloading the GEOS-Chem Unit Tester ==
 
== Downloading the GEOS-Chem Unit Tester ==
Line 139: Line 147:
 
  Ref/
 
  Ref/
 
  summarizeDiagDiffs.sh
 
  summarizeDiagDiffs.sh
 +
 +
These files and directories are described below in more detail.
 +
 +
{| border=1 cellspacing=0 cellpadding=5
 +
|-bgcolor="#CCCCCC"
 +
!width="100px"|Name
 +
!width="825px"|Description
 +
 +
|-bgcolor="#CCCCCC"
 +
!colspan="2"|Directories
 +
 +
|-valign="top"
 +
|<tt>Dev/</tt>
 +
|Run directory containing [[GEOS-Chem_Input_Files|input files]] for the simulation using the development (Dev) code.
 +
 +
|-valign="top"
 +
|<tt>Ref/</tt>
 +
|Run directory containing [[GEOS-Chem_Input_Files|input files]] for the simulation using the reference (Ref) code. This directory contains mostly symbolic links to the <tt>Dev/</tt> directory. If you have updated any of the input files in your Dev run, you will need to remove the symbolic link and manually place a copy of that input file without your modifications in this directory .
 +
 +
|-valign="top"
 +
|<tt>logs/</tt>
 +
|Directory where the log files for the Dev and Ref simulations are stored. If you run a complete difference test, a file with the results will be saved here as well.
 +
 +
|-valign="top"
 +
|<tt>plots/</tt>
 +
|Directory containing files used to create plots comparing the Dev and Ref simulations. The IDL routine <tt>plot_diff.pro</tt> is set up to create species concentration maps, difference maps, ratio maps, zonal difference plots, and zonal concentration plots. That routines uses the input file <tt>PlotDiffs.input</tt>, which is set up to compare output in the <tt>Dev/</tt> and <tt>Ref/</tt> run directories.
 +
 +
|-bgcolor="#CCCCCC"
 +
!colspan="2"|Files
 +
 +
|-valign="top"
 +
|<tt>README</tt>
 +
|Text file containing a description of the DiffTest directory and instructions for running difference tests.
 +
 +
|-valign="top"
 +
|<tt>Makefile</tt>
 +
|Makefile used to drive the entire difference test. For more information, use the command <tt>make help</tt>.
 +
 +
|-valign="top"
 +
|<tt>summarizeDiagDiffs.sh</tt>
 +
|Bash script used to compare binary punch output files using IDL routine <tt>ctm_summarizediff.pro</tt>. Output is printed to the screen and saved to a file in the <tt>logs/</tt> directory.
 +
 +
|-valign="top"
 +
|<tt>ctm_summarizediff.pro</tt>
 +
|IDL script used to locate datablocks that differ in two binary punch files. This routine is called from <tt>summarizeDiagDiffs.sh</tt>.
 +
 +
|-valign="top"
 +
|<tt>locateDiagDiffs.sh</tt>
 +
|Bash script used to compare binary punch output files using <tt>ctm_locatediff.pro</tt>. Output is printed to the screen and saved to a file in the <tt>logs/</tt> directory. We recommend using <tt>summarizeDiff.sh</tt> first for an overview the differences because the differences returned by this script may be numerous, resulting in a very large log file.

Revision as of 17:06, 12 January 2017

On this page we show you how to use the GEOS-Chem Unit Tester to generate and use difference tests to evaluate modifications to GEOS-Chem.

Overview

A difference test validates two versions of GEOS-Chem. It compares model output from a version of GEOS-Chem in which you have made updates (aka the development code, or "Dev") against a version of GEOS-Chem with known behavior (aka the reference code, or "Ref").

Typically you can only expect a difference test to pass if you expect the Dev code run to produce results identical to the Ref run code. This will be true if Dev differs from Ref only by structural changes (i.e. modifications in how data gets passed from one place to another, replacing old Fortran code with more modern equivalent code). If Dev contains scientific changes (new chemistry reactions, addition of tracers, new photolysis reactions, etc.), then you can still do a difference but the DiffTest will not pass. However, there are additional tools to explore the differences that may be useful for validation.

DiffTests are not meant to replace benchmarking. The GEOS-Chem Support Team will continue to rely on the 1-month and 1-year benchmarks to validate the code.

Downloading the GEOS-Chem Unit Tester

First, make sure that your system has these software packages installed. (Most of these come standard with your Unix-based operating system.)

Next, clone the GEOS-Chem Unit Tester package with the command:

git clone https://bitbucket.org/gcst/geos-chem-unittest UT

This will create a copy of the GEOS-Chem Unit Tester package in a directory named UT for short.

NOTE: The Git clone process may take a few minutes to complete depending on your connection speed.

Editing the DiffTest.input file

Once you have downloaded the GEOS-Chem Unit Tester to your disk space, switch to the perl/ directory:

cd UT/perl

In this directory there is a Perl script named gcCreateDiffTest that you will use to generate fresh copies of GEOS-Chem difference test run directories. This script uses an input file named DiffTest.input, which is also located in the perl directory.

Your DiffTest.input file will look something like this:

#------------------------------------------------------------------------------
#                  GEOS-Chem Global Chemical Transport Model                  !
#------------------------------------------------------------------------------
#BOP
#
# !DESCRIPTION: Input file that specifies configuration for creating a
#  DiffTest directory from the UnitTester. 
#\\
#\\
# !REMARKS:
#  For a complete description of how to customize the settings in the
#  INPUTS and RUNS sections, see the following wiki posts:
#
#   wiki.geos-chem.org/Creating_GEOS-Chem_run_directories#Section_1:_INPUTS
#   wiki.geos-chem.org/Creating_GEOS-Chem_run_directories#Section_2:_RUNS
#
# !REVISION HISTORY: 
#  01 Oct 2015 - E. Lundgren - Initial version, based on CopyRunDirs.input
#EOP
#------------------------------------------------------------------------------
#
# !INPUTS:
#
# %%% ID tags %%%
#
   VERSION        : v11-01
   DESCRIPTION    : Create run directory from UnitTest
#
# %%% Data path and HEMCO settings %%%
#
   DATA_ROOT      : /n/holylfs/EXTERNAL_REPOS/GEOS-CHEM/gcgrid/data/ExtData
   HEMCO_ROOT     : {DATAROOT}/HEMCO
   VERBOSE        : 0
   WARNINGS       : 1
#
# %%% Unit tester path names %%%
#
   UNIT_TEST_ROOT : {HOME}/UT
   RUN_ROOT       : {UTROOT}/runs
   RUN_DIR        : {RUNROOT}/{RUNDIR}
   PERL_DIR       : {UTROOT}/perl
#
# %%% Reference (Ref) and Development (Dev) codes %%%
#
   CODE_REF       : {HOME}/GC/Code.Ref
   CODE_DEV       : {HOME}/GC/Code.Dev
#
# %%% Target directory and copy command %%%
#
   COPY_PATH      : {HOME}/GC/DiffTest/v11-01
   COPY_CMD       : cp -rfL
#
# !RUNS:
#  Specify the runs directories that you want to make DiffTests out of.
#  Here we provide a few examples, but you may copy additional entries from
#  UnitTest.input and modify the dates as needed. You can deactivate
#  certain directories by commenting them out with "#".
#
#--------|-----------|------|------------|------------|------------|---------|
# MET    | GRID      | NEST | SIMULATION | START DATE | END DATE   | EXTRA?  |
#--------|-----------|------|------------|------------|------------|---------|
   geosfp   4x5         -      standard     2013070100   2013070101   -
#  geosfp   4x5         -      tropchem     2013070100   2013070101   -
#  geosfp   4x5         -      soa          2013070100   2013070101   -
#  geosfp   4x5         -      soa_svpoa    2013070100   2013070101   -
#  geosfp   4x5         -      aciduptake   2013070100   2013070101   -
#  geosfp   4x5         -      UCX          2013070100   2013070101   -
#  geosfp   4x5         -      RRTMG        2013070100   2013070101   -
#  geosfp   4x5         -      RnPbBe       2013070100   2013070101   -
#  geosfp   4x5         -      Hg           2013010100   2013010101   -
#  geosfp   4x5         -      POPs         2013070100   2013070101   -
#  geosfp   4x5         -      TOMAS40      2013070100   2013070101   -
#  geosfp   4x5         -      CH4          2013070100   2013070101   -
#  geosfp   4x5         -      tagO3        2013070100   2013070101   -
#  geosfp   4x5         -      tagCO        2013070100   2013070101   -
#  geosfp   2x25        -      CO2          2013070100   2013070101   -
#  geosfp   4x5         -      aerosol      2013070100   2013070101   - 
#  geosfp   025x03125   ch     tropchem     2013070100   201307010010 -
#  geosfp   025x03125   na     tropchem     2013070100   201307010010 -
!END OF RUNS:
#EOP
#------------------------------------------------------------------------------

NOTE: Lines starting with a # character will be treated as comments.

The DiffTest.input file has a layout that is very similar to the CopyRunDirs.input file located in the same directory. The only difference is the addition of CODE_REF and CODE_DEV lines in the INPUTS section. Those lines can be used to specify the paths of the reference (Ref) and development (Dev) codes that you would like to compare.

Generating a GEOS-Chem DiffTest Directory

Once you have edited the DiffTest.input script to your liking, you can use that to generate fresh copies of GEOS-Chem DiffTest directories. Make sure you are in the perl directory, and then type:

./gcCreateDiffTest

If you do not pass a file name to gcCreateDiffTest, then the gcCreateDiffTest script will use the DiffTest.input file that you just modified.

Executing gcCreateDiffTest will create a new GEOS-Chem DiffTest directory corresponding to each entry that you specified in the input file RUNS section. Each run directory will be created as a subdirectory of COPY_PATH that you specified in the input file INPUTS section.

Let's examine the contents of a sample DiffTest_geosfp_4x5_standard run directory. Issue the following commands:

cd ~/GC/rundirs/geosfp_4x5_fullchem  # Change to geosfp_4x5_fullchem run dir 
  
make fileclean                       # Remove any files left over from previous unit test runs

ls -1                                # Get directory listing

And you will see this directory listing:

ctm_summarizediff.pro
Dev/
locateDiagDiffs.sh
logs/
Makefile
plots/
README
Ref/
summarizeDiagDiffs.sh

These files and directories are described below in more detail.

Name Description
Directories
Dev/ Run directory containing input files for the simulation using the development (Dev) code.
Ref/ Run directory containing input files for the simulation using the reference (Ref) code. This directory contains mostly symbolic links to the Dev/ directory. If you have updated any of the input files in your Dev run, you will need to remove the symbolic link and manually place a copy of that input file without your modifications in this directory .
logs/ Directory where the log files for the Dev and Ref simulations are stored. If you run a complete difference test, a file with the results will be saved here as well.
plots/ Directory containing files used to create plots comparing the Dev and Ref simulations. The IDL routine plot_diff.pro is set up to create species concentration maps, difference maps, ratio maps, zonal difference plots, and zonal concentration plots. That routines uses the input file PlotDiffs.input, which is set up to compare output in the Dev/ and Ref/ run directories.
Files
README Text file containing a description of the DiffTest directory and instructions for running difference tests.
Makefile Makefile used to drive the entire difference test. For more information, use the command make help.
summarizeDiagDiffs.sh Bash script used to compare binary punch output files using IDL routine ctm_summarizediff.pro. Output is printed to the screen and saved to a file in the logs/ directory.
ctm_summarizediff.pro IDL script used to locate datablocks that differ in two binary punch files. This routine is called from summarizeDiagDiffs.sh.
locateDiagDiffs.sh Bash script used to compare binary punch output files using ctm_locatediff.pro. Output is printed to the screen and saved to a file in the logs/ directory. We recommend using summarizeDiff.sh first for an overview the differences because the differences returned by this script may be numerous, resulting in a very large log file.