Difference between revisions of "Creating GEOS-Chem run directories"

Revision as of 15:50, 12 January 2017

On this page we show you how to use the GEOS-Chem Unit Tester to generate and use fresh copies of run directories for your GEOS-Chem simulations.

Downloading the GEOS-Chem Unit Tester

First, make sure that your system has these software packages installed. (Most of these come standard with your Unix-based operating system.)

Next, clone the GEOS-Chem Unit Tester package with the command:

git clone https://bitbucket.org/gcst/geos-chem-unittest UT

This will create a copy of the GEOS-Chem Unit Tester package in a directory named UT for short.

NOTE: The Git clone process may take a few minutes to complete depending on your connection speed.

--Bob Yantosca (talk) 15:12, 20 December 2016 (UTC)

Available run directories

Benchmark run directory

The GEOS-Chem Unit Tester includes a run directory called 4x5_standard (GEOS-Chem v11-01 and later) or geosfp_4x5_benchmark (GEOS-Chem v10-01). The standard GEOS-Chem benchmark simulation utilizes the GEOS-FP met fields (4° x 5 °, 72 levels) with the UCX tropospheric-stratospheric chemistry mechanism and secondary organic aerosols included.

To reproduce our latest 1-month benchmark simulation for v11-01, you can edit the CopyRunDirs.input file so that the following line is uncommented under RUNS:

 geosfp   4x5         -      standard     2013070100   2013080100   -

Make sure to check the start and end date so that your simulation will run for July 2013. You can then create the GEOS-Chem benchmark run directory by typing ./gcCopyRunDirs. Navigate to your the newly created geosfp_4x5_benchmark run directory. To compile and run your benchmark simulation, type:

make -j4 mp

To compile only, type:

make -j4 mpbuild

That will create a geos.mp executable file that you can use to submit your GEOS-Chem benchmark simulation to a queue system.

NOTE: If you are compiling GEOS-Chem within the code directory, and not within a run directory created from the GEOS-Chem Unit Tester, you will need to pass the UCX=y option in your make command.

--Melissa Sulprizio (talk) 19:38, 23 June 2015 (UTC)

Other run directories

Run directories are provided for a combination of met field, grid, and simulation type and are named accordingly. For more details on the simulation types, please see our GEOS-Chem chemistry mechanism wiki page.

The following table summarizes the run directories that are available in the GEOS-Chem Unit Tester. Please choose the run directories that are most relevant for your research.

LEGEND

--Melissa Sulprizio (talk) 20:10, 8 December 2016 (UTC)

Editing the CopyRunDirs.input file

Once you have downloaded the GEOS-Chem Unit Tester to your disk space, switch to the perl/ directory:

cd UT/perl

In this directory there is a Perl script named gcCopyRunDirs that you will use to generate fresh copies of GEOS-Chem run directories. This script uses an input file named CopyRunDirs.input, which is also located in the perl directory.

Your CopyRunDirs.input file will look something like this:

#------------------------------------------------------------------------------
#                  GEOS-Chem Global Chemical Transport Model                  !
#------------------------------------------------------------------------------
#BOP
#
# !DESCRIPTION: Input file that specifies configuration for creating and
#  copying a run directory from the UnitTester. 
#\\
#\\
# !REMARKS:
#  For a complete description of how to customize the settings in the
#  INPUTS and RUNS sections, see the following wiki posts:
#
#   wiki.geos-chem.org/Creating_GEOS-Chem_run_directories#Section_1:_INPUTS
#   wiki.geos-chem.org/Creating_GEOS-Chem_run_directories#Section_2:_RUNS
#
# !REVISION HISTORY: 
#  18 Mar 2015 - R. Yantosca - Initial version
#  19 Mar 2015 - E. Lundgren - Simplify content for only copying run dirs
#  19 May 2015 - R. Yantosca - Now can specify VERBOSE and WARNINGS options
#EOP
#------------------------------------------------------------------------------
#
# !INPUTS:
#
# %%% ID tags %%%
#
   VERSION        : v11-01
   DESCRIPTION    : Create run directory from UnitTest
#
# %%% Data path and HEMCO settings %%%
#
   DATA_ROOT      : /n/holylfs/EXTERNAL_REPOS/GEOS-CHEM/gcgrid/data/ExtData
   HEMCO_ROOT     : {DATAROOT}/HEMCO
   VERBOSE        : 0
   WARNINGS       : 1
#
# %%% Unit tester path names %%%
#
   UNIT_TEST_ROOT : {HOME}/UT
   RUN_ROOT       : {UTROOT}/runs
   RUN_DIR        : {RUNROOT}/{RUNDIR}
   PERL_DIR       : {UTROOT}/perl
#
# %%% Target directory and copy command %%%
# 
   COPY_PATH      : {HOME}/GC/rundirs
   COPY_CMD       : cp -rfL
#
# !RUNS:
#  Specify the runs directories that you want to copy below.
#  Here we provide a few examples, but you may copy additional entries from
#  UnitTest.input and modify the dates as needed. You can deactivate copying
#  run certain directories by commenting them out with "#".
#
#--------|-----------|------|------------|------------|------------|---------|
# MET    | GRID      | NEST | SIMULATION | START DATE | END DATE   | EXTRA?  |
#--------|-----------|------|------------|------------|------------|---------|
  geosfp   4x5         -      standard     2013070100   2013080100   -
# geosfp   4x5         -      gc_timing    2013070100   2013070800   -
# geosfp   4x5         -      tropchem     2013070100   2013070101   -
# geosfp   4x5         -      soa          2013070100   2013070101   -
# geosfp   4x5         -      soa_svpoa    2013070100   2013070101   -
# geosfp   4x5         -      aciduptake   2013070100   2013070101   -
# geosfp   4x5         -      UCX          2013070100   2013070101   -
# geosfp   4x5         -      RRTMG        2013070100   2013070101   -
# geosfp   4x5         -      RnPbBe       2013070100   2013070101   -
# geosfp   4x5         -      Hg           2013010100   2013010101   -
# geosfp   4x5         -      POPs         2013070100   2013070101   -
# geosfp   4x5         -      TOMAS40      2013070100   2013070101   -
# geosfp   4x5         -      CH4          2013070100   2013070101   -
# geosfp   4x5         -      tagO3        2013070100   2013070101   -
# geosfp   4x5         -      tagCO        2013070100   2013070101   -
# geosfp   2x25        -      CO2          2013070100   201307010030 -
# geosfp   4x5         -      aerosol      2013070100   2013070101   - 
# geosfp   025x03125   ch     tropchem     2013070100   201307010010 -
# geosfp   025x03125   na     tropchem     2013070100   201307010010 -
# gchp     4x5         -      tropchem     2013070100   2013070101   -
!END OF RUNS:
#EOP
#------------------------------------------------------------------------------

NOTE: Lines starting with a # character will be treated as comments.

The CopyRunDirs.input file has a layout that is very similar to the GEOS-Chem Unit Tester input files (UnitTest.input located within this directory). Like the GEOS-Chem Unit Tester input files, CopyRunDirs.input is composed of an INPUTS section and a RUNS section, which are described below.

Section 1: INPUTS

Under the INPUTS section, you can customize the directory paths and other options for your system. Each configurable input is described in the table below.

Section 2: RUNS

The layout of the RUNS section is identical to the RUNS section in the GEOS-Chem Unit Tester input file. This enables copying and pasting simulation settings text from UnitTest.input into CopyRunDirs.input.

For example, the following line:

#
# !RUNS:
#  Specify the debugging runs that you want to perform below.
#  You can deactivate runs by commenting them out with "#".
#
#--------|-----------|------|------------|------------|------------|---------|
# MET    | GRID      | NEST | SIMULATION | START DATE | END DATE   | EXTRA?  |
#--------|-----------|------|------------|------------|------------|---------|
  geosfp   4x5         -      standard     2013070100   2013080100   -

will tell the gcCopyRunDirs script to create a run directory for a GEOS-Chem simulation with the following settings:

• Using GEOS-FP met fields
• On the 4° x 5° GEOS-Chem horizontal grid
• For the GEOS-Chem standard full-chemistry simulation
• Starting at 00:00 GMT on 2013/07/01
• Ending at 00:00 GMT on 2013/08/01

The date range will be used to initialize the input.geos file that is read during a GEOS-Chem simulation. Once the run directory is created, you may edit these dates within the input.geos file. Note, however, that time ranges must remain within the time range covered by the MET field you are using. You can check the MET field temporal coverage on the GMAO met data products wiki page. Also note that we recommend using the restart files to spin up GEOS-Chem for at least one year prior to your simulation start date. You may use the resulting output restart files as initial values for your simulation.

You can add as many entries to the RUNS section as you wish. Simply comment out the lines containing runs you may wish to copy in the future.

Generating a GEOS-Chem Run Directory

Once you have edited the CopyRunDirs.input script to your liking, you can use that to generate fresh copies of GEOS-Chem run directories. Make sure you are in the perl directory, and then type:

./gcCopyRunDirs

If you do not pass a file name to gcCopyRunDirs, then the gcCopyRunDirs script will use the CopyRunDirs.input file that you just modified.

If you wish, you can create many customized copies of CopyRunDirs.input. For example, suppose you edit CopyRunDirs.input to generate a full-chemistry run directory. You can then save it as a separate file and use it explicitly with gcCopyRunDirs.

cp CopyRunDirs.input CopyRunDirs.fullchem   # Input file set up to only copy the full-chemistry run directories
 
gcCopyRunDirs CopyRunDirs.Hg                 

Executing gcCopyRunDirs will create a new GEOS-Chem run directory corresponding to each entry that you specified in the input file RUNS section. Each run directory will be created as a subdirectory of COPY_PATH that you specified in the input file INPUTS section.

Let's examine the contents of a sample geosfp_4x5_standard run directory. Navigate to your newly created run directory and tssue the following commands:

make fileclean                       # Remove any files left over from previous unit test runs

ls -1                                # Get directory listing

And you will see this directory listing:

brc.dat
dust.dat
FJX_j2j.dat
FJX_spec.dat
getRunInfo
h2so4.dat
HEMCO_Config.rc
HEMCO_restart.201307010000.nc
input.geos
jv_spec_mie.dat
Makefile
org.dat
output/
README
so4.dat
soot.dat
ssa.dat
ssc.dat
v11-01.run
validate

NOTE: Run directories for other simulations may contain other files not pictured here.

The input.geos and HEMCO_Config.rc files have been customized for this particular simulation. They were created from the corresponding template files input.geos.template and HEMCO_Config.template in the Unit Tester. The Perl script getRunInfo is used by the Makefile to extract information about the simulation from input.geos. HEMCO and tracer restart files are also included in every run directory but care must be taken when using them. See the section below for more information about restart files.

Important Notes About Restart Files

Restart files are included in every run directory. The dates corresponding to the restart files are not necessarily the same as your intended simulation start date, or the start date you specified when creating the run directory from the Unit Tester. The actual date for the restart files is shown in the filename of the HEMCO restart file (e.g. HEMCO_restart.200607010000.nc). When creating a run directory for a GEOS-Chem simulation, take note of what this date is and use the information in this section to determine how to properly use the restart files that come with the run directory.

The restart files included in the run directory are for use with spinning up the model to the start date of your production run. We recommend running GEOS-Chem for at least one year to generate new restart files prior to performing your intended simulations. For the case of tagged ozone, we recommend spinning up the model for 10 years.

The start date of your spin-up must correspond to the month of the tracer restart file (initial_trac_rst.{RUNDIR_NAME}), but not necessarily the year, so that seasonality is correctly incorporated. The end date of your spin-up should match the start date of your intended production run so that you end up with restart files for that day.

If the start date of your spin-up does not exactly match that of the HEMCO restart file that comes with the run directory,(HEMCO_restart.{DATE}.nc), then you should not use the HEMCO restart file. You should instead use default emissions values by setting 'HEMCO_RESTART' from 'true' to 'false' in HEMCO_Config.rc. Be sure to remember to re-enable the HEMCO restart file after your spin-up run to use the newly generated HEMCO restart file corresponding to your production run start date.

--Lizzie Lundgren (talk) 16:24, 22 December 2015 (UTC)

Tips and tricks for creating run directories

• In each run directory, we provide GEOS-Chem restart files that you can use to initialize your simulation. However, it is highly recommended that you do not use these restart files for any of your production runs. Rather, you should generate your own restart files by spinning up the model for at least a year. See the Important Notes About Restart Files section above for more information.

• You can create a run directory for the nested grid simulations from a 4° x 5° or 2° x 2.5° run directory by following these steps:
  1. Modify the input.geos file for the nested grid simulation. Complete instructions can be found on this wiki page.
  2. Regrid the restart file(s) using the GAMAP regridding routines. For example, REGRIDH_RESTART can be used to regrid the tracer restart file from one horizontal resolution to another. You can then use CREATE_NESTED to crop the restart file to your nested domain (CH, EU, or NA).

• You can create a MERRA run directory from a GEOS-5 or GEOS-FP run directory, since all three met fields have the same horizontal and vertical resolution. Gabriele Curci found found that the following procedure can be used to create a MERRA run directory from a GEOS-FP run directory (using a geosfp_4x5_UCX run directory as an example).
  1. Create a run directory for GEOS-FP using gcCopyRunDirs
  2. Type cp -R geosfp_4x5_UCX merra_4x5_UCX
  3. Type cd merra_4x5_UCX
  4. Type rename geosfp merra *
  5. Open input.geos in a text editor and change all the occurrences of "geosfp" to "merra"
  6. Type make superclean
  7. Type make distclean in the Code dir
  8. Compile and run

• If you would like to run a simulation and the run directory is not available in the Unit Tester, you can follow these steps:
  1. Modify the input.geos file for your simulation type following the relevant checklist in Chapter 6 of the GEOS-Chem Manual.
  2. Create a "fake" restart file using GAMAP routine MAKE_RESTART.
  3. Alternatively, you can send an email to the relevant Working Group to see if another member can provide you with a run directory or restart file.

• The GEOS-Chem Unit Tester can be used to create run directories for GEOS-Chem v10-01 and later versions. If you wish to download a run directory for an older version of GEOS-Chem, you can follow these instructions.

--Melissa Sulprizio (talk) 18:42, 26 August 2015 (UTC)

Compiling and Running GEOS-Chem

Once you have created one or more GEOS-Chem run directories, you may use them for your GEOS-Chem simulations. First, double-check that the settings in the input.geos and HEMCO_Config.rc files are correct for the simulation that you are trying to run. See the GEOS-Chem Manual for more information about setting simulation options. Also see the Important notes about restart files section above for important information about the appropriate use of restart files and the necessity of spinning up the model.

Next, open the Makefile in a text editor and define the variables CODE_DIR, LOG_DIR, and VERSION specific to your simulation. CODE_DIR is the path to the GEOS-Chem source code you wish to use. LOG_DIR is the path to which you wish to send your log files. VERSION is text you wish to include as a prefix in your output log filename. You can also pass any of these three variables manually to the make command, overwriting the definitions stored in the Makefile.

To start a GEOS-Chem simulation, type the following commands. (We'll use the example of the geosfp_4x5_fullchem simulation, from the previous sections.)

make -j4 TRACEBACK=y mp

The above command turns on traceback for debugging and invokes the mp target in the Makefile. This compiles the source code at path CODE_DIR with OpenMP parallelization and then runs the resultant GEOS-Chem executable using the simulation settings in the configuration files. The suffix .mp is included in each output filename to indicate that you enabled OpenMP parallelization for your GEOS-Chem run.

There are many other targets you may use with make. For example, you can use target sp to compile and run code using a single processor rather than multiple processors. This is useful for generating a baseline from which you can diagnose and debug parallelization errors. Output files will have a .sp rather than .mp suffix to distinguish them from the OpenMP parallelized run output. Alternatively, you can build GEOS-Chem source code without running a simulation (mpbuild or spbuild) or run a simulation without rebuilding (mprun or sprun). There are also several housekeeping targets that clean the run directory (e.g. mpdataclean, mpexeclean, mplogclean, and mpclean) and the source code (e.g. realclean).

See the Makefile for all make options and the GEOS-Chem Makefile Structure wiki page wiki page for more information about compiling GEOS-Chem and using the run directory Makefile.

--Lizzie Lundgren 13:12, 15 April 2015 (EDT)

GEOS-Chem Output Files

Once the GEOS-Chem simulation is complete, you will see several new files in the run directory. Some of these are generated for all simulations while others may be simulation-dependent. For more information, see our GEOS-Chem Output Files wiki page.

--Melissa Sulprizio (talk) 22:25, 5 February 2016 (UTC)