Running GCHP: Configuration

Previous | Next | User Manual Home | GCHP Home

Overview

All default GCHP run directories are set up to run at c24 resolution with 0.25x0.325 GEOS-FP meteorology, 6 cores, and 1 node. This is the simplest possible run and a good test case for your initial setup. However, you will want to change these settings, and potentially several others, for your research runs. This page goes over how to do this.

GCHP has several configuration files, most of which end in suffix ".rc". Rather than update many files, some of which contain redundant information, we instead use utility shell script runConfig.sh to set most options in a single location. Sourcing the file automatically updates other configuration files prior to the run and eliminates the need for remembering what to update and where. However, it is important to note that that doing this will overwrite settings in other configuration files. You therefore should never manually update other configuration files unless you know the specific option is not available for setting in runConfig.sh.

All sample run scripts include sourcing runConfig.sh. When runConfig.sh is sourced it prints out information on what settings are being changed to what value and in what file. Typically this information will be stored in output log runConfig.log or it will be printed to your job scheduler log file.

You generally will not need to know more about the GCHP configuration files beyond what is listed on this page. However, for more detailed information about the configuration files used by GCHP see the last section of this user manual which includes a list and description of all contents as well as a more detailed display of what runConfig.sh is actually doing.

If there is something you want to configure in your GCHP run that is not described on this page please contact the GEOS-Chem Support Team with feedback.

Pre-run Checklist

Prior to running GCHP, run through the following checklist to ensure everything is set up properly.

Your run directory contains the executable geos.
All symbolic links are present in your run directory and point to a valid path. These include TileFiles, MetDir, MainDataDir, ChemDataDir, CodeDir, and an initial restart files.
The input meteorology resolution and source are as you intend (inspect with "grep MetDir ExtData.rc" and "file MetDir"). For versions 12.1.0 and later, create a new run directory to use a different Met source.
You have looked through and set all configurable settings in runConfig.sh.
You have a run script
The resource allocation in runConfig.sh and your run script are consistent (# nodes and cores).
The run script sources your environment file that you used for compiling GCHP (gchp.env for version 12.1.0 and later).
If reusing a run directory, you have archived your last run or discarded it with 'make cleanup_output' (see section at end of this page on reusing a run directory)

Run Configuration Options

Compute Configuration

Set Number of Nodes and Cores

To change the number of nodes and cores for your run you must update settings in two places: (1) runConfig.sh, and (2) your run script. The runConfig.sh file contains detailed instructions on how to set resource parameter options as show below.

The sample SLURM run script will assign GCHP run resources based on settings in runConfig.sh. However, you must request the same number of nodes in your run script as in runConfig.sh. You may request additional cores and full memory per node to maximize available memory per core. For example, the below settings request 32 cores per node (entire nodes) and all memory per node. However, further down in the script only 16 cores per node are allocated for the GCHP run, consistent with the settings in the example of runConfig.sh above.

It is important to be smart about your resource allocation. To do this it is useful to understand how GCHP works with respect to distribution of nodes and cores across the grid. At least one unique core is assigned to each face on the cubed sphere, resulting in a constraint of at least six cores to run GCHP. The same number of cores must be assigned to each face, resulting in another constraint of total number of cores being a multiple of six. Communication between the cores occurs only during transport processes.

While any number of cores is valid as long as it is a multiple of six, you will typically start to see negative effects due to excessive communication if a core is handling less than around one hundred grid cells or a cluster of grid cells that are not approximately square. You can determine how many grid cells are handled per core by analyzing your grid resolution and resource allocation. For example, if running at C24 with six cores each face is handled by one core (6 faces / 6 cores) and contains 576 cells (24x24). Each core therefore processes 576 cells. Since each core handles one face, each core communicates with four other cores (four surrounding faces).

You can configure approximately how the cores are assigned to grid cell geometry by using the NX and NY configuration variables in GCHP as shown above. But what is this actually doing? Imagine lining up the six face grids adjacent to each other to get a single rectangular array. The rectangle will have N grid cells width (e.g. 24 if a C24 grid), and 6N grid cells height (since 6 faces). NX is the number of segments the width N is broken into for core distribution. NY is the number of segments the height 6N is broken into and must always be a multiple of six. NX * NY is always the total number of cores.

For the case of a six core run, NX is equal to 1 and NY is equal to 6. This is because the entire N grid cells width is handled by 1 core (NX) and the 6N grid cells height is handled by 6 cores (NY), or one per face. If you instead wanted each face to be handled by four cores, and further constrain each core to handle one face quadrant, you would set NX equal to two and NY equal to twelve. A simple way of thinking about this is that core distribution across each face is with geometry NX x NY/6. In this last example that would be equivalent to 2x2.

Split a Simulation Into Multiple Jobs

There is an option to split up a single simulation into separate serial jobs. To use this option, do the following:

Update runConfig.sh with your full simulation (all runs) start and end dates, and the duration per segment (single run). Also update the number of runs options to reflect to total number of jobs that will be submitted. Carefully read these parts of runConfig.sh to ensure you understand how it works.
Use gchp.multirun.run as your run script, or adapt it if your cluster does not use SLURM. It is located in the runScriptSamples subdirectory of your run directory. As with the regular gchp.run, you will need to update the file with compute resources consistent with runConfig.sh. Note that you should not submit the run script directly. It will be done automatically by the file described in the next step.
Use gchp.multirun.sh to submit your job, or adapt it if your cluster does not use SLURM. It is located in the runScriptSamples subdirectory of your run directory.

There is much documentation in the headers of both gchp.multirun.run and gchp.multirun.sh that is worth reading and getting familiar with. If you have not done so already, it is worth trying out a simple multi-segmented run of short duration to demonstrate that the multi-segmented run configuration and scripts work on your system. For example, you could do a 3 hour simulation with 1 hour duration and number of runs equal to 3.

Besides the regular GCHP log file, which will be appended to for each consecutive run, there will be a "multirun.log" file with high-level information such as the start, end, duration, and job ids for all jobs submitted. Inspect this and your other log files, as well as output in the OutputDir/ directory prior to using for longer duration runs.

Change Domains Stack Size

For runs at very high resolution or small number of processors you may run into a domains stack size error. This is caused by exceeding the domains stack size memory limit set at run-time and the error will be apparent from the message in your log file. If this occurs you can increase the domains stack size in file input.nml. The default is set to 20000000.

Basic Run Settings

Set Cubed Sphere Grid Resolution

GCHP uses a cubed sphere grid rather than the traditional lat-lon grid used in GEOS-Chem Classic. While regular lat-lon grids are typically designated as ΔLat ⨉ ΔLon (e.g. 4⨉5), cubed sphere grids are designated by the side-length of the cube. In GCHP we specify this as CX (e.g. C24 or C180). The simple rule of thumb for determining the roughly equivalent lat/lon for a given cubed sphere resolution is to divide the side length by 90. Using this rule you can quickly match C24 with 4x5, C90 with 1 degree, C360 with quarter degree, and so on.

To change your grid resolution in the run directory edit the "CS_RES" integer parameter in runConfig.sh to the cube side-length you wish to use.

Turn On/Off Model Components

You can toggle all primary GEOS-Chem components, including type of mixing, from within runConfig.sh. The settings in that file will update input.geos automatically.

Change Model Timesteps

Model timesteps, both chemistry and dynamic, are configured within runConfig.sh. They are set to match GEOS-Chem Classic default values for comparison purposes but can be updated, with caution. Read the documentation in runConfig.sh for setting them to be fully aware of recommended settings and their implications.

Set Simulation Start and End Dates

Set simulation start and end in runConfig.sh.

There is also a "Duration" field in the file which must be set to reflect how long your run will last. If your end date is earlier than your start date plus duration then your GCHP run will fail. If your end date is later than your start date plus duration then your job will not make it to your configured end date; it will end at start date plus duration. If your end date is multiple durations past your start date then subsequent job submissions will start where your last run ended, so long as you do not delete file cap_restart. That file contains a new start string that will always be used if the file is present. You can take advantage of this file for splitting up a long simulation into multiple jobs. See further down on this page for automation of this task built into the run directory.

Typically a "CAP" error indicates a problem with start, end, and duration settings. If you encounter an error with the words "CAP" near it then double-check that these settings make sense.

Inputs

Change Input Meteorology Grid Resolution and/or Source

For versions 12.1.0 and later you can specify meteorology source when creating a run directory. The grid resolutions will automatically be set to 0.25x0.3125 for GEOS-FP and 0.5x0.625 for MERRA2. If you wish to change meteorology source using these versions then simply create a new run directory. To change the grid resolution of the meteorology, update all meteorology paths and filenames in ExtData.rc.

For versions prior to 12.1.0 the GCHP run directories are set by default to use 0.25x0.3125 GEOS-FP meteorology. To change to MERRA2, redefine the MetDir symbolic link to point to the MERRA2 directory. To change the grid resolution of input meteorology, update all meteorology paths and filenames in ExtData.rc.

When changing meteorology source and/or grid resolution, be sure that you have the data available for the time period you plan on simulating. In addition, note that meteorology listed in ExtData.rc includes both data for the time period you plan on running at as well as constants files (2011 to GEOS-FP and 2015 for MERRA2). See the downloading GEOS-Chem data page for more information on meteorology sources available and how to download them.

Change Your Initial Restart File

All GCHP run directories come with symbolic links to initial restart files for commonly used cubed sphere resolutions. The appropriate restart file is automatically chosen based on the cubed sphere resolution you set in runConfig.sh. All of the restart files are simply GEOS-Chem Classic restart files regridded to the cubed sphere.

You may over-write the default restart file with your own by specifying the restart filename in runConfig.sh. Beware that it is your responsibility to make sure it is the proper grid resolution. Publicly available tools for regridding are listed in the GCHP Output Files page of this user manual.

Unlike GEOS-Chem Classic, HEMCO restart files are not used in GCHP. HEMCO restart variables may be included in the initial species restart file, or they may be excluded and HEMCO will start with default values. GCHP initial restart files that come with the run directories do not include HEMCO restart variables unless "HEMCO" appears in the filename. This is only the case for the benchmark restart files used for the 1-year benchmark simulation that relies on a valid spin-up.

Turn On/Off Emissions Inventories

Because file I/O impacts GCHP performance it is a good idea to turn of file read of emissions that you do not need. You can turn emissions inventories on or off the same way you would in GEOS-Chem Classic, by setting the inventories to true or false at the top of configuration file HEMCO_Config.rc. All emissions that are turned off in this way will be ignored when GCHP uses ExtData.rc to read files, thereby speeding up the model.

For emissions that do not have an on/off toggle at the top of the file, you can prevent GCHP from reading them by commenting them out in HEMCO_Config.rc. No updates to ExtData.rc would be necessary. If you alternatively comment out the emissions in ExtData.rc but not HEMCO_Config.rc then GCHP will fail with an error when looking for the file information.

Another option to skip file read for certain files is to replace the file path in ExtData.rc with /dev/null. However, if you want to turn these inputs back on at a later time you should preserve the original path in a comment.

Add New Input Files

There are three main requirements for adding new emissions inventories to GCHP:

Add the inventory information to HEMCO_Config.rc. If you wish to add new inputs to the model that are not handled by HEMCO then you can skip this step.
Add the inventory information to ExtData.rc.
Have a tile file available that maps the inventory's lat/lon grid to the cubed sphere grid for the resolution you will use.

To add information to HEMCO_Config.rc, follow the same rules as you would for adding a new emission inventory to GEOS-Chem Classic. Note that not all information in HEMCO_Config.rc is used by GCHP. This is because HEMCO is only used by GCHP to handle emissions after they are read, e.g. scaling and applying hierarchy. All functions related to HEMCO file read are skipped. This means that you could put garbage for the file path and units in HEMCO_Config.rc without running into problems with GCHP. However, we recommend that you fill in HEMCO_Config.rc in the same way you would for GEOS-Chem Classic for consistency and also to avoid potential format check errors.

Staying consistent with the information that you put into HEMCO_Config.rc, add the inventory information to ExtData.rc following the guidelines listed at the top of the file and using existing inventories as examples. You can ignore all entries in HEMCO_Config.rc that are copies of another entry since putting these in ExtData.rc would result in reading the same variable in the same file twice. Doing so would be costly in GCHP because each file is opened and closed for each variable in the file. HEMCO interprets the copied variables, denoted by having dashes in the HEMCO_Config.rc entry, separate from file read.

At this point it is best to run a very short simulation with GCHP with MAPL debug prints on (see section on debugging below). If your file(s) need a new tile file then the model will crash. Tile files have already been created for many lat/lon grids and these are stored in ExtData/GCHP/TileFiles. The GCHP log file error will include the tile file name that GCHP expects to be available for regridding your new inventory. In that filename DC = dateline centered, PC = pole centered, DE = dateline edge, and PE = pole edge. UU is reserved for files on regional grids. Once you have this information you should be able to generate your own tile file by downloading the tempestremap and CSGrid repositories from GitHub and following these steps:

tempestremap: This tool will generate a netcdf tile file for mapping lat/lon coordinates to cubed sphere. This is a fortran tool that should work in your existing GCHP environment. Simply do make clean and then make to build the tempestrehap code. Then use runGlobal.sh or runRegional.template.sh to generate global or regional bound tile files. When using runGlobal.sh, you will need to specify whether your data is dateline-centered and/or pole-centered, as determined from the log file error message. We recommend generating a tile file for all supported cubed-sphere resolutions (nC = 24, 48, 90, 180, and 360).
CSGrid: This tool will convert the netCDF file created by tempestremap to binary for compatibility with GCHP. CSGrid requires a Matlab license. If you have Matlab you can use exampleScripts/create_Tempest_TileFile_LL2CS.m to convert your netCDF output in tempestremap/TileFiles from netcdf to binary. Send the resulting file to the GCST and they can add it to ExtData/GCHP/TileFiles.

Once read in by GCHP, your data will be stored as MAPL Import variables with the same names that appear in the first column of ExtData.rc. Beware that the MAPL I/O component in GCHP does not read or write integers. If your data contains integers then you should reprocess the file to contain floating point values instead. If you try to input integers you will get an error such as this:

>> Reading  TESTDATA from ./MainDataDir/testfile.nc
CFIO: Reading ./MainDataDir/testfile.nc at 19850101 000000
 CFIO_GetVar: error getting scale
 CFIO_CFIO_GetVar failed
 problem in ESMF_CFIOSdfVarRead

If your input files are handled by HEMCO then you do not need to do anything else to handle the MAPL Imports. However, if your new inputs are not handled by HEMCO then you will need to take the additional steps of adding source code to transfer your MAPL Imports to something that GEOS-Chem can understand. If you wish to assign a MAPL Import directly to a State_Met or other state field in GEOS-Chem, you can do this in GCHP file "Includes_Before_Run.H". The lines in that file are executed prior to every dynamic tilmestep in GCHP and currently contain the setting of all State_Met fields derived from MAPL Imports. For more advanced use cases, read through GCHP file Chem_GridCompMod.F90 for examples, specifically searching for calls to subroutine MAPL_GetPointer. Contact the GEOS-Chem Support Team for more information on how to use MAPL Imports within GEOS-Chem.

Outputs

Output Restart Files at Regular Frequency

The MAPL component in GCHP has the option to output restart files (also called checkpoint files) at regular intervals. Unlike the final restart file output at the end of a simulation, these regularly output restart files contain the date and time in their filename. Enabling this feature is a good idea if you plan on doing a long simulation and you are not splitting your run into multiple jobs. If the run crashes unexpectedly then you can restart mid-run rather than start over from the beginning. To set the checkpoint frequency, simply update the HHmmSS string for "Checkpoint_Freq" in runConfig.rc. Minutes and seconds must each be two digits but hours can be more than two.

Turn On/Off Diagnostics

All GCHP run directories have four collections on by default: time-averaged species concentrations, instantaneous species concentrations, time-averaged meteorology, and instantaneous meteorology. All species are enabled while only a subset of meteorology variables are enabled. There are several other collections already implemented but they are off by default for the standard and benchmark simulations, and on by default for the RnPbBe simulation.

To turn collections on or off, comment ("#") collection names in the "COLLECTIONS" list at the top of file HISTORY.rc.

Once a collection is turned on, you can comment diagnostics within it further down in the file by searching for the collection name with ".fields" suffix. Be aware that you cannot comment out the diagnostic that appears on the same line as the fields keyword. If you wish to suppress that specific diagnostic then move it to the next line and replace it with a diagnostic that you want to output.

Set Diagnostic Frequency, Duration, and Mode

All diagnostic collections that come with the run directory have frequency, duration, and mode defined within runConfig.sh. With the exception of SpeciesConc_inst and StateMet_inst, all collections are time-averaged (mode) with frequency and duration set to the simulation length you specified in CopyRunDirs.input when creating the run directory. Any of these defaults can be over-written by editing runConfig.sh. Be aware that manual updates of HISTORY.rc will be over-written by runConfig.sh settings.

Add a New Diagnostics Collection

Adding a new diagnostics collection in GCHP is the same as for GEOS-Chem Classic netcdf diagnostics. You must add your collection to the collection list in HISTORY.rc and then define it further down in the file. Any 2D or 3D arrays that are stored within State_Met, State_Chm, or State_Diag, and that are successfully incorporated into the GEOS-Chem Registry may be included as fields in a collection. State_Met variables must be preceded by "met_", State_Chm variables must be preceded by "chm_", and State_Diag variables should not have a prefix. See GeosCore/state_diag_mod.F90 for examples of how existing State_Diag arrays are implemented.

Once implemented, you can either incorporate the new collection settings into runConfig.sh for auto-update, or you can manually configure all settings in HISTORY.rc.

Generate Monthly Mean Diagnostics

There is an option to automatically generate monthly diagnostics by submitting month-long simulations as separate jobs. Splitting up the simulation into separate jobs is a requirement for monthly diagnostics because MAPL History requires a fixed number of hours set for diagnostic frequency and file duration. The monthly mean diagnostic option automatically updates HISTORY.rc diagnostic settings each month to reflect the number of days in that month taking into account leap years.

To use the monthly diagnostics option, first read and follow instructions for splitting a simulation into multiple jobs (see separate section on this page). Prior to submitting your run, enable monthly diagnostics in gchp.multirun.run by searching for variable "Monthly_Diag" and changing its value from 0 to 1. Be sure to read the documentation surrounding the monthly diagnostic option in that file to be sure you understand what you are doing and are meeting all the requirements.

Debugging

Enable Maximum Print Output

Besides compiling with "make compile_debug", there are a few run settings you can configure to boost your chance of successful debugging. All of them involve sending additional print statements to the log files.

Change "ND70" in input.geos from 0 to 1 to turn on extra GEOS-Chem print statements in the main log file.
Set the "DEBUG" variable in runConfig.sh to a number greater than 0 to turn on extra MAPL print statements. The higher the number the more prints will be sent to the log (and the slower your run will be). Usually 20 is sufficient, although you can go higher.
Set the "Verbose" and "Warnings" settings in HEMCO_Config.rc to maximum values of 3 to send the maximum number of prints to HEMCO.log.

None of these options require recompiling. Be aware that all of them will slow down your simulation. Be sure to set them back to the default values after you are finished debugging.

Turn On/Off MAPL Timers and Memory Logging

Your GCHP log file will include timing and memory information by default, and this is usually a good thing. If for some reason you want to turn these features off you can do so in file CAP.rc. Search for "MAPL_ENABLE_TIMERS" and "MAPL_ENABLE_MEMUTILS" and simply change "YES" to "NO". Remember to turn them back on again if you later need to to debug.

Reusing a Run Directory

Archiving a Run

One of the benefits of GCHP relative to GEOS-Chem Classic is that you can reuse a run directory for different grid resolutions and meteorology sources without recompiling. This comes with the perils of losing your old work. To mitigate this issue there is utility shell script archiveRun.sh to archive output and configuration files from your last run within a subdirectory in the run directory. All you need to do is pass a non-existent subdirectory name of your choosing where the files should be stored. Here is an example:

./archiveRun.sh c48_test

The following output is then printed to screen to show you exactly what is being archived and where:

Archiving files...
-> c48_test/build/lastbuild
-> c48_test/build/compile.log
-> c48_test/config/input.geos
-> c48_test/config/CAP.rc
-> c48_test/config/ExtData.rc
-> c48_test/config/fvcore_layout.rc
-> c48_test/config/GCHP.rc
-> c48_test/config/HEMCO_Config.rc
-> c48_test/config/HEMCO_Diagn.rc
-> c48_test/config/HISTORY.rc
-> c48_test/restarts/gcchem_internal_checkpoint_c24.nc
-> c48_test/logs/compile.log
-> c48_test/logs/gchp.log
-> c48_test/logs/HEMCO.log
-> c48_test/logs/PET00000.GEOSCHEMchem.log
-> c48_test/logs/runConfig.log
-> c48_test/logs/slurm-50168021.out
-> c48_test/run/runConfig.sh
-> c48_test/run/gchp.run
-> c48_test/run/gchp.ifort17_openmpi_odyssey.bashrc
Warning: *.multirun.sh not found
Complete!

There is a file structure within the archive directory automatically set up to store files of various typed (e.g. logs files in the logs subdirectory). This particular archived run was a single segment run (single job) which is why there is a warning about a multirun file being missing. This can be ignored.

Cleaning the Run Directory

If you do not want to save your last run you can discard its remnants by doing "make cleanup_output". All sample bashrc files have an alias for this, "mco", to make it easier to do. Here is an example of output printed when cleaning the run directory:

rm -f /n/home08/elundgren/GC/testruns/12.0.0/Aug01/gchp_RnPbBe/OutputDir/*.nc4
rm -f trac_avg.*
rm -f tracerinfo.dat
rm -f diaginfo.dat
rm -f cap_restart
rm -f gcchem*
rm -f *.rcx
rm -f *~
rm -f gchp.log
rm -f HEMCO.log
rm -f PET*.log
rm -f runConfig*log
rm -f multirun.log
rm -f logfile.000000.out
rm -f slurm-*
rm -f 1
rm -f EGRESS

Rerunning Without Cleaning

You can reuse a run directory without cleaning it and without archiving your last run. Files will generally simply be replaced by files generated in the next run. This will work okay with one exception. The output cap_restart file must be removed prior to subsequent runs if you are starting a run from scratch. The cap_restart file contains a date and time string for the end of your last run. GCHP will attempt to start your next run at this date and time if the file is present. This is useful for splitting up a run into multiple jobs but is generally not desirable. Therefore you should always delete cap_restart before a new run. This is included in all sample run scripts except the multi-run run script which has special handling of the file.