Difference between revisions of "Running GCHP: Configuration"

Revision as of 22:45, 7 August 2018

Previous | Next | Getting Started with GCHP

1. Hardware and Software Requirements
2. Downloading Source Code
3. Obtaining a Run Directory
4. Setting Up the GCHP Environment
5. Compiling
6. Basic Example Run
7. Run Configuration Files
8. Advanced Run Examples
9. Output Data
10. Developing GCHP

Please note that this page is under construction as the run set up process has been greatly simplified between versions v11-02b and v11-02c. Please refer to the Tutorial slides for v11-02c HP to see how to configure a GCHP run using the runConfig.sh bash script in v11-02c. Contact the GEOS-Chem Support Team with questions about advanced run setup.

Overview

All default GCHP run directories are set up to run at c24 resolution with 0.25x0.325 GEOS-FP meteorology, 6 cores, and 1 node. This is the simplest possible run and a good test case for your initial setup. However, you will want to change these settings, and potentially several others, for your research runs. This page goes over how to do this.

Set Number of Nodes and Cores

To change the number of nodes and cores for your run you must update settings in two places: (1) runConfig.sh, and (2) your run script.

Set Cubed Sphere Grid Resolution

Changing your grid resolution involves simply changing the "CS_RES" parameter in runConfig.sh.

Change Input Meteorology Grid Resolution and/or Source

Changing input meteorology requires two updates: (1) redefine the MetDir symbolic link to point to the appropriate source directory, and (2) update all meteorology paths and filenames in ExtData.rc. Currently only GEOS-FP and MERRA2 meteorology are supported in GCHP. Be sure that you have data available at the grid resolution you wish to run at for time period you plan on simulating. See the primary GEOS-Chem wiki page for information on meteorology data available.

Change Your Initial Restart File

All GCHP run directories come with symbolic links to initial restart files for commonly used cubed sphere resolutions. The appropriate restart file is automatically chosen based on the cubed sphere resolution you set in runConfig.sh. All of the restart files are simply GEOS-Chem Classic restart files regridded to the cubed sphere.

Unlike GEOS-Chem Classic, HEMCO restart files are not used in GCHP. HEMCO restart variables may be included in the initial species restart file, or they may be excluded and HEMCO will start with default values. GCHP initial restart files that come with the run directories do not include HEMCO restart variables unless "HEMCO" appears in the filename. This is only the case for the benchmark restart files used for the 1-year benchmark simulation that relies on a valid spin-up.

You may over-write the default restart file with your own by specifying the restart filename in runConfig.sh. Beware that it is your responsibility to make sure it is the proper grid resolution. Publicly available tools for regridding are listed in the GCHP Output Files page of this user manual.

Output Restart Files at Regular Frequency

While most of the GCHP run-time options are set from runConfig.sh, the option for outputting restart files beyond the usual end-of-run restart file is not. This is simply because the default setting of every 30 days is usually adequate. To change this frequency, update the HHmmSS string for "RECORD_FREQUENCY" in file GCHP.rc. Minutes and seconds must each be two digits but hours can be more than two.

Turn On/Off Model Components

You can turn all primary GEOS-Chem components, including type of PBL mixing, from within runConfig.sh. The settings in that file will update input.geos automatically.

Change Model Timesteps

Model timesteps, both chemistry and dynamic, are configured within runConfig.sh. They are set to match GEOS-Chem Classic default values for comparison purposes but can be updated, with caution. Read the documentation in runConfig.sh for setting them to be fully aware of recommended settings and their implications.

Set Simulation Start and End Dates

Set simulation start and end in runConfig.sh. There is also a "DURATION" field in the file which must be set to reflect how long your run will last. If your end date is earlier than your start date plus duration then your GCHP run will fail. If your end date is later than your start date plus duration then your job will not make it to your configured end date; it will end at start date plus duration. If your end date is multiple durations past your start date then subsequent job submissions will start where your last run ended, so long as you do not delete file cap_restart. That file contains a new start string that will always be used if the file is present. You can take advantage of this file for splitting up a long simulation into multiple jobs. See further down on this page for automation of this task built into the run directory.

Typically a "CAP" error indicates a problem with start, end, and duration settings. If you encounter an error with the words "CAP" near it then double-check that these settings make sense.

Turn On/Off Diagnostics

All GCHP have four collections on by default: time-averaged species concentrations, instantaneous species concentrations, time-averaged meteorology, and instantaneous meteorology. All species are enabled while only a subset of meteorology variables are enabled. There are several other collections already implemented but they are off by default for the standard and benchmark simulations, and on by default for the RnPbBe simulation.

To turn collections on or off, comment ("#") collection names in the "COLLECTIONS" list at the top of file HISTORY.rc. Once a collection is turned on, you can comment diagnostics within it further down in the file by searching for the collection name with ".fields" suffix. Be aware that you cannot comment out the diagnostic that appears on the same line as the fields keyword. If you wish to suppress that specific diagnostic then move it to the next line and replace it with a diagnostic that you want to output.

Set Diagnostic Frequency, Duration, and Mode

All diagnostic collections that come with the run directory have frequency, duration, and mode defined within runConfig.sh. With the exception of SpeciesConc_inst and StateMet_inst, all collections are time-averaged (mode) with frequency and duration set to the simulation length you specified in CopyRunDirs.input when creating the run directory. Any of these defaults can be over-written by editing runConfig.sh. Be aware that manual updates of HISTORY.rc will be over-written by runConfig.sh settings.

Add a New Diagnostics Collection

Adding a new diagnostics collection in GCHP is the same as for GEOS-Chem Classic netcdf diagnostics. You must add your collection to the collection list in HISTORY.rc and then define it further down in the file. Any 2D or 3D arrays that are stored within State_Met, State_Chm, or State_Diag, and that are successfully incorporated into the GEOS-Chem Registry may be included as fields in a collection. State_Met variables must be preceded by "met_", State_Chm variables must be preceded by "chm_", and State_Diag variables should not have a prefix. See GeosCore/state_diag_mod.F90 for examples of how existing State_Diag arrays are implemented.

Once implemented, you can either incorporate the new collection settings into runConfig.sh for auto-update, or you can manually configure all settings in HISTORY.rc.

Split a Simulation Into Multiple Jobs

There is an option to split up a single simulation into separate serial jobs. To use this option, do the following:

1. Update runConfig.sh with your full simulation (all runs) start and end dates, and the duration per segment (single run). Also update the number of runs options to reflect to total number of jobs that will be submitted. Carefully read these parts of runConfig.sh to ensure you understand how it works.
2. Use gchp.multirun.run as your run script, or adapt it if your cluster does not use SLURM. As with the regular gchp.run, you will need to update the file with compute resources consistent with runConfig.sh and with your local bashrc. It is located in the runScriptSamples subdirectory of your run directory. Note that you should not submit the run script directly. It will be done automatically by the file described in the next step.
3. Use gchp.multirun.sh to submit your job, or adapt it if your cluster does not use SLURM. It is located in the runScriptSamples subdirectory of your run directory.

There is much documentation in the headers of both gchp.multirun.run and gchp.multirun.sh that is worth reading and getting familiar with. If you have not done so already, it is worth trying out a simple multi-segmented run of short duration to demonstrate that the multi-segmented run configuration and scripts work on your system. For example, you could do a 3 hour simulation with 1 hour duration and number of runs equal to 3.

Besides the regular GCHP log file, which will be appended to for each consecutive run, there will be a "multirun.log" file with high-level information such as the start, end, duration, and job ids for all jobs submitted. Inspect this and your other log files, as well as output in the OutputDir/ directory prior to using the monthly diagnostics option.

Generate Monthly Mean Diagnostics

There is an option to automatically generate monthly diagnostics by submitting month-long simulations as separate jobs. Splitting up the simulation into separate jobs is a requirement for monthly diagnostics because MAPL History requires a fixed number of hours set for diagnostic frequency and file duration. The monthly mean diagnostic option automatically updates HISTORY.rc</rr> diagnostic settings each month to reflect the number of days in that month taking into account leap years.

To use the monthly diagnostics option, first read and follow instructions for splitting a simulation into multiple jobs (see section above). Prior to submitting your run, enable monthly diagnostics in <tt>gchp.multirun.run by searching for variable "Monthly_Diag" and changing its value from 0 to 1. Be sure to read the documentation surrounding the monthly diagnostic option in that file to be sure you understand what you are doing and are meeting all the requirements.

Debug Using Maximum Print Output

Besides compiling with "make compile_debug", there are a few run settings you can configure to booster your chance of successful debugging. All of them involve sending additional print statements to the log files.

1. You can change "ND70" in input.geos from 0 to 1 to turn on extra GEOS-Chem print statements in the main log file.
2. You can set the "DEBUG" variable in runConfig.sh to a number greater than 0 to turn on extra MAPL print statements. The higher the number the more prints will be sent to the log (and the slower your run will be). Usually 20 is sufficient, although you can go higher.
3. You can set the "Verbose" and "Warnings" settings in HEMCO_Config.rc to maximum values of 3 to send the maximum number of prints to HEMCO.log.

None of these options require recompiling. Be aware that all of them will slow down your simulation. Be sure to set them back to the default values after you are finished debugging.

Change Domains Stack Size

For runs at very high resolution or small number of processors you may run into a domains stack size error. This is caused by exceeding the domains stack size memory limit set at run-time and will be apparent in your log file. If this occurs you can increase the domains stack size in file input.nml. The default is set to 20000000.

Turn Off MAPL Timers and Memory Logging

Your GCHP log file will includes timing and memory information by default, and this is usually a good thing. If for some reason you want to turn these features off you can do so in file CAP.rc. Search for "MAPL_ENABLE_TIMERS" and "MAPL_ENABLE_MEMUTILS" and simply change "YES" to "NO".

Feedback Welcome!

If there is something you want to do that is not readily explained on this page then we want to hear from you. Please contact the GCST or direct message Lizzie Lundgren on the GCHP Slack workspace with your feedback.

Previous | Next | GCHP Home