GEOS-Chem Newsletter Summer-Fall 2013: Difference between revisions
| Line 152: | Line 152: | ||
== GEOS-Chem Unit Tester == | == GEOS-Chem Unit Tester == | ||
We (the GEOS-Chem Support Team) often receive code submissions that have not been adequately debugged. In some cases, several months may elapse before an error is discovered. Then we have to send out a bug fix patch, which is a inconvenience. Furthermore, some code submissions that we receive have only ever been tested on one type of met field (e.g. GEOS-5 but not MERRA or GCAP). Hidden errors may later manifest themselves when someone tries to run G-C with a different set of met fields. | We (the GEOS-Chem Support Team) often receive code submissions that have not been adequately debugged. In some cases, several months may elapse before an error is discovered. Then we have to send out a bug fix patch, which is a inconvenience. Furthermore, some code submissions that we receive have only ever been tested on one type of met field (e.g. [[GEOS-5]] but not [[MERRA]] or [[GCAP]]). Hidden errors may later manifest themselves when someone tries to run G-C with a different set of met fields. | ||
One of the "wish list" items that came out of the IGC6 software engineering discussions was to have the ability to compile G-C with a standard set of debugging flags in order to find any numerical issues (parallelization problems, out-of-bounds errors, div-by-zero errors, etc.). Having such a package would help us to find and fix these common errors before we run the benchmark simulations. | One of the "wish list" items that came out of the IGC6 software engineering discussions was to have the ability to compile G-C with a standard set of debugging flags in order to find any [[Floating point math issues|numerical issues]] (parallelization problems, out-of-bounds errors, div-by-zero errors, etc.). Having such a package would help us to find and fix these common errors before we run the benchmark simulations. | ||
We are happy to report that we now have such a "Unit Tester" package in place. The [[Debugging with the GEOS-Chem unit tester|GEOS-Chem Unit Tester]] is an external package of scripts, Makefiles, and run directories that can be downloaded via Git. It will compile and run short GEOS-Chem simulations (typically 1 or 2 model hours) for various combinations of met fields, horizontal resolutions, and simulation types. The Unit Tester uses a very strict set of compiler debugging flags designed to trap several common error conditions (out-of-bounds, div-by-zero, Infinity, NaN, etc.). For each combination of met field/horizontal grid/simulation, the G-C Unit Tester will perform two simulations: one with OpenMP parallelization turned on, and a second with OpenMP turned off. Ideally, the output of both simulations should be identical; if not, an error in the parallelization exists. | We are happy to report that we now have such a "Unit Tester" package in place. The [[Debugging with the GEOS-Chem unit tester|GEOS-Chem Unit Tester]] is an external package of scripts, Makefiles, and run directories that can be downloaded via Git. It will compile and run short GEOS-Chem simulations (typically 1 or 2 model hours) for various combinations of met fields, horizontal resolutions, and simulation types. The Unit Tester uses a very strict set of compiler debugging flags designed to trap several common error conditions (out-of-bounds, div-by-zero, Infinity, NaN, etc.). For each combination of met field/horizontal grid/simulation, the G-C Unit Tester will perform two simulations: one with OpenMP parallelization turned on, and a second with OpenMP turned off. Ideally, the output of both simulations should be identical; if not, an error in the parallelization exists. | ||
Revision as of 21:15, 18 September 2013
Dear GEOS-Chem users,
We are happy to bring you the Summer-Fall edition of the GEOS-Chem Newsletter! Please read below to find out the latest goings-on in the GEOS-Chem community.
Sincerely,
The GEOS-Chem Support Team
geos-chem-support@as.harvard.edu
GEOS-Chem Steering Committee
The GEOS-Chem Steering Committee (GCSC) met via telecon on 17 September 2013. We invite you to read the meeting minutes (PDF format).
The GCSC will meet again in late November or early December, date TBD.
Ongoing GEOS-Chem development
GEOS-Chem v9-02 is our development version. It contains several important scientific updates over the previous version, GEOS-Chem v9-01-03. We have also restructured the source code significantly so that GEOS-Chem v9-02 can run within the NASA GEOS-5 GCM.
Features already added to GEOS-Chem v9-02
We have completed the following benchmarks since the 6th International GEOS-Chem Meeting. Please see the GEOS-Chem v9-02 wiki page for more information about individual features.
| Version | Date | Features |
|---|---|---|
| v9-02h | 15 May 2013 |
|
| v9-02i | 17 May 2013 |
|
| 28 May 2013 |
| |
| v9-02k | 07 Jun 2013 |
|
| v9-02l | 26 Jun 2013 |
|
| v9-02m | 30 Jul 2013 |
|
| v9-02n | 12 Aug 2013 |
|
| v9-02o | 03 Sep 2013 |
|
| v9-02p | 13 Sep 2013 |
|
| v9-02q | 17 Sep 2013 |
|
Still to be added to GEOS-Chem v9-02
The following features are slated to be added into v9-02:
| Version | Features |
|---|---|
| v9-02r |
|
| v9-02s |
|
| v9-02t |
|
Progress
The progress on v9-02 has been slower than anticipated. We attribute this to the following:
- Several of the science updates (esp. the chemistry mechanism updates) also required 1-year benchmarks. This took additional time to set up.
- We kept finding and fixing bugs and/or numerical issues as we were adding the science updates. On more than one occasion, we received incorrect code and/or data. In other instances, we found that code updates submitted to us by users were causing numerical problems (i.e. div-by-zero, out-of-bounds errors) under certain conditions.
- Some of the updates (e.g. Havala's SOA code) came from a very old version (v8-02-01) that predated Git. Therefore, we had to do the code-merging manually, which was much more time-consuming.
- While implementing these scientific updates, we were making in parallel widespread structural changes for our Grid-Independent GEOS-Chem project, in order to allow GEOS-Chem to be compatible with the GEOS-5 GCM. We tested the code rigorously in order to ensure that these structural modifications did not change any scientific results. At frequent intervals, we have to synchronize code updates from the GIGC development stream back into the standard code, so that the codes will not diverge.
GEOS-Chem Unit Tester
We (the GEOS-Chem Support Team) often receive code submissions that have not been adequately debugged. In some cases, several months may elapse before an error is discovered. Then we have to send out a bug fix patch, which is a inconvenience. Furthermore, some code submissions that we receive have only ever been tested on one type of met field (e.g. GEOS-5 but not MERRA or GCAP). Hidden errors may later manifest themselves when someone tries to run G-C with a different set of met fields.
One of the "wish list" items that came out of the IGC6 software engineering discussions was to have the ability to compile G-C with a standard set of debugging flags in order to find any numerical issues (parallelization problems, out-of-bounds errors, div-by-zero errors, etc.). Having such a package would help us to find and fix these common errors before we run the benchmark simulations.
We are happy to report that we now have such a "Unit Tester" package in place. The GEOS-Chem Unit Tester is an external package of scripts, Makefiles, and run directories that can be downloaded via Git. It will compile and run short GEOS-Chem simulations (typically 1 or 2 model hours) for various combinations of met fields, horizontal resolutions, and simulation types. The Unit Tester uses a very strict set of compiler debugging flags designed to trap several common error conditions (out-of-bounds, div-by-zero, Infinity, NaN, etc.). For each combination of met field/horizontal grid/simulation, the G-C Unit Tester will perform two simulations: one with OpenMP parallelization turned on, and a second with OpenMP turned off. Ideally, the output of both simulations should be identical; if not, an error in the parallelization exists.
The G-C Unit Tester is compatible with versions GEOS-Chem v9-02o and higher versions. In v9-02o, we have now removed the Headers/define.h file, in which several C-preprocessor switches were defined. All C-preprocessor switches are now set with Makefile options from the command line. For example, to compile GEOS-Chem for the 4x5 grid with GEOS-5 meteorology, you would type:
make -j4 MET=geos5 GRID=4x5 ...
or if you wanted to compile for the 0.5 x 0.667 NA nested grid, you would also add an additional flag:
make -j4 MET=geos5 GRID=05x0666 NEST=na ...
Setting all of the C-preprocessor switches with command-line Makefile options allows the G-C Unit Tester to repeatedly compile G-C from the main script. This would not have been possible before.
Using the GEOS-Chem Unit Tester, we have already identified and corrected several numerical issues in v9-02p, as shown on our Numerical issues discovered in GEOS-Chem wiki page.
For more information, please see our Debugging with the GEOS-Chem unit tester wiki page.
GEOS-FP met fields
GEOS-FP ("forward processing") is the latest GEOS-5 met data product from NASA/GMAO. It was originally introduced in 2011 as GEOS-5.7.2, but this nomenclature has now become obsolete. GEOS-FP—which is now produced with version 5.11.0 of the GEOS-DAS (Data Assimilation System)— has finer native horizontal resolution (0.25° lat x 0.3125° lon) and temporal resolution (hourly data and 3-hourly data) than older GMAO met products such as GEOS-5.2.0, the MERRA reanalysis, and GEOS-4.
For more information, please see the following GEOS-FP resources:
- GEOS-FP (v1.0) file specification document (11 Jun 2013)
- List of GEOS-FP met fields for GEOS-Chem
- Information about implementing GEOS-FP into GEOS-Chem
Current status
At present, the G-C v9-02 development version "unofficially" supports the GMAO GEOS-FP met fields. The final release of v9-02 will "officially" support the GEOS-FP met product.
For the NASA SEAC4RS mission (August thru mid-September 2013), we are currently running GEOS-Chem in near-real-time mode using the 0.25° x 0.3125° North America nested grid. This code is a research product, and some issues will still need to be corrected before we can bring this capability into the standard GEOS-Chem code. We will also have to spin up some stratospheric quantities so that we can do a 1-yr benchmark full-chemistry simulation with GEOS-FP met.
Karen Yu has completed a 1-year Rn-Pb-Be benchmark comparing GEOS-5 and GEOS-FP met data. The results are being posted on our Rn-Pb-Be simulation wiki page.
Editing the existing GEOS-FP data archive
The GEOS-Chem Support Team also recommends that we make adjustments to our existing GEOS-FP data archive prior to the release of v9-02:
When the GEOS-FP data first debuted, NASA/GMAO called this product "GEOS-5.7.2", which reflected the version of the GEOS-DAS system used to create the met product. Since then, the nomenclature has changed to "GEOS5-FP" (what we call GEOS-FP), where "FP" stands for "forward processing". The names of the GEOS-FP processed data files that we read into GEOS-Chem still carry the "GEOS-5.7.2" designation. I recommend that we rename the files so that they carry the GEOS-FP designation before the official release of v9-02. This will avoid confusion in the future.
We have since learned that some of the netCDF attributes in our processed GEOS-FP data files do not conform to accepted community standards (e.g COARDS, CF). For example, the unit string "unitless" should be replaced with "1", the string "conventions" should be replaced with "Conventions", etc. I propose that we adjust the internal file attributes accordingly before the release of v9-02. This can be done by running the netCDF operators (NCO) on the existing data files. We do NOT have to reprocess the files from scratch.
We have also since learned that the ESMF/MAPL libraries (which the Grid-Independent GEOS-Chem code relies on) cannot read data files containing fields with different numbers of vertical levels. All of the data fields in a single file must be of the same vertical dimension. In our current GEOS-FP data files that we have already processed, one file type contains some fields that are stored on both level edges (73 vertical points) and other fields that are stored on level centers (72 vertical points). We would therefore have to split off the field that is stored on level edges into a separate file. Again, this can be done in a straightforward manner by applying the netCDF operators (NCO) to the existing files. We do not have to recreate the files from the raw data.
Grid-Independent GEOS-Chem Updates
Much progress has been made on the Grid-Independent GEOS-Chem project this summer. In particular:
Integration into the GEOS Data Assimilation System (GEOS-DAS)
Mike Long & Christoph Keller visited NASA/GMAO in August. They have implemented the Grid-Independent GEOS-Chem (GIGC) Emissions Component into the GEOS-5 GCM. For initial testing purposes, the Emissions Component has been implemented as part of the existing [Grid-Independent GEOS-Chem: Chemistry Component|Chemistry Component]], rather than as a separate component. The Emissions Component will be later separated into its own component within the GCM.
The Emissions Component successfully reads external emissions data (in netCDF format) from various inventories and processes it into resultant emissions per species. At present, the Emissions are not being fed into the Chemistry, but this will be the next step.
At some point we should devote some effort into rewriting the existing GEOS-Chem dry deposition routines. The current dry deposition code relies on a fixed Olson vegetation map. This map is difficult to regrid to arbitrary resolution within the GEOS-5 GCM, as you have to account for the # of land types and the fraction of each land type in the coarse grid box. Ideally, we would like to remove dependence on the Olson land map from the GEOS-Chem dry deposition code so that we can better interface it with other land maps or dynamic vegetation models.
Grid-Independent GEOS-Chem: HPC standalone version
In addition to integrating the Grid-Independent GEOS-Chem (GIGC) into the GEOS-5/GCM, we are also working towards creating a standalone version of the GIGC that will use ESMF/MAPL libraries to run with message-passing (MPI) parallelization. This will allow G-C to run on hundreds or thousands of CPUs on high-performance computing (HPC) systems. We are collaborating with Kevin Bowman and his team at JPL on this effort.
The initial challenge has been to create the directory structure and build system for the standalone GIGC. The GEOS-5 GCM has a complicated build system. It relies on many hardwired makefiles and scripts, which are difficult to use, let alone modify. We do not want to replicate the GEOS-5 build system in its entirety, but rather would like to take essential components and merge them into the existing GEOS-Chem's build system. The work is ongoing.
Flexchem
We are still waiting on Mat Evans to implement the rate law library in KPP.
= TAU and KPPA
The GEOS-Chem Support Team recently met with John Linford from ParaTools, Inc. (John is a former student of Adrian Sandu at VT). John gave us a demonstration of two software items: (1) TAU and (2) KPPA.
TAU is a performance profiler tool. It can examine a software product like GEOS-Chem and provide reports of which routines are taking the most time to execute. We feel that this can be an invaulable tool in helping us to identify and eliminate bottlenecks from the GEOS-Chem code.
KPPA (the Kinetic pre-processor, accelerated) is an updated version of KPP that was totally rewritten from scratch. It is compatible with the existing KPP input files, and can be quickly compiled. It can generate chemical solver code that is compatible with OpenMP parallelization or for GPU (graphics processor) chips. Furthermore, KPPA has some very useful diagnostics that can let you see the online evolution of species and the model error during a run.
John has expressed willingness to work with the GEOS-Chem community in the future. We are going to explore ways these tools may be used with GEOS-Chem.
NCL (NCAR command language)
Several of us at Harvard had the opportunity to participate in a tutorial course on the NCAR command language. NCL is an extremely useful file I/O and graphics suite that is geared towards working with netCDF and similar file formats.
With NCL, it is almost trivial to read and process data from a netCDF, HDF, or GRIB file. To do this in other languages like Fortran, IDL, or Python involves several more steps. Furthermore, NCL has some useful regridding utilities:
- Bilinear interpolation -- used for fields w/o an area dependence (e.g. T, PS, etc).
- Area conserving regridding – used for fluxes (kg/m2/s) or precipitation quantities
- ESMF regridding – used for regridding from nonlinear or other complex grids
As part of the Grid-Independent GEOS-Chem development, we will be migrating away from binary file format and towards netCDF file format. NCL will allow us to read and regrid files (such as restart files, emisisons data, etc) with ease. As time allows, we will create a repository of NCL scripts that can be shared with the G-C user community.
NOTES:
- NCL will not totally replace other plotting packages like Matlab or GAMAP. We have a lot of GAMAP infrastructure already in place (e.g. the code that creates the benchmark plots, etc), so we will continue to use that for the foreseeable future. But NCL gives us additional capabilities that we currently lack in IDL, so we should take advantage of those. For example, we can move away from our IDL regridding code towards NCL's regridding code, especially when we are dealing with netCDF files.
- NCL functions can be interfaced with Python or Fortran code.
--Bob Y. 17:13, 18 September 2013 (EDT)