GEOS-Chem Newsletter Summer-Fall 2013: Difference between revisions

From Geos-chem
Jump to navigation Jump to search
No edit summary
 
(23 intermediate revisions by 2 users not shown)
Line 16: Line 16:
== Ongoing GEOS-Chem development ==
== Ongoing GEOS-Chem development ==


(1) Ongoing GEOS-Chem Development
[[GEOS-Chem v9-02]] is our development version.  It contains several important scientific updates over the previous version, [[GEOS-Chem v9-01-03]].  We have also restructured the source code significantly as part of our [[#Grid-Independent GEOS-Chem Updates|Grid-Independent GEOS-Chem project]] so that GEOS-Chem v9-02 can run within the NASA GEOS-5 GCM.


Since the [http://acmg.seas.harvard.edu/geos/meetings/2013/index.html 6th International GEOS-Chem Meeting], we have completed the following benchmarks.  Please see the [[GEOS-Chem v9-02]] wiki page for more information about individual features.
=== Features already added to GEOS-Chem v9-02 ===
 
We have completed the following benchmarks since the [http://acmg.seas.harvard.edu/geos/meetings/2013/index.html 6th International GEOS-Chem Meeting].  Please see the [[GEOS-Chem v9-02]] wiki page for more information about individual features.


{| border=1 cellspacing=0 cellpadding=5  
{| border=1 cellspacing=0 cellpadding=5  
Line 45: Line 47:
|
|
*Improved HO2 uptake by aerosol (J. Mao)
*Improved HO2 uptake by aerosol (J. Mao)


|-valign="top"
|-valign="top"
Line 110: Line 111:
|}
|}


Still left to add:
=== Still to be added to GEOS-Chem v9-02 ===
 
The following features are slated to be added into v9-02:


{| border=1 cellspacing=0 cellpadding=5  
{| border=1 cellspacing=0 cellpadding=5  
Line 119: Line 122:


|-valign="top"
|-valign="top"
|[[GEOS-Chem v9-02 benchmark history#v9-02h|v9-02h]]v9-02r:
|v9-02r
|
|
*Acid uptake on dust aerosols (T. D. Fairlie)
*Acid uptake on dust aerosols (T. D. Fairlie)


|-valign="top"
|-valign="top"
[[GEOS-Chem v9-02 benchmark history#v9-02h|v9-02h]]v9-02s:
|v9-02s
|
|
*Standardize global & nested simulations w/ [[GEOS-FP|GEOS-FP met data]]
*Standardize global & nested simulations w/ [[GEOS-FP|GEOS-FP met data]]


|-valign="top"
|-valign="top"
|[[GEOS-Chem v9-02 benchmark history#v9-02h|v9-02h]]v9-02t:
|v9-02t
|
|
*Update to EDGAR v4.2 anthro emissions (S. Philip)
*Update to EDGAR v4.2 anthro emissions (S. Philip)
Line 137: Line 140:


|}
|}
=== Progress ===


The progress on v9-02 has been slower than anticipated.  We attribute this to the following:
The progress on v9-02 has been slower than anticipated.  We attribute this to the following:
Line 142: Line 147:
#Several of the science updates (esp. the chemistry mechanism updates) also required 1-year benchmarks.  This took additional time to set up.  
#Several of the science updates (esp. the chemistry mechanism updates) also required 1-year benchmarks.  This took additional time to set up.  
#We kept finding and fixing bugs and/or numerical issues as we were adding the science updates.  On more than one occasion, we received incorrect code and/or data.  In other instances, we found that code updates submitted to us by users were causing numerical problems (i.e. div-by-zero, out-of-bounds errors) under certain conditions.  
#We kept finding and fixing bugs and/or numerical issues as we were adding the science updates.  On more than one occasion, we received incorrect code and/or data.  In other instances, we found that code updates submitted to us by users were causing numerical problems (i.e. div-by-zero, out-of-bounds errors) under certain conditions.  
#Some of the updates (e.g. Havala's SOA code) came from a very old version (v8-02-01) that predated Git.  Therefore, Melissa had to do the code-merging manually, which was much more time-consuming.
#Some of the updates (e.g. Havala Pye's SOA code) came from a very old version (v8-02-01) that predated Git.  Therefore, we had to do the code-merging manually, which was much more time-consuming.
#While implementing these scientific updates, we were making in parallel widespread structural changes for our Grid-Independent GEOS-Chem project, in order to allow GEOS-Chem to be compatible with the GEOS-5 GCM.  We tested the code rigorously in order to ensure that these structural modifications did not change any scientific results.  At frequent intervals, we have to synchronize code updates from the GIGC development stream back into the standard code, so that the codes will not diverge.
#While implementing these scientific updates, we were making in parallel widespread structural changes for our [[#Grid-Independent GEOS-Chem Updates|Grid-Independent GEOS-Chem project]], in order to allow GEOS-Chem to be compatible with the GEOS-5 GCM.  We tested the code rigorously in order to ensure that these structural modifications did not change any scientific results.  At frequent intervals, we have to synchronize code updates from the GIGC development stream back into the standard code, so that the codes will not diverge.
 
== GEOS-Chem Unit Tester ==
 
We (the GEOS-Chem Support Team) often receive code submissions that have not been adequately debugged.  In some cases, several months may elapse before an error is discovered.  Then we have to send out a bug fix patch, which is a inconvenience. Furthermore, some code submissions that we receive have only ever been tested on one type of met field (e.g. [[GEOS-5]] but not [[MERRA]] or [[GCAP]]).  Hidden errors may later manifest themselves when someone tries to run G-C with a different set of met fields.
 
One of the "wish list" items that came out of the IGC6 software engineering discussions was to have the ability to compile G-C with a standard set of debugging flags in order to find any [[Floating point math issues|numerical issues]] (parallelization problems, out-of-bounds errors, div-by-zero errors, etc.).  Having such a package would help us to find and fix these common errors before we run the benchmark simulations.
 
We are happy to report that we now have such a "Unit Tester" package in place.  The [[Debugging with the GEOS-Chem unit tester|GEOS-Chem Unit Tester]] is an external package of scripts, Makefiles, and run directories that can be downloaded via Git.  It will compile and run short GEOS-Chem simulations (typically 1 or 2 model hours) for various combinations of met fields, horizontal resolutions, and simulation types.  The Unit Tester uses a very strict set of compiler debugging flags designed to trap several common error conditions (out-of-bounds, div-by-zero, Infinity, NaN, etc.).  For each combination of met field/horizontal grid/simulation, the G-C Unit Tester will perform two simulations: one with OpenMP parallelization turned on, and a second with OpenMP turned off.  Ideally, the output of both simulations should be identical; if not, an error in the parallelization exists.
 
The G-C Unit Tester is compatible with versions [[GEOS-Chem v9-02 benchmark history#v9-02o|GEOS-Chem v9-02o]] and higher versions.  In v9-02o, we have now removed the Headers/define.h file, in which several C-preprocessor switches were defined.  All C-preprocessor switches are now set with Makefile options from the command line.  For example, to compile GEOS-Chem for the 4x5 grid with GEOS-5 meteorology, you would type:


(2) GEOS-Chem Unit Tester
We have found that G-C developers often send us code that has not been adequately debugged.  In some cases, several months may elapse before an error is discovered.  Then we have to send out a patch to the G-C user community, which is an inconvenience. Furthermore, some code submissions that we receive have only ever been tested on one type of met field (e.g. GEOS-5 but not MERRA or GCAP).  Hidden errors may later manifest themselves when someone tries to run G-C with a different set of met fields.
One of the "wish list" items that came out of the IGC6 software engineering discussions was to have the ability to compile G-C with a standard set of debugging flags in order to find any numerical issues (parallelization problems, out-of-bounds errors, div-by-zero errors, etc.).  Having such a package would help us to find and fix these common errors before we run the benchmark simulations.
We are happy to report that we now have such a "Unit Tester" package in place.  The G-C Unit Tester is an external package of scripts, Makefiles, and run directories that can be downloaded via Git.  It will compile and run short GEOS-Chem simulations (typically 1 or 2 model hours) for various combinations of met fields, horizontal resolutions, and simulation types.  The Unit Tester uses a very strict set of compiler debugging flags designed to trap several common error conditions (out-of-bounds, div-by-zero, Infinity, NaN, etc.).  For each combination of met field/horizontal grid/simulation, the G-C Unit Tester will perform two simulations: one with OpenMP parallelization turned on, and a second with OpenMP turned off.  Ideally, the output of both simulations should be identical; if not, an error in the parallelization exists.
The G-C Unit Tester is compatible with versions v9-02o and higher versions of GEOS-Chem.  In v9-02o, we have now removed the Headers/define.h file, in which several C-preprocessor switches were defined.  All C-preprocessor switches are now set with Makefile options from the command line.  For example, to compile GEOS-Chem for the 4x5 grid with GEOS-5 meteorology, you would type:
   make -j4 MET=geos5 GRID=4x5 ...
   make -j4 MET=geos5 GRID=4x5 ...
or if you wanted to compile for the 0.5 x 0.667 NA nested grid, you would also add an additional flag:
or if you wanted to compile for the 0.5 x 0.667 NA nested grid, you would also add an additional flag:
   make -j4 MET=geos5 GRID=05x0666 NEST=na ...
   make -j4 MET=geos5 GRID=05x0666 NEST=na ...
Setting all of the C-preprocessor switches with command-line Makefile options allows the G-C Unit Tester to repeatedly compile G-C from the main script. This would not have been possible before.
Setting all of the C-preprocessor switches with command-line Makefile options allows the G-C Unit Tester to repeatedly compile G-C from the main script. This would not have been possible before.
Using the GEOS-Chem Unit Tester, we have already identified and corrected several numerical issues in v9-02p, as shown on this wiki page.
For more information, please see the GEOS-Chem Unit Tester wiki page.


(3) GEOS-FP met fields
Using the GEOS-Chem Unit Tester, we have already identified and corrected several numerical issues in v9-02p, as shown on our [[Numerical issues discovered in GEOS-Chem]] wiki page.
At present, the G-C v9-02 development version "unofficially" supports the GMAO GEOS-FP met fields.  The final release of v9-02 will "officially" support the GEOS-FP met product.
 
For the SEAC4RS mission, we are currently running GEOS-Chem in near-real-time mode using the 0.25 x 0.3125 N. America nested grid. This code is a research product, and some issues will still need to be corrected before we can bring this capability into the standard code.  We will also have to spin up some stratospheric quantities so that we can do a 1-yr benchmark full-chemistry simulation with GEOS-FP met.  
For more information, please see [[Debugging with the GEOS-Chem unit tester|our ''Debugging with the GEOS-Chem unit tester'' wiki page]].
Karen Yu has completed a 1-year Rn-Pb-Be benchmark comparing GEOS-5 and GEOS-FP met data.  The results are being posted on our Rn-Pb-Be simulation wiki page.
 
I also recommend that we make adjustments to our existing GEOS-FP data archive prior to the release of v9-02:
== GEOS-FP met fields ==
(a) When the GEOS-FP data first debuted, NASA/GMAO called this product "GEOS-5.7.2", which reflected the version of the GEOS-DAS system used to create the met product.  Since then, the nomenclature has changed to "GEOS5-FP" (what we call GEOS-FP), where "FP" stands for "forward processing".  The names of the GEOS-FP processed data files that we read into GEOS-Chem still carry the "GEOS-5.7.2" designation.  I recommend that we rename the files so that they carry the GEOS-FP designation before the official release of v9-02.  This will avoid confusion in the future.
 
[[GEOS-FP]] ("forward processing") is the latest GEOS-5 met data product from [http://gmao.gsfc.nasa.gov NASA/GMAO].  It was originally introduced in 2011 as GEOS-5.7.2, but this nomenclature has now become obsolete.  GEOS-FP—which is now produced with version 5.11.0 of the GEOS-DAS (Data Assimilation System)— has finer native horizontal resolution (0.25° lat x 0.3125° lon) and temporal resolution (hourly data and 3-hourly data) than older GMAO met products such as [[GEOS-5#GEOS-5.2.0|GEOS-5.2.0]], [[MERRA|the MERRA reanalysis]], and [[GMAO GEOS-4|GEOS-4]].
 
For more information, please see the following GEOS-FP resources:
 
#[http://acmg.seas.harvard.edu/geos/wiki_docs/geos5/GEOS_5_FP_File_Specification_ON4v1_0.pdf GEOS-FP (v1.0) file specification document (11 Jun 2013)]
#[[List of GEOS-FP met fields|List of GEOS-FP met fields for GEOS-Chem]]
#[[GEOS-FP implementation details|Information about implementing GEOS-FP into GEOS-Chem]]
 
=== Current status ===
 
At present, the G-C v9-02 development version "unofficially" supports the GMAO [[GEOS-FP]] met fields.  The final release of v9-02 will "officially" support the GEOS-FP met product.  
 
For the [https://espo.nasa.gov/missions/seac4rs/content/SEAC4RS NASA SEAC4RS mission] (August thru mid-September 2013), we are currently running GEOS-Chem in near-real-time mode using the 0.25° x 0.3125° North America nested grid. This code is a research product, and some issues will still need to be corrected before we can bring this capability into the standard GEOS-Chem code.  We will also have to spin up some stratospheric quantities so that we can do a 1-yr benchmark full-chemistry simulation with GEOS-FP met.  
 
Karen Yu has completed a 1-year Rn-Pb-Be benchmark comparing GEOS-5 and GEOS-FP met data.  The results are being posted on [[Rn-Pb-Be simulation|our ''Rn-Pb-Be simulation'' wiki page]].
 
=== Editing the existing GEOS-FP data archive ===
 
The GEOS-Chem Support Team also recommends that we make adjustments to our existing [[GEOS-FP]] data archive prior to the release of v9-02:  
 
#<p>When the GEOS-FP data first debuted, NASA/GMAO called this product "GEOS-5.7.2", which reflected the version of the GEOS-DAS system used to create the met product.  Since then, the nomenclature has changed to "GEOS5-FP" (what we call GEOS-FP), where "FP" stands for "forward processing".  The names of the GEOS-FP processed data files that we read into GEOS-Chem still carry the "GEOS-5.7.2" designation.  I recommend that we rename the files so that they carry the GEOS-FP designation before the official release of v9-02.  This will avoid confusion in the future.</p>
#<p>We have since learned that some of the netCDF attributes in our processed GEOS-FP data files do not conform to accepted community standards (e.g [http://ferret.wrc.noaa.gov/noaa_coop/coop_cdf_profile.html COARDS], [http://cf-pcmdi.llnl.gov/ CF]).  For example, the unit string "unitless" should be replaced with "1", the string "conventions" should be replaced with "Conventions", etc.  I propose that we adjust the internal file attributes accordingly before the release of v9-02.  This can be done by running the netCDF operators (NCO) on the existing data files.  We do NOT have to reprocess the files from scratch.</p>
#<p>We have also since learned that the ESMF/MAPL libraries (which the Grid-Independent GEOS-Chem code relies on) cannot read data files containing fields with different numbers of vertical levels.  All of the data fields in a single file must be of the same vertical dimension.  In our current GEOS-FP data files that we have already processed, one file type contains some fields that are stored on both level edges (73 vertical points) and other fields that are stored on level centers (72 vertical points).  We would therefore have to split off the field that is stored on level edges into a separate file.  Again, this can be done in a straightforward manner by applying the netCDF operators (NCO) to the existing files.  We do not have to recreate the files from the raw data.</p>
 
== Grid-Independent GEOS-Chem Updates==
 
Much progress has been made on the [[Grid-Independent GEOS-Chem]] project (aka GIGC) this summer.  In particular:


(b) We have since learned that some of the netCDF attributes in our processed GEOS-FP data files do not conform to accepted community standards.  For example, the unit string "unitless" should be replaced with "1", the string "conventions" should be replaced with "Conventions", etc.  I propose that we adjust the internal file attributes accordingly before the release of v9-02.  This can be done by running the netCDF operators (NCO) on the existing data files.  We do NOT have to reprocess the files from scratch.
=== Integration into the GEOS Data Assimilation System (GEOS-DAS) ===


(c) We have also since learned that the ESMF/MAPL libraries (which the Grid-Independent GEOS-Chem code relies on) cannot read data files containing fields with different numbers of vertical levels.  All of the data fields in a single file must be of the same vertical dimension.  In our current GEOS-FP data files that we have already processed, one file type contains some fields that are stored on both level edges (73 vertical points) and other fields that are stored on level centers (72 vertical points).  We would therefore have to split off the field that is stored on level edges into a separate file. Again, this can be done in a straightforward manner by applying the netCDF operators (NCO) to the existing files.  We do not have to recreate the files from the raw data.
Mike Long & Christoph Keller visited NASA/GMAO in August. They have implemented the [[HEMCO|Grid-Independent GEOS-Chem Emissions Component]] into the GEOS-5 GCM.  For initial testing purposes, the Emissions Component has been implemented as part of the existing [[Grid-Independent GEOS-Chem: Chemistry Component|Chemistry Component]], rather than as a separate component. The Emissions Component will be later separated into its own component within the GCM.


(4) Grid-Independent GEOS-Chem: into GEOS-DAS
Mike Long & Christoph Keller visited NASA/GMAO in August.  They have implemented the Grid-Independent GEOS-Chem (GIGC) Emissions Component into the GEOS-5 GCM.  For initial testing purposes, the Emissions Component has been implemented as part of the existing Chemistry Component, rather than as a separate component. The Emissions Component will be later separated into its own component within the GCM.
The Emissions Component successfully reads external emissions data (in netCDF format) from various inventories and processes it into resultant emissions per species.  At present, the Emissions are not being fed into the Chemistry, but this will be the next step.
The Emissions Component successfully reads external emissions data (in netCDF format) from various inventories and processes it into resultant emissions per species.  At present, the Emissions are not being fed into the Chemistry, but this will be the next step.
At some point we should devote some effort into rewriting the existing GEOS-Chem dry deposition routines.  The current dry deposition code relies on a fixed Olson vegetation map.  This map is difficult to regrid to arbitrary resolution within the GEOS-5 GCM, as you have to account for the # of land types and the fraction of each land type in the coarse grid box.  Ideally, we would like to remove dependence on the Olson land map from the GEOS-Chem dry deposition code so that we can better interface it with other land maps or dynamic vegetation models. 


(5) Grid-Independent GEOS-Chem: ESMF standalone version
At some point we should devote some effort into rewriting the existing GEOS-Chem dry deposition routines.  The current dry deposition code relies on a fixed Olson vegetation map.  This map is difficult to regrid to arbitrary resolution within the GEOS-5 GCM, as you have to account for the # of land types and the fraction of each land type in the coarse grid box.  Ideally, we would like to remove dependence on the Olson land map from the GEOS-Chem dry deposition code so that we can better interface it with other land maps or dynamic vegetation models.
In addition to integrating the Grid-Independent GEOS-Chem (GIGC) into the GEOS-5/GCM, we are also working towards creating a standalone version of the GIGC that will use ESMF/MAPL libraries to run with message-passing (MPI) parallelization.  This will allow G-C to run on hundreds or thousands of CPUs.  We are collaborating with Kevin Bowman and his team at JPL on this effort.
 
=== Grid-Independent GEOS-Chem: HPC standalone version ===
 
In addition to integrating the [[Grid-Independent GEOS-Chem]] into the GEOS-5/GCM, we are also working towards creating a standalone version of the GIGC that will use ESMF/MAPL libraries to run with message-passing (MPI) parallelization.  This will allow G-C to run on hundreds or thousands of CPUs on high-performance computing (HPC) systems.  We are collaborating with Kevin Bowman and his team at JPL on this effort.
 
The initial challenge has been to create the directory structure and build system for the standalone GIGC.  The GEOS-5 GCM has a complicated build system.  It relies on many hardwired makefiles and scripts, which are difficult to use, let alone modify.  We do not want to replicate the GEOS-5 build system in its entirety, but rather would like to take essential components and merge them into the existing GEOS-Chem's build system.  The work is ongoing.
The initial challenge has been to create the directory structure and build system for the standalone GIGC.  The GEOS-5 GCM has a complicated build system.  It relies on many hardwired makefiles and scripts, which are difficult to use, let alone modify.  We do not want to replicate the GEOS-5 build system in its entirety, but rather would like to take essential components and merge them into the existing GEOS-Chem's build system.  The work is ongoing.


(6) Flexchem
== Flexchem ==
 
We are still waiting on Mat Evans to implement the rate law library in KPP.
We are still waiting on Mat Evans to implement the rate law library in KPP.


(7) TAU / KPPA
== TAU and KPPA ==
 
The GEOS-Chem Support Team recently met with John Linford from ParaTools, Inc.  (John is a former student of Adrian Sandu at VT).  John gave us a demonstration of two software items: (1) TAU and (2) KPPA.
The GEOS-Chem Support Team recently met with John Linford from ParaTools, Inc.  (John is a former student of Adrian Sandu at VT).  John gave us a demonstration of two software items: (1) TAU and (2) KPPA.
TAU is a performance profiler tool.  It can examine a software product like GEOS-Chem and provide reports of which routines are taking the most time to execute.  We feel that this can be an invaulable tool in helping us to identify and eliminate bottlenecks from the GEOS-Chem code.
KPPA (the Kinetic pre-processor, accelerated) is an updated version of KPP that was totally rewritten from scratch.  It is compatible with the existing KPP input files, and can be quickly compiled.  It can generate chemical solver code that is compatible with OpenMP parallelization or for GPU (graphics processor) chips. Furthermore, KPPA has some very useful diagnostics that can let you see the online evolution of species and the model error during a run.
John has expressed willingness to work with us in the future.  We are going to try to get him to come to give a tutorial at Harvard in the near future (maybe early next year).


(7) NCL (NCAR command language)
*<p>[http://www.cs.uoregon.edu/research/tau/home.php TAU] is a performance profiler tool.  It can examine a software product like GEOS-Chem and provide reports of which routines are taking the most time to execute.  We feel that this can be an invaulable tool in helping us to identify and eliminate bottlenecks from the GEOS-Chem code.</p>
Several of us at Harvard had the opportunity to participate in a tutorial course on NCL, the NCAR command language.  NCL is an extremely useful file I/O and graphics suite that is geared towards working with netCDF and similar file formats.   
 
*<p>[http://www.paratools.com/Kppa KPPA] (the Kinetic pre-processor, accelerated) is an updated version of KPP that was totally rewritten from scratch.  It is compatible with the existing KPP input files, and can be quickly compiled.  It can generate chemical solver code that is compatible with OpenMP parallelization or for GPU (graphics processor) chips. Furthermore, KPPA has some very useful diagnostics that can let you see the online evolution of species and the model error during a run.</p>
 
John has expressed willingness to work with the GEOS-Chem community in the future.  We are going to explore ways these tools may be used with GEOS-Chem.
 
== NCL (NCAR command language) ==
 
Several of us at Harvard had the opportunity to participate in a tutorial course on [http://www.ncl.ucar.edu/NCL, the NCAR command language].  NCL is an extremely useful file I/O and graphics suite that is geared towards working with netCDF and similar file formats.   
 
With NCL, it is almost trivial to read and process data from a netCDF, HDF, or GRIB file.  To do this in other languages like Fortran, IDL, or Python involves several more steps.  Furthermore, NCL has some useful regridding utilities:
With NCL, it is almost trivial to read and process data from a netCDF, HDF, or GRIB file.  To do this in other languages like Fortran, IDL, or Python involves several more steps.  Furthermore, NCL has some useful regridding utilities:
• Bilinear interpolation  -- used for fields w/o an area dependence (e.g. T, PS, etc).
• Area conserving regridding – used for fluxes (kg/m2/s) or precipitation quantities
• ESMF regridding – used for regridding from nonlinear or other complex grids
As part of the GIGC development, we will be migrating away from binary file format and towards netCDF file format.  NCL will allow us to read and regrid files (such as restart files, emisisons data, etc) with ease.  As time allows, we will create a repository of NCL scripts that can be shared with the G-C user community. 
NOTE: NCL will not totally replace other plotting packages like Matlab or GAMAP.  We have a lot of GAMAP infrastructure already in place (e.g. the code that creates the benchmark plots, etc), so we will continue to use that for the foreseeable future.  But NCL gives us additional capabilities that we currently lack in IDL, so we should take advantage of those.  For example, we can move away from our IDL regridding code towards NCL's regridding code, especially when we are dealing with netCDF files.


13 Sep 2013
*Bilinear interpolation  -- used for fields w/o an area dependence (e.g. T, PS, etc).
Bob Yantosca
*Area conserving regridding – used for fluxes (kg/m2/s) or precipitation quantities
yantosca@seas.harvard.edu
*ESMF regridding – used for regridding from nonlinear or other complex grids
 
As part of the [[Grid-Independent GEOS-Chem]] development, we will be migrating away from binary file format and towards netCDF file format.  NCL will allow us to read and regrid files (such as restart files, emisisons data, etc) with ease.  As time allows, we will create a repository of NCL scripts that can be shared with the G-C user community.
 
NOTES:
 
#NCL will not totally replace other plotting packages like Matlab or [http://acmg.seas.harvard.edu/gamap/ GAMAP].  We have a lot of GAMAP infrastructure already in place (e.g. the code that creates the benchmark plots, etc), so we will continue to use that for the foreseeable future.  But NCL gives us additional capabilities that we currently lack in IDL, so we should take advantage of those.  For example, we can move away from our IDL regridding code towards NCL's regridding code, especially when we are dealing with netCDF files.
#NCL functions can be interfaced with Python or Fortran code.
 
--[[User:Bmy|Bob Y.]] 17:13, 18 September 2013 (EDT)

Latest revision as of 16:50, 8 December 2016

Dear GEOS-Chem users,

We are happy to bring you the Summer-Fall edition of the GEOS-Chem Newsletter! Please read below to find out the latest goings-on in the GEOS-Chem community.

Sincerely,

The GEOS-Chem Support Team
geos-chem-support@as.harvard.edu

GEOS-Chem Steering Committee

The GEOS-Chem Steering Committee (GCSC) met via telecon on 17 September 2013. We invite you to read the meeting minutes (PDF format).

The GCSC will meet again in late November or early December, date TBD.

Ongoing GEOS-Chem development

GEOS-Chem v9-02 is our development version. It contains several important scientific updates over the previous version, GEOS-Chem v9-01-03. We have also restructured the source code significantly as part of our Grid-Independent GEOS-Chem project so that GEOS-Chem v9-02 can run within the NASA GEOS-5 GCM.

Features already added to GEOS-Chem v9-02

We have completed the following benchmarks since the 6th International GEOS-Chem Meeting. Please see the GEOS-Chem v9-02 wiki page for more information about individual features.

Version Date Features
v9-02h 15 May 2013
  • Removal of NOx-Ox partitioning (GEOS-Chem Support Team)
v9-02i 17 May 2013
  • Inhibition of N2O5 uptake by aerosol (L. Zhang)

v9-02j

28 May 2013
  • Improved HO2 uptake by aerosol (J. Mao)
v9-02k 07 Jun 2013
  • Cleaner integration of TOMAS into GEOS-Chem (S. Farina)
  • Replacing data arrays with derived types (for GIGC project)
  • Streamlining of the Hg code (Chris Holmes)
  • Removing inefficient subroutine calls (GCST)
v9-02l 26 Jun 2013
  • Updated NH3 seasonality over the USA (Lin Zhang)
  • Updated Canadian NH3 inventory (Wai-Ho Lo)
  • Many bug fixes, especially in emissions (see wiki for details)
v9-02m 30 Jul 2013
  • RCP emission scenarios (C. Holmes)
  • EDGAR v4.2 for CH4 (K. Wecht)
  • Updated anthro scale factors thru 2010 (A. van Donkelaar)
  • GFED3 emissions to 2011 (P. Kasibhatla)
  • Updated TOMS overhead O3 to 2010 (J. Fisher)
  • Add TOMAS12, TOMAS15 (S. Farina)
  • Prevent negative emissions over Canada (F. Paulot, C. Keller)
  • Minor bug fixes (GCST)
v9-02n 12 Aug 2013
  • AEIC aircraft emissions (S. Eastham, S. Barrett)
v9-02o 03 Sep 2013
  • SOA with semi-volatile POA (H.O.T. Pye)
  • Non-local PBL mixing in Rn-Pb-Be (J. Lin)
  • Removal of define.h, updates to Makefiles (GCST)
  • Bug fixes in ND36 when ship emissions are turned off (Y. Davila)
  • Bug fixes in TOMAS (S. Farina)
v9-02p 13 Sep 2013
  • Cloudwater pH for sulfate formation (B. Alexander)
  • Fix molecular weights of RIP and IEPOX in drydep (GCST)
  • Bug fixes to prevent floating-point invalid errors (GCST)
v9-02q 17 Sep 2013
  • Update jv_spec.dat and jv_spec_aod.dat with better representation of OC growth with RH and correction to sulfate optics (D. Ridley)
  • Bug fix in jv_spec_aod.dat for dust optics (G. Curci)

Still to be added to GEOS-Chem v9-02

The following features are slated to be added into v9-02:

Version Features
v9-02r
  • Acid uptake on dust aerosols (T. D. Fairlie)
v9-02s
v9-02t
  • Update to EDGAR v4.2 anthro emissions (S. Philip)
  • Fix bug in CAC NH3 emission files (Wai Ho Lo)
  • Improve temporal resolution of anthro CO2 (R. Nassar)
  • Replace CASA CO2 w/ year-specific fluxes (R. Nassar)

Progress

The progress on v9-02 has been slower than anticipated. We attribute this to the following:

  1. Several of the science updates (esp. the chemistry mechanism updates) also required 1-year benchmarks. This took additional time to set up.
  2. We kept finding and fixing bugs and/or numerical issues as we were adding the science updates. On more than one occasion, we received incorrect code and/or data. In other instances, we found that code updates submitted to us by users were causing numerical problems (i.e. div-by-zero, out-of-bounds errors) under certain conditions.
  3. Some of the updates (e.g. Havala Pye's SOA code) came from a very old version (v8-02-01) that predated Git. Therefore, we had to do the code-merging manually, which was much more time-consuming.
  4. While implementing these scientific updates, we were making in parallel widespread structural changes for our Grid-Independent GEOS-Chem project, in order to allow GEOS-Chem to be compatible with the GEOS-5 GCM. We tested the code rigorously in order to ensure that these structural modifications did not change any scientific results. At frequent intervals, we have to synchronize code updates from the GIGC development stream back into the standard code, so that the codes will not diverge.

GEOS-Chem Unit Tester

We (the GEOS-Chem Support Team) often receive code submissions that have not been adequately debugged. In some cases, several months may elapse before an error is discovered. Then we have to send out a bug fix patch, which is a inconvenience. Furthermore, some code submissions that we receive have only ever been tested on one type of met field (e.g. GEOS-5 but not MERRA or GCAP). Hidden errors may later manifest themselves when someone tries to run G-C with a different set of met fields.

One of the "wish list" items that came out of the IGC6 software engineering discussions was to have the ability to compile G-C with a standard set of debugging flags in order to find any numerical issues (parallelization problems, out-of-bounds errors, div-by-zero errors, etc.). Having such a package would help us to find and fix these common errors before we run the benchmark simulations.

We are happy to report that we now have such a "Unit Tester" package in place. The GEOS-Chem Unit Tester is an external package of scripts, Makefiles, and run directories that can be downloaded via Git. It will compile and run short GEOS-Chem simulations (typically 1 or 2 model hours) for various combinations of met fields, horizontal resolutions, and simulation types. The Unit Tester uses a very strict set of compiler debugging flags designed to trap several common error conditions (out-of-bounds, div-by-zero, Infinity, NaN, etc.). For each combination of met field/horizontal grid/simulation, the G-C Unit Tester will perform two simulations: one with OpenMP parallelization turned on, and a second with OpenMP turned off. Ideally, the output of both simulations should be identical; if not, an error in the parallelization exists.

The G-C Unit Tester is compatible with versions GEOS-Chem v9-02o and higher versions. In v9-02o, we have now removed the Headers/define.h file, in which several C-preprocessor switches were defined. All C-preprocessor switches are now set with Makefile options from the command line. For example, to compile GEOS-Chem for the 4x5 grid with GEOS-5 meteorology, you would type: 

  make -j4 MET=geos5 GRID=4x5 ...

or if you wanted to compile for the 0.5 x 0.667 NA nested grid, you would also add an additional flag: 

  make -j4 MET=geos5 GRID=05x0666 NEST=na ...

Setting all of the C-preprocessor switches with command-line Makefile options allows the G-C Unit Tester to repeatedly compile G-C from the main script. This would not have been possible before.

Using the GEOS-Chem Unit Tester, we have already identified and corrected several numerical issues in v9-02p, as shown on our Numerical issues discovered in GEOS-Chem wiki page.

For more information, please see our Debugging with the GEOS-Chem unit tester wiki page.

GEOS-FP met fields

GEOS-FP ("forward processing") is the latest GEOS-5 met data product from NASA/GMAO. It was originally introduced in 2011 as GEOS-5.7.2, but this nomenclature has now become obsolete. GEOS-FP—which is now produced with version 5.11.0 of the GEOS-DAS (Data Assimilation System)— has finer native horizontal resolution (0.25° lat x 0.3125° lon) and temporal resolution (hourly data and 3-hourly data) than older GMAO met products such as GEOS-5.2.0, the MERRA reanalysis, and GEOS-4.

For more information, please see the following GEOS-FP resources:

  1. GEOS-FP (v1.0) file specification document (11 Jun 2013)
  2. List of GEOS-FP met fields for GEOS-Chem
  3. Information about implementing GEOS-FP into GEOS-Chem

Current status

At present, the G-C v9-02 development version "unofficially" supports the GMAO GEOS-FP met fields. The final release of v9-02 will "officially" support the GEOS-FP met product.

For the NASA SEAC4RS mission (August thru mid-September 2013), we are currently running GEOS-Chem in near-real-time mode using the 0.25° x 0.3125° North America nested grid. This code is a research product, and some issues will still need to be corrected before we can bring this capability into the standard GEOS-Chem code. We will also have to spin up some stratospheric quantities so that we can do a 1-yr benchmark full-chemistry simulation with GEOS-FP met.

Karen Yu has completed a 1-year Rn-Pb-Be benchmark comparing GEOS-5 and GEOS-FP met data. The results are being posted on our Rn-Pb-Be simulation wiki page.

Editing the existing GEOS-FP data archive

The GEOS-Chem Support Team also recommends that we make adjustments to our existing GEOS-FP data archive prior to the release of v9-02:

  1. When the GEOS-FP data first debuted, NASA/GMAO called this product "GEOS-5.7.2", which reflected the version of the GEOS-DAS system used to create the met product. Since then, the nomenclature has changed to "GEOS5-FP" (what we call GEOS-FP), where "FP" stands for "forward processing". The names of the GEOS-FP processed data files that we read into GEOS-Chem still carry the "GEOS-5.7.2" designation. I recommend that we rename the files so that they carry the GEOS-FP designation before the official release of v9-02. This will avoid confusion in the future.

  2. We have since learned that some of the netCDF attributes in our processed GEOS-FP data files do not conform to accepted community standards (e.g COARDS, CF). For example, the unit string "unitless" should be replaced with "1", the string "conventions" should be replaced with "Conventions", etc. I propose that we adjust the internal file attributes accordingly before the release of v9-02. This can be done by running the netCDF operators (NCO) on the existing data files. We do NOT have to reprocess the files from scratch.

  3. We have also since learned that the ESMF/MAPL libraries (which the Grid-Independent GEOS-Chem code relies on) cannot read data files containing fields with different numbers of vertical levels. All of the data fields in a single file must be of the same vertical dimension. In our current GEOS-FP data files that we have already processed, one file type contains some fields that are stored on both level edges (73 vertical points) and other fields that are stored on level centers (72 vertical points). We would therefore have to split off the field that is stored on level edges into a separate file. Again, this can be done in a straightforward manner by applying the netCDF operators (NCO) to the existing files. We do not have to recreate the files from the raw data.

Grid-Independent GEOS-Chem Updates

Much progress has been made on the Grid-Independent GEOS-Chem project (aka GIGC) this summer. In particular:

Integration into the GEOS Data Assimilation System (GEOS-DAS)

Mike Long & Christoph Keller visited NASA/GMAO in August. They have implemented the Grid-Independent GEOS-Chem Emissions Component into the GEOS-5 GCM. For initial testing purposes, the Emissions Component has been implemented as part of the existing Chemistry Component, rather than as a separate component. The Emissions Component will be later separated into its own component within the GCM.

The Emissions Component successfully reads external emissions data (in netCDF format) from various inventories and processes it into resultant emissions per species. At present, the Emissions are not being fed into the Chemistry, but this will be the next step.

At some point we should devote some effort into rewriting the existing GEOS-Chem dry deposition routines. The current dry deposition code relies on a fixed Olson vegetation map. This map is difficult to regrid to arbitrary resolution within the GEOS-5 GCM, as you have to account for the # of land types and the fraction of each land type in the coarse grid box. Ideally, we would like to remove dependence on the Olson land map from the GEOS-Chem dry deposition code so that we can better interface it with other land maps or dynamic vegetation models.

Grid-Independent GEOS-Chem: HPC standalone version

In addition to integrating the Grid-Independent GEOS-Chem into the GEOS-5/GCM, we are also working towards creating a standalone version of the GIGC that will use ESMF/MAPL libraries to run with message-passing (MPI) parallelization. This will allow G-C to run on hundreds or thousands of CPUs on high-performance computing (HPC) systems. We are collaborating with Kevin Bowman and his team at JPL on this effort.

The initial challenge has been to create the directory structure and build system for the standalone GIGC. The GEOS-5 GCM has a complicated build system. It relies on many hardwired makefiles and scripts, which are difficult to use, let alone modify. We do not want to replicate the GEOS-5 build system in its entirety, but rather would like to take essential components and merge them into the existing GEOS-Chem's build system. The work is ongoing.

Flexchem

We are still waiting on Mat Evans to implement the rate law library in KPP.

TAU and KPPA

The GEOS-Chem Support Team recently met with John Linford from ParaTools, Inc. (John is a former student of Adrian Sandu at VT). John gave us a demonstration of two software items: (1) TAU and (2) KPPA.

  • TAU is a performance profiler tool. It can examine a software product like GEOS-Chem and provide reports of which routines are taking the most time to execute. We feel that this can be an invaulable tool in helping us to identify and eliminate bottlenecks from the GEOS-Chem code.

  • KPPA (the Kinetic pre-processor, accelerated) is an updated version of KPP that was totally rewritten from scratch. It is compatible with the existing KPP input files, and can be quickly compiled. It can generate chemical solver code that is compatible with OpenMP parallelization or for GPU (graphics processor) chips. Furthermore, KPPA has some very useful diagnostics that can let you see the online evolution of species and the model error during a run.

John has expressed willingness to work with the GEOS-Chem community in the future. We are going to explore ways these tools may be used with GEOS-Chem.

NCL (NCAR command language)

Several of us at Harvard had the opportunity to participate in a tutorial course on the NCAR command language. NCL is an extremely useful file I/O and graphics suite that is geared towards working with netCDF and similar file formats.

With NCL, it is almost trivial to read and process data from a netCDF, HDF, or GRIB file. To do this in other languages like Fortran, IDL, or Python involves several more steps. Furthermore, NCL has some useful regridding utilities:

  • Bilinear interpolation -- used for fields w/o an area dependence (e.g. T, PS, etc).
  • Area conserving regridding – used for fluxes (kg/m2/s) or precipitation quantities
  • ESMF regridding – used for regridding from nonlinear or other complex grids

As part of the Grid-Independent GEOS-Chem development, we will be migrating away from binary file format and towards netCDF file format. NCL will allow us to read and regrid files (such as restart files, emisisons data, etc) with ease. As time allows, we will create a repository of NCL scripts that can be shared with the G-C user community.

NOTES:

  1. NCL will not totally replace other plotting packages like Matlab or GAMAP. We have a lot of GAMAP infrastructure already in place (e.g. the code that creates the benchmark plots, etc), so we will continue to use that for the foreseeable future. But NCL gives us additional capabilities that we currently lack in IDL, so we should take advantage of those. For example, we can move away from our IDL regridding code towards NCL's regridding code, especially when we are dealing with netCDF files.
  2. NCL functions can be interfaced with Python or Fortran code.

--Bob Y. 17:13, 18 September 2013 (EDT)

Retrieved from "https://wiki.seas.harvard.edu/geos-chem/index.php?title=GEOS-Chem_Newsletter_Summer-Fall_2013&oldid=29339"