GEOS-Chem benchmarking
Jump to navigation
Jump to search
Objectives
Benchmarking supports the maintenance of GEOS-Chem as a robust state-of-the-science facility with a nimble grass-roots approach and strong version control. Benchmarking has four main objectives:
- Document a consistent GEOS-Chem model configuration, and the expected characteristics of that configuration.
- Support version control through traceability, and by confirming the expected behavior of model developments submitted by the community.
- Track the evolution of the model over the years.
- Promote scientific transparency of GEOS-Chem.
Procedure
The GEOS-Chem benchmarking procedure is described below.
- Any update to the GEOS-Chem source code or run directories will change the GEOS-Chem version number (X.Y.Z).
- Z versions will be released at intervals determined by the GEOS-Chem Support Team (GCST) and may include bug fixes or updates that do not impact the full-chemistry simulation.
- Any change impacting the standard full-chemistry simulation will require a Y version change and a dedicated 1-month benchmark. The benchmark results will be posted on the wiki and an email will be sent to the developer(s) and the GEOS-Chem Steering Committee (GCSC).
- The developer(s) and GCSC will assess the benchmark results and review a benchmark assessment form on the wiki. If there are any concerns about the benchmark results, the GCST will be notified and further investigation and/or benchmarking may be required.
- If the update is for a specialty simulation (e.g. CO2, CH4, Hg), then a further benchmark may be conducted by the appropriate Working Group.
- Once the developer is satisfied with the changes in the 1-month benchmark, GEOS-Chem Model Scientist Daniel Jacob will review the results and approve the new internal version.
- 1-year full-chemistry and/or transport tracer benchmarks for Y versions will be conducted only if justifiably requested by the developer or by GEOS-Chem Steering Committee members.
- Each new major version release (i.e. X version) will be subject to a 1-year benchmark to be inspected by the GEOS-Chem Steering Committee before approval.
List of GEOS-Chem benchmarks
Links to past 1-month and 1-year benchmark simulations can be found on the GEOS-Chem versions wiki page.
Benchmark output archive
Output files and evaluation plots for 1-month and 1-year benchmark simulations are archived at Harvard as summarized below. GEOS-Chem users may utilize these output for comparisons against their own simulations.
| Directory | Description |
|---|---|
| http://ftp.as.harvard.edu/gcgrid/geos-chem/1mo_benchmarks/ | Contains the following data from the 1-month benchmarks used to evaluate GEOS-Chem:
|
| http://ftp.as.harvard.edu/gcgrid/geos-chem/1yr_benchmarks/ | Contains the following data from the 1-year benchmarks used to evaluate GEOS-Chem:
|
1-month benchmark plotting routines
The 1-month benchmark plotting routines are included with GCPy, a Python took kit available for GEOS-Chem.
1-year benchmark plotting routines
The 1-year benchmark plotting routines are currently written in IDL and can be downloaded from Bitbucket via Git using:
git clone https://bitbucket.org/gcst/gc_1yr_benchmark
The data files needed for the plots are included in the repository and are summarized in the table below.
| Directory | Data year | Species | Data source | Provided by |
|---|---|---|---|---|
| BrO/ | 2007-2008 | BrO | GOME-2 | Justin Parrella (Data used to reproduce Figure 5 of Parrella et al. (2012)) |
| cmdl/ | 2005, 2009 New data available |
CO | GMD (formerly CMDL) | Jennifer Logan and Inna Megretskaia |
| co.prof.for.gmi/ | 2001-2008 (varies by station) |
CO | MOZAIC | Jennifer Logan and Inna Megretskaia |
| eval/aircraft/ NOTE: Other directories in eval/ aren't currently used |
1995-2003 (varies by aircraft campaign) |
C2H6, C3H8, CO, H2O2, HNO3, NO, O3, PAN (Not currently used: ACET, ALK4, C2H2, CH2Br2, CH2BrCl, CH3Br, CH3OH, CH4, CHBr2Cl, CHBrCl2, CO2, ETHE, PRPE) |
See README | Jennifer Logan and Inna Megretskaia |
| netCDF/ | N/A | C2H6, C3H8, CH3I, CO, H2O2, HNO3, NO, O3, OH, PAN, Rn | Various (contains information about station locations) |
Jennifer Logan and Inna Megretskaia |
| PAN/ | Varies | PAN | ARCTAS-A, ARCTAS-B, ARCPAC, INTEX-B, ITCT-2K2 | Emily Fischer (Data used to reproduce Figure S1 of Fischer et al. (2014) Supplement) |
| pm25_data/ | 2005 | PM2.5 | IMPROVE | Colette Heald |
| sondes.for.gmi/ | 1990-2008 (varies by station) |
O3 | Jennifer Logan and Inna Megretskaia | |
| strat/ | 2001-2010 | NOy, O3 | OSIRIS | Dylan Jones |
| surface_ozone/ | O3 | CMDL | Jennifer Logan and Inna Megretskaia |
Update 2005/2009 CMDL CO data to 2013/2014 GMD CO data
This update was included in v11-02c and approved on 21 Sep 2017.
Jenny Fisher wrote:
- Attached is a zip file containing routines to process the GMD (previously CMDL) CO data for use in the benchmarking scripts, as well as the 2013 and 2014 data. I added the 2014 data because there was no ships data for 2013 (or 2012), so this seemed like the best approximation.
- To replace the existing benchmark data with these, you will want to do the following in the 1-year benchmark code:
- Put the 2013data/ and 2014data/ in the data/cmdl/ directory of the 1-year benchmark code
- Change newdata/ to 2013data/ in plot_cmdl_3_models_4_months.pro
- Change Feb09/ to 2013data/ in plot_surface_co_geos_3_models.pro
- Change newdata/ to 2014data/ in plot_ships_3_models_4_months.pro and plot_ships_3_models_co.pro
- Change the line file=pre+name_sta(kk)+'.mn' to file=pre+name_sta(kk)+’.mn.2013' in plot_cmdl_3_models_4_months.pro
- Change the line file=pre+name_sta(kk)+’.mn.2005' to file=pre+name_sta(kk)+’.mn.2013' in plot_surface_co_geos_3_models.pro
- Change the line file=pre+name_sta(kk)+'.mn' to file=pre+name_sta(kk)+’.mn.2014' in plot_ships_3_models_4_months.pro and plot_ships_3_models_co.pro
- As I said before, I think this would be a useful improvement to our benchmarks, rather than using data >10 years old!