Benchmarking supports the maintenance of GEOS-Chem as a robust state-of-the-science facility with a nimble grass-roots approach and strong version control. Benchmarking has four main objectives:
- Document a consistent GEOS-Chem model configuration, and the expected characteristics of that configuration.
- Support version control through traceability, and by confirming the expected behavior of model developments submitted by the community.
- Track the evolution of the model over the years.
- Promote scientific transparency of GEOS-Chem.
The GEOS-Chem benchmarking procedure is described below.
- Any update to the GEOS-Chem source code or run directories will change the GEOS-Chem version number (X.Y.Z).
- Z versions will be released at intervals determined by the GEOS-Chem Support Team (GCST) and may include bug fixes or updates that do not impact the full-chemistry simulation.
- Any change impacting the standard full-chemistry simulation will require a Y version change and a dedicated 1-month benchmark. The benchmark results will be posted on the wiki and an email will be sent to the developer(s) and the GEOS-Chem Steering Committee (GCSC).
- The developer(s) and GCSC will assess the benchmark results and review a benchmark assessment form on the wiki. If there are any concerns about the benchmark results, the GCST will be notified and further investigation and/or benchmarking may be required.
- If the update is for a specialty simulation (e.g. CO2, CH4, Hg), then a further benchmark may be conducted by the appropriate Working Group.
- Once the developer is satisfied with the changes in the 1-month benchmark, GEOS-Chem Model Scientist Daniel Jacob will review the results and approve the new internal version.
- 1-year full-chemistry and/or transport tracer benchmarks for Y versions will be conducted only if justifiably requested by the developer or by GEOS-Chem Steering Committee members.
- Each new major version release (i.e. X version) will be subject to a 1-year benchmark to be inspected by the GEOS-Chem Steering Committee before approval.
List of GEOS-Chem benchmarks
Links to past 1-month and 1-year benchmark simulations can be found on the GEOS-Chem versions wiki page.
Benchmark output archive
Output files and evaluation plots for 1-month and 1-year benchmark simulations are archived at Harvard as summarized below. GEOS-Chem users may utilize these output for comparisons against their own simulations.
|http://ftp.as.harvard.edu/gcgrid/geos-chem/1mo_benchmarks/||Contains the following data from the 1-month benchmarks used to evaluate GEOS-Chem:
|http://ftp.as.harvard.edu/gcgrid/geos-chem/1yr_benchmarks/||Contains the following data from the 1-year benchmarks used to evaluate GEOS-Chem:
1-month benchmark plotting routines
The 1-month benchmark plotting routines are included with GCPy, a Python took kit available for GEOS-Chem.
1-year benchmark plotting routines
The 1-year benchmark plotting routines are currently written in IDL and can be downloaded from Bitbucket via Git using:
git clone https://bitbucket.org/gcst/gc_1yr_benchmark
The data files needed for the plots are included in the repository and are summarized in the table below.
|Directory||Data year||Species||Data source||Provided by|
(Data used to reproduce Figure 5 of Parrella et al. (2012))
New data available
|CO||GMD (formerly CMDL)||Jennifer Logan and Inna Megretskaia|
(varies by station)
|CO||MOZAIC||Jennifer Logan and Inna Megretskaia|
NOTE: Other directories in eval/ aren't currently used
(varies by aircraft campaign)
|C2H6, C3H8, CO, H2O2, HNO3, NO, O3, PAN
(Not currently used: ACET, ALK4, C2H2, CH2Br2, CH2BrCl, CH3Br, CH3OH, CH4, CHBr2Cl, CHBrCl2, CO2, ETHE, PRPE)
|See README||Jennifer Logan and Inna Megretskaia|
|netCDF/||N/A||C2H6, C3H8, CH3I, CO, H2O2, HNO3, NO, O3, OH, PAN, Rn||Various
(contains information about station locations)
|Jennifer Logan and Inna Megretskaia|
|PAN/||Varies||PAN||ARCTAS-A, ARCTAS-B, ARCPAC, INTEX-B, ITCT-2K2||Emily Fischer|
(Data used to reproduce Figure S1 of Fischer et al. (2014) Supplement)
(varies by station)
|O3||Jennifer Logan and Inna Megretskaia|
|strat/||2001-2010||NOy, O3||OSIRIS||Dylan Jones|
|surface_ozone/||O3||CMDL||Jennifer Logan and Inna Megretskaia|
Update 2005/2009 CMDL CO data to 2013/2014 GMD CO data
This update was included in v11-02c and approved on 21 Sep 2017.
Jenny Fisher wrote:
- Attached is a zip file containing routines to process the GMD (previously CMDL) CO data for use in the benchmarking scripts, as well as the 2013 and 2014 data. I added the 2014 data because there was no ships data for 2013 (or 2012), so this seemed like the best approximation.
- To replace the existing benchmark data with these, you will want to do the following in the 1-year benchmark code:
- Put the 2013data/ and 2014data/ in the data/cmdl/ directory of the 1-year benchmark code
- Change newdata/ to 2013data/ in plot_cmdl_3_models_4_months.pro
- Change Feb09/ to 2013data/ in plot_surface_co_geos_3_models.pro
- Change newdata/ to 2014data/ in plot_ships_3_models_4_months.pro and plot_ships_3_models_co.pro
- Change the line file=pre+name_sta(kk)+'.mn' to file=pre+name_sta(kk)+’.mn.2013' in plot_cmdl_3_models_4_months.pro
- Change the line file=pre+name_sta(kk)+’.mn.2005' to file=pre+name_sta(kk)+’.mn.2013' in plot_surface_co_geos_3_models.pro
- Change the line file=pre+name_sta(kk)+'.mn' to file=pre+name_sta(kk)+’.mn.2014' in plot_ships_3_models_4_months.pro and plot_ships_3_models_co.pro
- As I said before, I think this would be a useful improvement to our benchmarks, rather than using data >10 years old!