Difference between revisions of "GEOS-Chem benchmarking"
(→1-year benchmark) |
(→Update 2005/2009 CMDL CO data to 2013/2014 GMD CO data) |
||
(34 intermediate revisions by 4 users not shown) | |||
Line 1: | Line 1: | ||
− | == | + | == Objectives == |
− | + | Benchmarking supports the maintenance of GEOS-Chem as a robust state-of-the-science facility with a nimble grass-roots approach and strong version control. Benchmarking has four main objectives: | |
+ | #Document a consistent GEOS-Chem model configuration, and the expected characteristics of that configuration. | ||
+ | #Support version control through traceability, and by confirming the expected behavior of model developments submitted by the community. | ||
+ | #Track the evolution of the model over the years. | ||
+ | #Promote scientific transparency of GEOS-Chem. | ||
− | + | == Procedure == | |
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | The GEOS-Chem benchmarking procedure is described below. | |
− | + | #Any update to the GEOS-Chem source code or run directories will change the [[GEOS-Chem_version_numbering_system#Numeric_versioning_system|GEOS-Chem version number ('''X.Y.Z''')]]. | |
+ | #'''Z''' versions will be released at intervals determined by the [[GCST|GEOS-Chem Support Team (GCST)]] and may include bug fixes or updates that do not impact the full-chemistry simulation. | ||
+ | #Any change impacting the [[GEOS-Chem_chemistry_mechanisms#Mechanisms_in_GEOS-Chem_v10-01_and_later_versions|standard full-chemistry simulation]] will require a '''Y''' version change and a dedicated 1-month benchmark. The benchmark results will be posted on the wiki and an email will be sent to the developer(s) and the [https://geos-chem.seas.harvard.edu/geos-steering-cmte GEOS-Chem Steering Committee (GCSC)]. | ||
+ | #The developer(s) and GCSC will assess the benchmark results and review a benchmark assessment form on the wiki. If there are any concerns about the benchmark results, the GCST will be notified and further investigation and/or benchmarking may be required. | ||
+ | #If the update is for a [[GEOS-Chem_chemistry_mechanisms#Specialty_simulations|specialty simulation]] (e.g. CO2, CH4, Hg), then a further benchmark may be conducted by the appropriate Working Group. | ||
+ | #Once the developer is satisfied with the changes in the 1-month benchmark, GEOS-Chem Model Scientist Daniel Jacob will review the results and approve the new internal version. | ||
+ | #1-year full-chemistry and/or transport tracer benchmarks for '''Y''' versions will be conducted only if justifiably requested by the developer or by GEOS-Chem Steering Committee members. | ||
+ | #Each new major version release (i.e. '''X''' version) will be subject to a 1-year benchmark to be inspected by the GEOS-Chem Steering Committee before approval. | ||
− | + | == List of GEOS-Chem benchmarks == | |
− | + | Links to past 1-month and 1-year benchmark simulations can be found on the [[GEOS-Chem_versions|''GEOS-Chem versions'']] wiki page. | |
− | + | ||
− | + | ||
− | + | == Benchmark output archive == | |
− | + | ||
− | + | ||
− | + | Output files and evaluation plots for 1-month and 1-year benchmark simulations are archived at Harvard as summarized below. GEOS-Chem users may utilize these output for comparisons against their own simulations. | |
− | + | ||
− | + | ||
− | |- | + | {| border=1 cellpadding=5 cellspacing=0 |
− | ! | + | |-bgcolor="#CCCCCC" |
− | + | !width="450px"|Directory | |
− | + | !width="550px"|Description | |
− | < | + | |-valign="top" |
− | + | |<tt>http://ftp.as.harvard.edu/gcgrid/geos-chem/1mo_benchmarks/</tt> | |
+ | |Contains the following data from the [[GEOS-Chem_benchmarking#1-month_benchmark|1-month benchmarks]] used to evaluate GEOS-Chem: | ||
+ | *Restart files | ||
+ | *Model output | ||
+ | *Log files | ||
+ | *Input files | ||
+ | *Evaluation plots | ||
− | < | + | |-valign="top" |
− | + | |<tt>http://ftp.as.harvard.edu/gcgrid/geos-chem/1yr_benchmarks/</tt> | |
+ | |Contains the following data from the [[GEOS-Chem_benchmarking#1-year_benchmark|1-year benchmarks]] used to evaluate GEOS-Chem: | ||
+ | *Restart files | ||
+ | *Model output | ||
+ | *Log files | ||
+ | *Input files | ||
+ | *Evaluation plots | ||
+ | |} | ||
− | |||
− | |||
− | + | == 1-month benchmark plotting routines == | |
− | + | ||
− | + | ||
− | + | The 1-month benchmark plotting routines are included with [https://github.com/geoschem/gcpy GCPy], a Python took kit available for GEOS-Chem. | |
− | + | ||
− | + | ||
− | + | == 1-year benchmark plotting routines == | |
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | The 1-year benchmark plotting routines are currently written in IDL and can be downloaded from Bitbucket via Git using: | |
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | git clone https://bitbucket.org/gcst/gc_1yr_benchmark | |
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | The data files needed for the plots are included in the repository and are summarized in the table below. | |
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
{| border=1 cellspacing=0 cellpadding=5 | {| border=1 cellspacing=0 cellpadding=5 | ||
+ | |-valign="top" bgcolor="#cccccc" | ||
+ | !Directory | ||
+ | !Data year | ||
+ | !Species | ||
+ | !Data source | ||
+ | !Provided by | ||
|-valign="top" | |-valign="top" | ||
− | + | |<tt>BrO/</tt> | |
− | | | + | |2007-2008 |
+ | |BrO | ||
+ | |[https://www.eumetsat.int/website/home/Satellites/CurrentSatellites/Metop/MetopDesign/GOME2/index.html GOME-2] | ||
+ | |Justin Parrella<br>(Data used to reproduce Figure 5 of [http://acmg.seas.harvard.edu/publications/2012/parrella2012.pdf Parrella et al. (2012)]) | ||
|-valign="top" | |-valign="top" | ||
− | + | |<tt>cmdl/</tt> | |
− | | | + | |2005, 2009<br>[[#Update 2005/2009 CMDL CO data to 2013/2014 GMD CO data|New data available]] |
+ | |CO | ||
+ | |[https://www.esrl.noaa.gov/gmd/ GMD] (formerly CMDL) | ||
+ | |Jennifer Logan and Inna Megretskaia | ||
|-valign="top" | |-valign="top" | ||
− | + | |<tt>co.prof.for.gmi/</tt> | |
− | |[ | + | |2001-2008<br>(varies by station) |
+ | |CO | ||
+ | |[http://iagos.sedoo.fr/ MOZAIC] | ||
+ | |Jennifer Logan and Inna Megretskaia | ||
|-valign="top" | |-valign="top" | ||
− | + | |<tt>eval/aircraft/</tt><br>''NOTE: Other directories in <tt>eval/</tt> aren't currently used'' | |
− | |[ | + | |1995-2003<br>(varies by aircraft campaign) |
+ | |C2H6, C3H8, CO, H2O2, HNO3, NO, O3, PAN<br> | ||
+ | ''(Not currently used: ACET, ALK4, C2H2, CH2Br2, CH2BrCl, CH3Br, CH3OH, CH4, CHBr2Cl, CHBrCl2, CO2, ETHE, PRPE)'' | ||
+ | |See [https://bitbucket.org/gcst/gc_1yr_benchmark/src/6495a17e5c0004f8290d09f141d585c30ce480ef/IDL/data/eval/aircraft/data/README.aircraft.data?at=master&fileviewer=file-view-default README] | ||
+ | |Jennifer Logan and Inna Megretskaia | ||
|-valign="top" | |-valign="top" | ||
− | + | |<tt>netCDF/</tt> | |
− | |< | + | |N/A |
− | + | |C2H6, C3H8, CH3I, CO, H2O2, HNO3, NO, O3, OH, PAN, Rn | |
− | + | |Various<br>(contains information about station locations) | |
− | < | + | |Jennifer Logan and Inna Megretskaia |
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
|-valign="top" | |-valign="top" | ||
− | + | |<tt>PAN/</tt> | |
− | | | + | |Varies |
+ | |PAN | ||
+ | |ARCTAS-A, ARCTAS-B, ARCPAC, INTEX-B, ITCT-2K2 | ||
+ | |Emily Fischer<br>(Data used to reproduce Figure S1 of [https://www.atmos-chem-phys.net/14/2679/2014/acp-14-2679-2014-supplement.pdf Fischer et al. (2014) Supplement]) | ||
|-valign="top" | |-valign="top" | ||
− | + | |<tt>pm25_data/</tt> | |
− | |[ | + | |2005 |
+ | |[[Particulate_matter_in_GEOS-Chem|PM2.5]] | ||
+ | |[http://vista.cira.colostate.edu/improve/ IMPROVE] | ||
+ | |Colette Heald | ||
|-valign="top" | |-valign="top" | ||
− | + | |<tt>sondes.for.gmi/</tt> | |
+ | |1990-2008<br>(varies by station) | ||
+ | |O3 | ||
| | | | ||
− | + | |Jennifer Logan and Inna Megretskaia | |
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
|-valign="top" | |-valign="top" | ||
− | + | |<tt>strat/</tt> | |
− | | | + | |2001-2010 |
− | + | |NOy, O3 | |
− | + | |[http://www.asc-csa.gc.ca/eng/satellites/odin.asp OSIRIS] | |
− | + | |Dylan Jones | |
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
|-valign="top" | |-valign="top" | ||
− | + | |<tt>surface_ozone/</tt> | |
− | + | | | |
− | + | |O3 | |
− | + | |[https://www.esrl.noaa.gov/gmd/ CMDL] | |
− | + | |Jennifer Logan and Inna Megretskaia | |
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
− | + | ||
|} | |} | ||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− |
Revision as of 14:35, 18 October 2023
Contents
Objectives
Benchmarking supports the maintenance of GEOS-Chem as a robust state-of-the-science facility with a nimble grass-roots approach and strong version control. Benchmarking has four main objectives:
- Document a consistent GEOS-Chem model configuration, and the expected characteristics of that configuration.
- Support version control through traceability, and by confirming the expected behavior of model developments submitted by the community.
- Track the evolution of the model over the years.
- Promote scientific transparency of GEOS-Chem.
Procedure
The GEOS-Chem benchmarking procedure is described below.
- Any update to the GEOS-Chem source code or run directories will change the GEOS-Chem version number (X.Y.Z).
- Z versions will be released at intervals determined by the GEOS-Chem Support Team (GCST) and may include bug fixes or updates that do not impact the full-chemistry simulation.
- Any change impacting the standard full-chemistry simulation will require a Y version change and a dedicated 1-month benchmark. The benchmark results will be posted on the wiki and an email will be sent to the developer(s) and the GEOS-Chem Steering Committee (GCSC).
- The developer(s) and GCSC will assess the benchmark results and review a benchmark assessment form on the wiki. If there are any concerns about the benchmark results, the GCST will be notified and further investigation and/or benchmarking may be required.
- If the update is for a specialty simulation (e.g. CO2, CH4, Hg), then a further benchmark may be conducted by the appropriate Working Group.
- Once the developer is satisfied with the changes in the 1-month benchmark, GEOS-Chem Model Scientist Daniel Jacob will review the results and approve the new internal version.
- 1-year full-chemistry and/or transport tracer benchmarks for Y versions will be conducted only if justifiably requested by the developer or by GEOS-Chem Steering Committee members.
- Each new major version release (i.e. X version) will be subject to a 1-year benchmark to be inspected by the GEOS-Chem Steering Committee before approval.
List of GEOS-Chem benchmarks
Links to past 1-month and 1-year benchmark simulations can be found on the GEOS-Chem versions wiki page.
Benchmark output archive
Output files and evaluation plots for 1-month and 1-year benchmark simulations are archived at Harvard as summarized below. GEOS-Chem users may utilize these output for comparisons against their own simulations.
Directory | Description |
---|---|
http://ftp.as.harvard.edu/gcgrid/geos-chem/1mo_benchmarks/ | Contains the following data from the 1-month benchmarks used to evaluate GEOS-Chem:
|
http://ftp.as.harvard.edu/gcgrid/geos-chem/1yr_benchmarks/ | Contains the following data from the 1-year benchmarks used to evaluate GEOS-Chem:
|
1-month benchmark plotting routines
The 1-month benchmark plotting routines are included with GCPy, a Python took kit available for GEOS-Chem.
1-year benchmark plotting routines
The 1-year benchmark plotting routines are currently written in IDL and can be downloaded from Bitbucket via Git using:
git clone https://bitbucket.org/gcst/gc_1yr_benchmark
The data files needed for the plots are included in the repository and are summarized in the table below.
Directory | Data year | Species | Data source | Provided by |
---|---|---|---|---|
BrO/ | 2007-2008 | BrO | GOME-2 | Justin Parrella (Data used to reproduce Figure 5 of Parrella et al. (2012)) |
cmdl/ | 2005, 2009 New data available |
CO | GMD (formerly CMDL) | Jennifer Logan and Inna Megretskaia |
co.prof.for.gmi/ | 2001-2008 (varies by station) |
CO | MOZAIC | Jennifer Logan and Inna Megretskaia |
eval/aircraft/ NOTE: Other directories in eval/ aren't currently used |
1995-2003 (varies by aircraft campaign) |
C2H6, C3H8, CO, H2O2, HNO3, NO, O3, PAN (Not currently used: ACET, ALK4, C2H2, CH2Br2, CH2BrCl, CH3Br, CH3OH, CH4, CHBr2Cl, CHBrCl2, CO2, ETHE, PRPE) |
See README | Jennifer Logan and Inna Megretskaia |
netCDF/ | N/A | C2H6, C3H8, CH3I, CO, H2O2, HNO3, NO, O3, OH, PAN, Rn | Various (contains information about station locations) |
Jennifer Logan and Inna Megretskaia |
PAN/ | Varies | PAN | ARCTAS-A, ARCTAS-B, ARCPAC, INTEX-B, ITCT-2K2 | Emily Fischer (Data used to reproduce Figure S1 of Fischer et al. (2014) Supplement) |
pm25_data/ | 2005 | PM2.5 | IMPROVE | Colette Heald |
sondes.for.gmi/ | 1990-2008 (varies by station) |
O3 | Jennifer Logan and Inna Megretskaia | |
strat/ | 2001-2010 | NOy, O3 | OSIRIS | Dylan Jones |
surface_ozone/ | O3 | CMDL | Jennifer Logan and Inna Megretskaia |