Difference between revisions of "GEOS-Chem benchmarking"

From Geos-chem
Jump to: navigation, search
(Update 2005/2009 CMDL CO data to 2013/2014 GMD CO data)
(18 intermediate revisions by 4 users not shown)
Line 1: Line 1:
== Overview ==
+
== Objectives ==
  
The following GEOS-Chem benchmarking procedure was adopted at the 5th International GEOS-Chem Meeting (May 2011).
+
Benchmarking supports the maintenance of GEOS-Chem as a robust state-of-the-science facility with a nimble grass-roots approach and strong version control. Benchmarking has four main objectives:
 +
#Document a consistent GEOS-Chem model configuration, and the expected characteristics of that configuration.
 +
#Support version control through traceability, and by confirming the expected behavior of model developments submitted by the community. 
 +
#Track the evolution of the model over the years.
 +
#Promote scientific transparency of GEOS-Chem.  
  
#Any change to the GEOS-Chem source code will require a dedicated [[#1-month benchmark|1-month benchmark]] with the [[GEOS-Chem_chemistry_mechanisms#Mechanisms_in_GEOS-Chem_v10-01_and_later_versions|standard chemistry mechanism]] and a version update code a, b, c, etc. (such as v10-01a, v10-01b, etc). These letter codes are for internal use only and do not represent a version release. The benchmark results will be posted publicly with an email sent to the [http://acmg.seas.harvard.edu/geos/geos_steering_cmte.html GEOS-Chem Steering Committee].
+
== Procedure ==
#The developer(s) will assess the benchmark results by filling out a form (created by Colette Heald and adapted to a wiki page) for all to see. If the developer has any concerns about the benchmark results they will contact the [[GCST|GEOS-Chem Support Team]], the relevant [http://acmg.seas.harvard.edu/geos/geos_working_groups.html GEOS-Chem Working Group Chairs], and/or [http://acmg.seas.harvard.edu/people/faculty/djj/index.html Model Scientist Daniel Jacob]. GEOS-Chem Steering Committee members are also encouraged to examine and comment on the benchmark results.
+
#Once the developer is satisfied with the changes in the 1-month benchmark, GEOS-Chem Model Scientist Daniel Jacob will promptly review the results and approve the new internal version.
+
#[[#1-year benchmark|1-year benchmarks]] for internal version updates will be conducted only if justifiably requested by the developer or by GEOS-Chem Steering Committee members.
+
#Each new public version release (e.g. GEOS-Chem v10-01) will be subject to a 1-year benchmark to be inspected by the GEOS-Chem Steering Committee before approval. Changes will be bundled so that we will have new versions released every 9 months or so — this has been our rhythm in recent years.
+
  
== List of GEOS-Chem benchmarks ==
+
The GEOS-Chem benchmarking procedure is described below.
  
Links to information about past 1-month and 1-year benchmark simulations can be found on [[GEOS-Chem_versions_under_development#GEOS-Chem_development_history|''GEOS-Chem versions under development'' wiki page]]. For information about 1-year benchmark simulations for GEOS-Chem v8-02-04 and earlier, see [http://acmg.seas.harvard.edu/geos/geos_1yr_plots.html this website].
+
#Any update to the GEOS-Chem source code or run directories will change the [[GEOS-Chem_version_numbering_system#Numeric_versioning_system|GEOS-Chem version number ('''X.Y.Z''')]].
 +
#'''Z''' versions will be released at intervals determined by the [[GCST|GEOS-Chem Support Team (GCST)]] and may include bug fixes or updates that do not impact the full-chemistry simulation.
 +
#Any change impacting the [[GEOS-Chem_chemistry_mechanisms#Mechanisms_in_GEOS-Chem_v10-01_and_later_versions|standard full-chemistry simulation]] will require a '''Y''' version change and a dedicated 1-month benchmark. The benchmark results will be posted on the wiki and an email will be sent to the developer(s) and the [https://geos-chem.seas.harvard.edu/geos-steering-cmte GEOS-Chem Steering Committee (GCSC)].
 +
#The developer(s) and GCSC will assess the benchmark results and review a benchmark assessment form on the wiki. If there are any concerns about the benchmark results, the GCST will be notified and further investigation and/or benchmarking may be required.
 +
#If the update is for a [[GEOS-Chem_chemistry_mechanisms#Specialty_simulations|specialty simulation]] (e.g. CO2, CH4, Hg), then a further benchmark may be conducted by the appropriate Working Group.
 +
#Once the developer is satisfied with the changes in the 1-month benchmark, GEOS-Chem Model Scientist Daniel Jacob will review the results and approve the new internal version.
 +
#1-year full-chemistry and/or transport tracer benchmarks for '''Y''' versions will be conducted only if justifiably requested by the developer or by GEOS-Chem Steering Committee members.
 +
#Each new major version release (i.e. '''X''' version) will be subject to a 1-year benchmark to be inspected by the GEOS-Chem Steering Committee before approval.
  
== 1-month benchmark ==
+
== List of GEOS-Chem benchmarks ==
  
=== Overview ===
+
Links to past 1-month and 1-year benchmark simulations can be found on the [[GEOS-Chem_versions|''GEOS-Chem versions'']] wiki page.
  
{| border=1 cellspacing=0 cellpadding=5
+
== Benchmark output archive ==
  
|-valign="top"
+
Output files and evaluation plots for 1-month and 1-year benchmark simulations are archived at Harvard as summarized below. GEOS-Chem users may utilize these output for comparisons against their own simulations.
!bgcolor="#cccccc"|Dates of run:
+
|July 1, 2013 – August 1, 2013
+
  
|-valign="top"
+
{| border=1 cellpadding=5 cellspacing=0
!bgcolor="#cccccc"|[[GEOS-Chem_vertical_grids|Vertical Resolution]]:
+
|-bgcolor="#CCCCCC"                        
|[[GEOS-FP]] with [[GEOS-Chem_vertical_grids#72-layer_vertical_grid|72 hybrid sigma-pressure levels]]
+
!width="450px"|Directory                 
 +
!width="550px"|Description
  
|-valign="top"
+
|-valign="top"
!bgcolor="#cccccc"|[[GEOS-Chem_horizontal_grids|Horizontal Resolution]]:
+
|<tt>http://ftp.as.harvard.edu/gcgrid/geos-chem/1mo_benchmarks/</tt>  
|[[GEOS-Chem_horizontal_grids#GMAO_4_x_5_grid|4&deg; latitude x 5&deg; longitude]]
+
|Contains the following data from the [[GEOS-Chem_benchmarking#1-month_benchmark|1-month benchmarks]] used to evaluate GEOS-Chem:
 
+
*Restart files
|-valign="top"
+
*Model output
!bgcolor="#cccccc"|[[Species_in_GEOS-Chem#Full-chemistry|Advected species]]:
+
*Log files
|<u>Gas-phase species:</u> NO, O3, PAN, CO, ALK4, ISOP, HNO3, H2O2, ACET, MEK, ALD2, RCHO, MVK, MACR, PMN (isoprene and non-isoprene), PPN, R4N2, PRPE, C3H8, CH2O, C2H6, N2O5, HNO4, MP, MPN, ISOPND, ISOPNB, MOBA, PROPNN, HAC, GLYC, MVKN, MACRN, MAP, NO2, NO3, HNO2, BENZ, TOLU, XYLE, MTPA, LIMO, MTPO, TSOG0-3, ISOG1-3, ASOG1-3, EOH, MGLY, GLYX, ACTA, HPALD, DHDN, ETHLN, HCOOH, IEPOXA, IEPOXB, IEPOXD, ISN1, RIPA, RIPB, RIPD, IMAE, LVOC, ISN1OG, MONITS, MONITU, HONIT, HC187
+
*Input files
 
+
*Evaluation plots
<u>Aerosol species:</u> DMS, SO2, SO4, MSA, NH3, NH4, NIT, BCPO, OCPI, BCPI, OCPI, DST1-4, SALA, SALC, SO4s, NITs, TSOA0-3, ISOA1-3, ASOAN, ASOA1-3, SOAIE, SOAME, SOAGX, SOAMG, LVOCOA, ISN1OA, IONITA, MONITA, INDIOL
+
 
+
<u>Bromine species:</u> Br2, Br, BrO, HOBr, HBr, BrNO2 BrNO3, CHBr3, CH2Br2, CH3Br
+
 
+
<u>Stratospherically-important species:</u> N2O, OCS, CH4, BrCl, HCl, CCl4, CH3Cl, CH3CCl3, CFCX, HCFCX, CFC11, CFC12, HCFC22, H1211, H2402, Cl, ClO, HOCl, ClNO3, ClNO2, ClOO, OClO, Cl2, Cl202, H2O
+
 
+
|-valign="top"
+
!bgcolor="#cccccc"|[[GEOS-Chem_chemistry_mechanisms|Chemical Mechanism]]:
+
|[[FlexChem]] chemical solver<br> using the standard chemistry mechanism ([[NOx-Ox-HC-Aer-Br_chemistry_mechanism|NOx-Ox-HC-Aer-Br]] + [[UCX_chemistry_mechanism|UCX]] + [[Secondary_organic_aerosols#Complex_SOA_scheme|complex SOA]])<br>Chemistry is done in both the troposphere and the stratosphere.
+
 
+
|-valign="top"
+
!bgcolor="#cccccc"|[[FAST-JX_v7.0_photolysis_mechanism|Photolysis Mechanism]]:
+
|[[FAST-JX_v7.0_photolysis_mechanism|FAST-JX v7.0]] (Prather, 2012, Wild et al, 2000).
+
 
+
|-valign="top"
+
!bgcolor="#cccccc"|Operations:
+
|
+
*[[Advection scheme TPCORE|Advection]]
+
*[[ISORROPIA II|Aerosol thermodynamical equilibriium]]
+
*[[Boundary layer mixing]]
+
*[[Chemistry Issues|Chemistry]]
+
*[[Cloud convection]] (with updraft scavenging)
+
*[[Dry deposition]]
+
*[[Emissions overview|Emissions]]
+
*[[Photolysis_mechanism|Photolysis]]
+
*[[Wet deposition]]
+
 
+
|-valign="top"
+
!bgcolor="#cccccc"|[[List_of_diagnostics_for_v11-01|Diagnostic Output]]:
+
|The following diagnostic outputs are archived as monthly means
+
*Sulfate production and loss quantities
+
*Dust aerosol sources
+
*Carbon aerosol sources
+
*Sea salt aerosol sources
+
*Acetone sources
+
*Sulfur sources
+
*Optical Depths
+
*Noontime J-values
+
*Biomass burning emission fluxes
+
*CO sources
+
*Surface Pressure
+
*NOx sources
+
*Biofuel emission fluxes
+
*Anthropogenic emission fluxes
+
*Scavenging loss from moist convection
+
*Scavenging loss from wet deposition
+
*Concentrations of chemically produced OH and HO2
+
*Dry deposition fluxes and velocities
+
*Tracer concentrations
+
*Biogenic emission fluxes
+
*Tropopause height
+
*DAO 3-D met fields
+
*DAO 2-D met fields
+
*Air masses and grid box heights
+
*Surface area
+
*Lifetime of Methylchloroform (CH3CCl3)
+
 
+
|-valign="top"
+
!bgcolor="#cccccc"|Benchmark Plots and Summaries:
+
|The following plots, budgets, and totals are created from the oxidant-aerosol simulation:
+
*Budget of Ox and CO
+
*Mean OH concentration
+
*Methyl Chloroform Lifetime (w/r/t loss by tropospheric OH)
+
*Aerosol optical depth maps (this benchmark)
+
*Aerosol optical depth differences (this benchmark - previous benchmark)
+
*Concentration maps (this benchmark) of tracers + OH + HO2 at the surface and at 500 hPa
+
*Difference maps (this benchmark - previous benchmark) of tracers + OH + HO2 at the surface and at 500 hPa
+
*Summary of emission totals (this benchmark vs. previous benchmark)
+
*Emissions maps (this benchmark) for all emitting tracers
+
*Emissions difference maps (this benchmark - previous benchmark) for all emitting tracers
+
*Emissions ratio maps (this benchmark / previous benchmark) for all emitting tracers
+
*Frequency distribution histogram of ratios (this benchmark vs previous benchmark) for tracers, OH, and optical depths
+
*J-value maps (this benchmark)
+
*J-value difference maps (this benchmark - previous benchmark)
+
*J-value ratio maps (this benchmark / previous benchmark)
+
*Ratio maps (this benchmark / previous benchmark) of tracers + OH + HO2 at the surface and at 500 hPa
+
*Zonal mean concentration maps (this benchmark) of tracers + OH + HO2
+
*Zonal mean difference maps (this benchmark - previous benchmark) of tracers + OH + HO2
+
  
 +
|-valign="top" 
 +
|<tt>http://ftp.as.harvard.edu/gcgrid/geos-chem/1yr_benchmarks/</tt>                       
 +
|Contains the following data from the [[GEOS-Chem_benchmarking#1-year_benchmark|1-year benchmarks]] used to evaluate GEOS-Chem:
 +
*Restart files
 +
*Model output
 +
*Log files
 +
*Input files
 +
*Evaluation plots
 
|}
 
|}
  
=== Plotting routines ===
 
  
''<span style="color:darkorange">NOTE: The 1-month benchmark plotting routines are currently written in IDL. We are planning on moving these routines to Python for the [[GEOS-Chem v11-02]] release as part of the [[Python_code_for_GEOS-Chem#GCPy|GCPy package]].</span>''
+
== 1-month benchmark plotting routines ==
  
The 1-month benchmark plotting routines are included with the [http://acmg.seas.harvard.edu/gamap/doc/ GAMAP]. For more information, see the [http://acmg.seas.harvard.edu/gamap/doc/by_category/Benchmarking.html this page in the GAMAP manual].
+
The 1-month benchmark plotting routines are included with [https://github.com/geoschem/gcpy GCPy], a Python took kit available for GEOS-Chem.
  
== 1-year benchmark ==
+
== 1-year benchmark plotting routines ==
  
=== Overview ===
+
The 1-year benchmark plotting routines are currently written in IDL and can be downloaded from Bitbucket via Git using:
 
+
{| border=1 cellspacing=0 cellpadding=5
+
 
+
|-valign="top"
+
!bgcolor="#cccccc"|Spin-up:
+
|January 1, 2013 &ndash; January 1, 2014
+
 
+
|-valign="top"
+
!bgcolor="#cccccc"|Dates of Run:
+
|January 1, 2013 &ndash; January 1, 2014
+
 
+
|-valign="top"
+
!bgcolor="#cccccc"|[[GEOS-Chem_vertical_grids|Vertical Resolution]]:
+
|[[GEOS-FP]] with [[GEOS-Chem_vertical_grids#72-layer_vertical_grid|72 hybrid sigma-pressure levels]]
+
 
+
|-valign="top"
+
!bgcolor="#cccccc"|[[GEOS-Chem_horizontal_grids|Horizontal Resolution]]:
+
|[[GEOS-Chem_horizontal_grids#GMAO_4_x_5_grid|4&deg; latitude x 5&deg; longitude]]
+
 
+
|-valign="top"
+
!bgcolor="#cccccc"|[[Species_in_GEOS-Chem#Full-chemistry|Advected species]]:
+
|<u>Gas-phase species:</u> NO, O3, PAN, CO, ALK4, ISOP, HNO3, H2O2, ACET, MEK, ALD2, RCHO, MVK, MACR, PMN (isoprene and non-isoprene), PPN, R4N2, PRPE, C3H8, CH2O, C2H6, N2O5, HNO4, MP, MPN, ISOPND, ISOPNB, MOBA, PROPNN, HAC, GLYC, MVKN, MACRN, MAP, NO2, NO3, HNO2, BENZ, TOLU, XYLE, MTPA, LIMO, MTPO, TSOG0-3, ISOG1-3, ASOG1-3, EOH, MGLY, GLYX, ACTA, HPALD, DHDN, ETHLN, HCOOH, IEPOXA, IEPOXB, IEPOXD, ISN1, RIPA, RIPB, RIPD, IMAE, LVOC, ISN1OG, MONITS, MONITU, HONIT, HC187
+
 
+
<u>Aerosol species:</u> DMS, SO2, SO4, MSA, NH3, NH4, NIT, BCPO, OCPI, BCPI, OCPI, DST1-4, SALA, SALC, SO4s, NITs, TSOA0-3, ISOA1-3, ASOAN, ASOA1-3, SOAIE, SOAME, SOAGX, SOAMG, LVOCOA, ISN1OA, IONITA, MONITA, INDIOL
+
 
+
<u>Bromine species:</u> Br2, Br, BrO, HOBr, HBr, BrNO2 BrNO3, CHBr3, CH2Br2, CH3Br
+
 
+
<u>Stratospherically-important species:</u> N2O, OCS, CH4, BrCl, HCl, CCl4, CH3Cl, CH3CCl3, CFCX, HCFCX, CFC11, CFC12, HCFC22, H1211, H2402, Cl, ClO, HOCl, ClNO3, ClNO2, ClOO, OClO, Cl2, Cl202, H2O
+
 
+
|-valign="top"
+
!bgcolor="#cccccc"|[[GEOS-Chem_chemistry_mechanisms|Chemical Mechanism]]:
+
|[[FlexChem]] chemical solver<br> using the standard chemistry mechanism ([[NOx-Ox-HC-Aer-Br_chemistry_mechanism|NOx-Ox-HC-Aer-Br]] + [[UCX_chemistry_mechanism|UCX]] + [[Secondary_organic_aerosols#Complex_SOA_scheme|complex SOA]])<br>Chemistry is done in both the troposphere and the stratosphere.
+
 
+
|-valign="top"
+
!bgcolor="#cccccc"|[[FAST-JX_v7.0_photolysis_mechanism|Photolysis Mechanism]]:
+
|[[FAST-JX_v7.0_photolysis_mechanism|FAST-JX v7.0]] (Prather, 2012, Wild et al, 2000).
+
 
+
|-valign="top"
+
!bgcolor="#cccccc"|Operations:
+
|
+
*[[Advection scheme TPCORE|Advection]]
+
*[[ISORROPIA II|Aerosol thermodynamical equilibriium]]
+
*[[Boundary layer mixing]]
+
*[[Chemistry Issues|Chemistry]]
+
*[[Cloud convection]] (with updraft scavenging)
+
*[[Dry deposition]]
+
*[[Emissions overview|Emissions]]
+
*[[Photolysis_mechanism|Photolysis]]
+
*[[Wet deposition]]
+
 
+
|-valign="top"
+
!bgcolor="#cccccc"|[[List_of_diagnostics_for_v11-01|Diagnostic Output]]:
+
|All diagnostics above are saved as monthly means.
+
*Sulfate production and loss quantities
+
*Dust aerosol sources
+
*Carbon aerosol sources
+
*Sea salt aerosol sources
+
*Acetone sources
+
*Sulfur sources
+
*Optical Depths
+
*Noontime J-values
+
*Transport fluxes (N/S, E/W, up/down)
+
*Biomass burning emission fluxes
+
*CO sources
+
*Surface Pressure
+
*NOx sources
+
*Biofuel emission fluxes
+
*Anthropogenic emission fluxes
+
*Scavenging loss from moist convection
+
*Scavenging loss from wet deposition
+
*Concentrations of chemically produced OH and HO2
+
*Dry deposition fluxes and velocities
+
*Tracer concentrations
+
*Biogenic emission fluxes
+
*Tropopause height
+
*DAO 3-D met fields
+
*DAO 2-D met fields
+
*Air masses and grid box heights
+
*Surface area
+
*Lifetime of Methylchloroform (CH3CCl3)
+
*Satellite timeseries of BrO, cloud fraction, and cloud top height
+
 
+
|-valign="top"
+
!bgcolor="#cccccc"|Benchmark Plots:
+
|The following plots compare advected species from 3 different benchmark simulations (versions 1-3, where version 3 is the current benchmark version). Plots are created for January, April, July, and October.
+
*Aerosol optical depth maps (version 1, version 2, and version 3)
+
*Emission maps (version 1, version 2, and version 3)
+
*Emisison difference maps (version 3 - version 1, version 3 - version 2)
+
*Emission ratio maps (version 3 / version 1, version 3 / version 2)
+
*J-value maps (version 1, version 2, and version 3)
+
*J-value difference maps (version 3 - version 1, version 3 - version 2)
+
*J-value ratio maps (version 3 / version 1, version 3 / version 2)
+
*Tracer difference maps at the surface and 500 hPa (version 3 - version 1, version 3 - version 2)
+
*Tracer ratio maps at the surface and 500 hPa (version 3 / version 1, version 3 / version 2)
+
*Tracer difference profiles along longitude slices (15S, 42N) (version 3 - version 1, version 3 - version 2)
+
 
+
The following plots show data from 3 benchmark simulations side-by-side with observations. GEOS-Chem output is plotted in red for version 1, in green for version 2, and in blue for version 3 (current benchmark version)
+
*BrO seasonal columns vs. observations
+
*C2H6 vertical profiles vs. observations from various aircraft campaigns
+
*C3H8 vertical profiles vs. observations from various aircraft campaigns
+
*CO seasonal cycle at the surface vs. MOZAIC observations
+
*CO vertical profiles vs. MOZAIC observations
+
*CO vertical profiles vs. various sonde observations
+
*H2O2 vertical profiles vs. observations from various aircraft campaigns
+
*IMPROVE difference maps (benchmark simulation output vs. surface sites)
+
*IMPROVE scatter plots (benchmark simulation output vs. surface sites)
+
*HNO3 vertical profiles vs. observations from various aircraft campaigns
+
*NO vertical profiles vs. observations from various aircraft campaigns
+
*O3 seasonal cycle @ the surface vs. MOZAIC observations
+
*O3 seasonal cycle @ 300, 500, and 800 hPa vs. MOZAIC observations
+
*O3 seasonal cycle @ 300, 500, and 800 hPa vs. sonde observations
+
*O3 seasonal cycle @ 150, 300, 500,and 800 hPa vs. sonde observations
+
*O3 vertical profiles vs. observations from various aircraft campaigns
+
*O3 vertical profiles vs. observations from various aircraft campaigns
+
*O3 vertical profiles vs. various sonde observations
+
*O3 vertical profiles vs. MOZAIC observations
+
*PAN vertical profiles vs. observations from various aircraft campaigns
+
*PM2.5 difference maps (benchmark simulation output vs. surface sites)
+
*PM2.5 scatter plots (benchmark simulation output vs. surface sites)
+
 
+
|}
+
 
+
=== Plotting routines ===
+
 
+
''<span style="color:darkorange">NOTE: The 1-month benchmark plotting routines are currently written in IDL. We are planning on moving these routines to Python for the [[GEOS-Chem v11-02]] release as part of the [[Python_code_for_GEOS-Chem#GCPy|GCPy package]].</span>''
+
 
+
The 1-year benchmark plotting routines can be downloaded from Bitbucket via Git using:
+
  
 
   git clone https://bitbucket.org/gcst/gc_1yr_benchmark
 
   git clone https://bitbucket.org/gcst/gc_1yr_benchmark
  
The data files needed for the plots are included in the repository and are discussed in more detail [[#Observations|below]].
+
The data files needed for the plots are included in the repository and are summarized in the table below.
 
+
=== Observations ===
+
 
+
Here we provide an overview of the data used in the 1-year benchmark plots.
+
  
 
{| border=1 cellspacing=0 cellpadding=5
 
{| border=1 cellspacing=0 cellpadding=5
Line 342: Line 146:
 
|}
 
|}
  
--[[User:Melissa Payer|Melissa Sulprizio]] ([[User talk:Melissa Payer|talk]]) 14:26, 16 August 2017 (UTC)
+
=== Update 2005/2009 CMDL CO data to 2013/2014 GMD CO data ===
 
+
==== Update 2005/2009 CMDL CO data to 2013/2014 GMD CO data ====
+
  
 
<span style="color:green">'''''This update was included in [[GEOS-Chem v11-02#v11-02c|v11-02c]] and approved on 21 Sep 2017.'''''</span>
 
<span style="color:green">'''''This update was included in [[GEOS-Chem v11-02#v11-02c|v11-02c]] and approved on 21 Sep 2017.'''''</span>
Line 363: Line 165:
  
 
:As I said before, I think this would be a useful improvement to our benchmarks, rather than using data >10 years old!
 
:As I said before, I think this would be a useful improvement to our benchmarks, rather than using data >10 years old!
 
--[[User:Melissa Payer|Melissa Sulprizio]] ([[User talk:Melissa Payer|talk]]) 22:10, 15 August 2017 (UTC)
 

Revision as of 21:13, 20 September 2022

Objectives

Benchmarking supports the maintenance of GEOS-Chem as a robust state-of-the-science facility with a nimble grass-roots approach and strong version control. Benchmarking has four main objectives:

  1. Document a consistent GEOS-Chem model configuration, and the expected characteristics of that configuration.
  2. Support version control through traceability, and by confirming the expected behavior of model developments submitted by the community.
  3. Track the evolution of the model over the years.
  4. Promote scientific transparency of GEOS-Chem.

Procedure

The GEOS-Chem benchmarking procedure is described below.

  1. Any update to the GEOS-Chem source code or run directories will change the GEOS-Chem version number (X.Y.Z).
  2. Z versions will be released at intervals determined by the GEOS-Chem Support Team (GCST) and may include bug fixes or updates that do not impact the full-chemistry simulation.
  3. Any change impacting the standard full-chemistry simulation will require a Y version change and a dedicated 1-month benchmark. The benchmark results will be posted on the wiki and an email will be sent to the developer(s) and the GEOS-Chem Steering Committee (GCSC).
  4. The developer(s) and GCSC will assess the benchmark results and review a benchmark assessment form on the wiki. If there are any concerns about the benchmark results, the GCST will be notified and further investigation and/or benchmarking may be required.
  5. If the update is for a specialty simulation (e.g. CO2, CH4, Hg), then a further benchmark may be conducted by the appropriate Working Group.
  6. Once the developer is satisfied with the changes in the 1-month benchmark, GEOS-Chem Model Scientist Daniel Jacob will review the results and approve the new internal version.
  7. 1-year full-chemistry and/or transport tracer benchmarks for Y versions will be conducted only if justifiably requested by the developer or by GEOS-Chem Steering Committee members.
  8. Each new major version release (i.e. X version) will be subject to a 1-year benchmark to be inspected by the GEOS-Chem Steering Committee before approval.

List of GEOS-Chem benchmarks

Links to past 1-month and 1-year benchmark simulations can be found on the GEOS-Chem versions wiki page.

Benchmark output archive

Output files and evaluation plots for 1-month and 1-year benchmark simulations are archived at Harvard as summarized below. GEOS-Chem users may utilize these output for comparisons against their own simulations.

Directory Description
http://ftp.as.harvard.edu/gcgrid/geos-chem/1mo_benchmarks/ Contains the following data from the 1-month benchmarks used to evaluate GEOS-Chem:
  • Restart files
  • Model output
  • Log files
  • Input files
  • Evaluation plots
http://ftp.as.harvard.edu/gcgrid/geos-chem/1yr_benchmarks/ Contains the following data from the 1-year benchmarks used to evaluate GEOS-Chem:
  • Restart files
  • Model output
  • Log files
  • Input files
  • Evaluation plots


1-month benchmark plotting routines

The 1-month benchmark plotting routines are included with GCPy, a Python took kit available for GEOS-Chem.

1-year benchmark plotting routines

The 1-year benchmark plotting routines are currently written in IDL and can be downloaded from Bitbucket via Git using:

 git clone https://bitbucket.org/gcst/gc_1yr_benchmark

The data files needed for the plots are included in the repository and are summarized in the table below.

Directory Data year Species Data source Provided by
BrO/ 2007-2008 BrO GOME-2 Justin Parrella
(Data used to reproduce Figure 5 of Parrella et al. (2012))
cmdl/ 2005, 2009
New data available
CO GMD (formerly CMDL) Jennifer Logan and Inna Megretskaia
co.prof.for.gmi/ 2001-2008
(varies by station)
CO MOZAIC Jennifer Logan and Inna Megretskaia
eval/aircraft/
NOTE: Other directories in eval/ aren't currently used
1995-2003
(varies by aircraft campaign)
C2H6, C3H8, CO, H2O2, HNO3, NO, O3, PAN

(Not currently used: ACET, ALK4, C2H2, CH2Br2, CH2BrCl, CH3Br, CH3OH, CH4, CHBr2Cl, CHBrCl2, CO2, ETHE, PRPE)

See README Jennifer Logan and Inna Megretskaia
netCDF/ N/A C2H6, C3H8, CH3I, CO, H2O2, HNO3, NO, O3, OH, PAN, Rn Various
(contains information about station locations)
Jennifer Logan and Inna Megretskaia
PAN/ Varies PAN ARCTAS-A, ARCTAS-B, ARCPAC, INTEX-B, ITCT-2K2 Emily Fischer
(Data used to reproduce Figure S1 of Fischer et al. (2014) Supplement)
pm25_data/ 2005 PM2.5 IMPROVE Colette Heald
sondes.for.gmi/ 1990-2008
(varies by station)
O3 Jennifer Logan and Inna Megretskaia
strat/ 2001-2010 NOy, O3 OSIRIS Dylan Jones
surface_ozone/ O3 CMDL Jennifer Logan and Inna Megretskaia

Update 2005/2009 CMDL CO data to 2013/2014 GMD CO data

This update was included in v11-02c and approved on 21 Sep 2017.

Jenny Fisher wrote:

Attached is a zip file containing routines to process the GMD (previously CMDL) CO data for use in the benchmarking scripts, as well as the 2013 and 2014 data. I added the 2014 data because there was no ships data for 2013 (or 2012), so this seemed like the best approximation.
To replace the existing benchmark data with these, you will want to do the following in the 1-year benchmark code:
  1. Put the 2013data/ and 2014data/ in the data/cmdl/ directory of the 1-year benchmark code
  2. Change newdata/ to 2013data/ in plot_cmdl_3_models_4_months.pro
  3. Change Feb09/ to 2013data/ in plot_surface_co_geos_3_models.pro
  4. Change newdata/ to 2014data/ in plot_ships_3_models_4_months.pro and plot_ships_3_models_co.pro
  5. Change the line file=pre+name_sta(kk)+'.mn' to file=pre+name_sta(kk)+’.mn.2013' in plot_cmdl_3_models_4_months.pro
  6. Change the line file=pre+name_sta(kk)+’.mn.2005' to file=pre+name_sta(kk)+’.mn.2013' in plot_surface_co_geos_3_models.pro
  7. Change the line file=pre+name_sta(kk)+'.mn' to file=pre+name_sta(kk)+’.mn.2014' in plot_ships_3_models_4_months.pro and plot_ships_3_models_co.pro
As I said before, I think this would be a useful improvement to our benchmarks, rather than using data >10 years old!