Difference between revisions of "GEOS-Chem benchmarking"

From Geos-chem
Jump to: navigation, search
(1-hour benchmarks)
(1-hour benchmarks)
Line 13: Line 13:
 
=== 1-hour benchmarks ===
 
=== 1-hour benchmarks ===
  
1-hour benchmarks   They are triggered when:
+
1-hour benchmarks are automatically triggered when:
  
 
#A commit is pushed to any development branch<sup>1</sup> in the [https://github.com/geoschem/GCClassic <tt>geoschem/GCClassic</tt> superproject repository]
 
#A commit is pushed to any development branch<sup>1</sup> in the [https://github.com/geoschem/GCClassic <tt>geoschem/GCClassic</tt> superproject repository]
Line 22: Line 22:
 
The 1-hour benchmarks are important "sanity checks" and are often used to verify zero difference between two successive commits
 
The 1-hour benchmarks are important "sanity checks" and are often used to verify zero difference between two successive commits
  
1. Development branches are '''dev/X.Y.Z''' or '''dev/no-diff-to-benchmark'''
+
NOTES:
 +
# Development branches are '''dev/X.Y.Z''' or '''dev/no-diff-to-benchmark'''.
  
 
=== 1-month benchmarks ===
 
=== 1-month benchmarks ===

Revision as of 18:00, 22 May 2024

Objectives

Benchmarking supports the maintenance of GEOS-Chem as a robust state-of-the-science facility with a nimble grass-roots approach and strong version control. Benchmarking has four main objectives:

  1. Document a consistent GEOS-Chem model configuration, and the expected characteristics of that configuration.
  2. Support version control through traceability, and by confirming the expected behavior of model developments submitted by the community.
  3. Track the evolution of the model over the years.
  4. Promote scientific transparency of GEOS-Chem.

Procedure

The GEOS-Chem benchmarking procedure is described below. GEOS-Chem uses semantic versioning (i.e. X.Y.Z version labels).

1-hour benchmarks

1-hour benchmarks are automatically triggered when:

  1. A commit is pushed to any development branch1 in the geoschem/GCClassic superproject repository
  2. A commit is puhsed to any development branch1 in the geoschem/GCHP "superproject" repository.

Evaluation tables are posted to gc-dashboard.org upon successful completion of each 1-hour benchmark simulation. The evaluation tables include information on OH metrics, emissions totals, global mass, and a summary table.

The 1-hour benchmarks are important "sanity checks" and are often used to verify zero difference between two successive commits

NOTES:

  1. Development branches are dev/X.Y.Z or dev/no-diff-to-benchmark.

1-month benchmarks

  1. A 1-month benchmark will run automatically on the AWS cloud whenever a tag is pushed to a development branch in the geoschem/GCClassic or geoschem/GCHP superproject repositories.
    • Tags are
    • Development branches are dev/X.Y.Z or dev/no-diff-to-benchmark

etc

  1. Any update to the GEOS-Chem source code or run directories will change the GEOS-Chem version number (X.Y.Z).
  2. Z versions will be released at intervals determined by the GEOS-Chem Support Team (GCST) and may include bug fixes or updates that do not impact the full-chemistry simulation.
  3. Any change impacting the standard full-chemistry simulation will require a Y version change and a dedicated 1-month benchmark. The benchmark results will be posted on the wiki and an email will be sent to the developer(s) and the GEOS-Chem Steering Committee (GCSC).
  4. The developer(s) and GCSC will assess the benchmark results and review a benchmark assessment form on the wiki. If there are any concerns about the benchmark results, the GCST will be notified and further investigation and/or benchmarking may be required.
  5. If the update is for a specialty simulation (e.g. CO2, CH4, Hg), then a further benchmark may be conducted by the appropriate Working Group.
  6. Once the developer is satisfied with the changes in the 1-month benchmark, GEOS-Chem Model Scientist Daniel Jacob will review the results and approve the new internal version.
  7. 1-year full-chemistry and/or transport tracer benchmarks for Y versions will be conducted only if justifiably requested by the developer or by GEOS-Chem Steering Committee members.
  8. Each new major version release (i.e. X version) will be subject to a 1-year benchmark to be inspected by the GEOS-Chem Steering Committee before approval.

List of GEOS-Chem benchmarks

Links to past 1-month and 1-year benchmark simulations can be found on the GEOS-Chem versions wiki page.

Benchmark output archive

Output files and evaluation plots for 1-month and 1-year benchmark simulations are archived at Harvard as summarized below. GEOS-Chem users may utilize these output for comparisons against their own simulations.

Directory Description
https://gc-dashboard.org/search?searchString=&1Hr=1Hr&GCHP=GCHP&GCC=GCC Contains the following data from the 1-hour benchmarks used to evaluate GEOS-Chem:
  • Evaluation plots & tables
  • Run log
  • Run directory (tarball)
  • Diagnostic files (tarball)
  • Restart Files (tarball)
https://gc-dashboard.org/search?searchString=&1Mon=1Mon&GCHP=GCHP&GCC=GCC Contains the following data from the 1-month benchmarks used to evaluate GEOS-Chem:
  • Evaluation plots & tables
  • Run log
  • Run directory (tarball)
  • Diagnostic files (tarball)
  • Restart Files (tarball)
http://ftp.as.harvard.edu/gcgrid/geos-chem/1yr_benchmarks/ Contains the following data from the 1-year benchmarks used to evaluate GEOS-Chem:
  • Evaluation plots
  • Restart files (tarball)
  • Model output (tarball)
  • Log files (tarball)
  • Input files (tarball)
http://ftp.as.harvard.edu/gcgrid/geos-chem/10yr_benchmarks/ Contains the following data from the 10-year benchmarks used to evaluate GEOS-Chem:
  • Evaluation plots & tables
  • Restart files (tarball)
  • Model output (tarball)
  • Log files (tarball)
  • Input files (tarball)

NOTE: "tarball" refers to a *.tar.gz file. This is an archive of files & folders created with tar cvzf and can be extracted with tar xzvf.

Benchmark plotting routines

The benchmark plotting routines are included with GCPy, a Python took kit available for GEOS-Chem.