Grid-independent GEOS-Chem

From Geos-chem

On this page we shall discuss the development of the Grid-Independent GEOS-Chem (the continuation of the GEOS-Chem column code work). We invite you to visit the related wiki pages:

1. Grid-Independent GEOS-Chem: Chemistry Component
2. Grid-Independent GEOS-Chem: Emissions Component
3. Derived type objects used by the Grid-Independent GEOS-Chem
4. Programming techniques used for the Grid-Independent GEOS-Chem

Contents 1 Overview 1.1 Personnel, objectives, and deliverables 1.2 Basic structure of the Grid-Independent GEOS-Chem 2 Initial implementation details 2.1 Source code cleanup and initial ESMF prep 2.2 Elimination of COMMON blocks 2.3 Removal of hardwired grid 3 Chemistry Component 4 Emissions Component 5 Physics Component 6 Programming techniques

Overview

As many of you may know, GEOS-Chem will form the core of a fully-coupled system that can be interfaced directly into the NASA GEOS-5/GCM. In order for this to happen, the GEOS-Chem standard model will become compliant with the Earth System Model Framework (ESMF). This will allow us to take advantage of Message Passing Interface (MPI) parallelization via ESMF, which has been a long-term goal of the GEOS-Chem development community. MPI parallelization will facilitate GEOS-Chem simulations at extremely fine horizontal resolution, running on 100’s or 1000’s of CPUs.

It is a priority of our implementation strategy to complete this project without disabling or significantly disrupting ongoing research utilizing the stand-alone GEOS-Chem. We believe that this is possible; but it won't be seamless.

As has always been the case, every effort will be made to provide timely and accurate documentation describing these changes and how the community can implement and adapt to them. And, as always, please do not hesitate to post questions, suggestions or comments to the GEOS-Chem Support Team.

Personnel, objectives, and deliverables

Here is a brief overview of the Grid-Independent GEOS-Chem project, and what we seek to achieve:

What is (are) ...	Description
Our Mission:	We seek to embed GEOS-Chem within the NASA GEOS-5/GCM, while preserving GEOS-Chem's existing functionality. Once completed, we seek to use the GEOS-5/GEOS-Chem combination to improve our data assimilation capabilities.
Our Team:	Daniel Jacob (Principal Investigator, Harvard U.) Bob Yantosca (GEOS-Chem Support Team) Michael Long (GEOS-Chem Support Team) Melissa Payer (GEOS-Chem Support Team) Christoph Keller (Post-doctoral research fellow, Harvard U.)
Our Partners:	NASA/GMAO (Global Modeling and Assimilation Office) Beijing Climate Center (collaboration funded by the National Science Foundation) BCC has been extremely helpful in testing the Grid-Independent GEOS-Chem with their local GCM. We will make the Grid-Independent GEOS-Chem code available to BCC as well pending completion and successful validation.
Our Short-Term Deliverables:	We intend to deliver a grid-independent Chemistry Component (i.e. GEOS-Chem's chemistry solver and dry deposition routines) to NASA/GMAO by mid-April 2012. The Chemistry Component that we deliver to NASA/GMAO shall be driven by an interface that is compatible with the Earth System Modeling Framework (ESMF). NOTE: The initial handoff of code to NASA/GMAO was made on June 1st, 2012.
Our Long-Term Deliverables:	We intend to create a grid-independent GEOS-Chem Emissions Component that will replace GEOS-Chem's existing emissions functionality. The GEOS-Chem Emissions Component will be able to be used within the NASA GEOS-5/GCM and Beijing Climate Center CAM/GCM. We intend to create a grid-independent GEOS-Chem Physics Component (i.e. for wet deposition, etc). This will allow the grid-independent GEOS-Chem to run in fully ESMF-compatible standalone mode.

--Bob Y. 15:51, 28 March 2012 (EDT)

Basic structure of the Grid-Independent GEOS-Chem

The Grid-Indpendent GEOS-Chem shall consist of the standard GEOS-Chem source code (with appropriate modifications), grouped into the following components. Click on the link to

1. The Chemistry Component
2. The Emissions Component
3. The Physics Component

--Bob Y. 14:37, 14 December 2012 (EST)

Initial implementation details

We needed to make several initial modifications into the standard GEOS-Chem code (listed below). Most of these changes were to remove Fortran-77 style coding with Fortran-90 style coding.

Source code cleanup and initial ESMF prep

This update was tested in the 1-month benchmark simulation v9-01-02d and approved on 28 Jun 2011.

The following changes were made to GEOS-Chem in order to enable compatibility with ESMF.

1. Global size parameters IGLOB and JGLOB no longer exist in this version.
2. Added new subdirectory ESMF to contain files for compatibility with ESMF. G-C users can ignore this for now.
3. Added targets esmf and esmf_testbed to the top-level Makefile. These will allow GEOS-Chem to compile in the ESMF environment. G-C users can ignore this for now.
4. In Headers/, as a temporary fast-fix, two versions of CMN_* header files are in place. Ones for legacy F77 based code are indicated with a F77_ prefix. These changes are reflected in the code, as well. e.g. #include "CMN_DIAG" -> #include "F77_CMN_DIAG" ... this must be revisited since its elegance is lacking.
5. All FORTRAN source files renamed to permit preprocessing. (i.e. file.f is now file.F, and file.f90 is now file.F90). All Makefiles were likewise modified to handle these changes.
6. Fixed the GeosCore/Makefile to force correct variable to be set for compiling with esmf or esmf_testbed targets.

--Bob Y. 15:20, 26 July 2011 (EDT)

Elimination of COMMON blocks

This update was tested in the 1-month benchmark simulation v9-01-02m and approved on 07 Sep 2011.

The following changes were made to GEOS-Chem to facilitate compatibility with ESMF:

1. Converted all *.h and CMN* in the Headers subdirectory F90 modules (see list below)
2. Replaced statements such as #include "CMN_SIZE" with USE CMN_SIZE_MOD (etc.) everywhere
3. Prevent uninitialized variable in file JRATET.F
4. Updated GEOS-Chem help screen
5. Compile all source code files with the -fp-model source option for precision-safe optimization (ifort compiler only)

List of header files that are now modules:

Common-block file	F90 module	Used for	Obsolete?	Status
CMN_SIZE	CMN_SIZE_mod.F	Size parameters	No (but could eventually be replaced by a derived type object)	Kept
smv_dimension.h	smv_dimension_mod.F	Size parameters	Yes (Could be bundled into CMN_SIZE_mod.F)	Kept (for now)
CMN_DEP	CMN_DEP_mod.F	Drydep / Olson land map	Yes	Removed in GEOS-Chem v9-02
CMN_VEL	CMN_VEL_mod.F	Drydep / Olson land map	Yes	Removed in GEOS-Chem v9-01-03
CMN_DIAG	CMN_DIAG_mod.F	Diagnostics	Yes (but we need it until we rewrite the diagnostics totally)	Kept
CMN_GCTM	CMN_GCTM_mod.F	Physical constants	No	Kept
smv_physconst.h	smv_physconst_mod.F	Physical constants	Yes (Could be bundled together with CMN_GCTM_mod.F)	Kept
CMN_ISOP	CMN_ISOP_mod.F	GEIA biogenic emissions	Yes	Removed in GEOS-Chem v9-01-03
CMN_MONOT	CMN_MONOT_mod.F	GEIA biogenic emissions	Yes	Removed in GEOS-Chem v9-01-03
CMN_NOX	CMN_NOX_mod.F	Soil NOx emissions	Yes (can maybe be merged with the new Soil NOx code)	Kept (for now)
commsoil.h	commsoil_mod.F	Soil NOx emissions	No (Updated for new Soil NOx emissions)	Kept
CMN_O3	CMN_O3_mod.F	Anthro emissions	Yes (Can be removed when emissions are cleaned up)	Kept (for now)
CMN	CMN_mod.F	Tropopause	Yes (can be bundled into tropopause_mod.F)	Kept (for now)
comode.h	comode_loop_mod.F	Chemistry	Kind of (can be replaced by KPP)	Kept (for now)
cmn_fj.h	cmn_fj_mod.F	FAST-J photolysis	No	Kept
jv_cmn.h	jv_cmn_mod.F	FAST-J photolysis	No	Kept
jv_mie.h	jv_mie_mod.F	FAST-J photolysis	No	Kept
smv_errcode.h	smv_errcode_mod.F	Error flags	Yes	Replaced by gigc_errcode_mod.F in GEOS-Chem v9-02

--Bob Y. 15:52, 17 December 2012 (EST)

Removal of hardwired grid

Mike Long wrote:

After a few false-starts, the system is now capable of operating on a flexible grid, primarily due to flexing F90 strengths in place of F77. I am right now in the process of validating that the model results are still good & I haven't broken anything. But, it has not yet been tested with ESMF. That will be next.

The data streams have been largely reconfigured in preparation for ESMF coupling through simple-bundle sockets, and that will be the largest next-step prior to running the 'sandbox' as Bob Yantosca and Arlindo da Silva had set up previously.

Also, it is noteworthy that this development has been in stride with GEOS-Chem science development and updates; and new features and fixes are being brought into this system as they are brought into the standard code.

--Bob Y. 10:50, 25 January 2012 (EST)

Chemistry Component

This discussion has been moved to the Grid-Independent GEOS-Chem: Chemistry Component wiki page.

--Bob Y. 14:17, 14 December 2012 (EST)

Emissions Component

This discussion has been moved to the Grid-Independent GEOS-Chem: Emissions Component wiki page.

--Bob Y. 14:20, 14 December 2012 (EST)

Physics Component

The Physics Component shall provide the following Grid-Independent GEOS-Chem modules with an ESMF-compatible interface.

GEOS-Chem Module	Consisting of
Wet deposition	Wet scavenging Rainout Washout Re-evaporation, etc.
Convection (for standalone configuration only)	Convective transport Loss of soluble tracer in convective transport
Diffusion, aka PBL mixing (for standalone configuration only)	Full PBL mixing option (TURBDAY) or Non-local PBL mixing option (VDIFF)

Development of the Physics Component is at a lower priority than the Chemistry and Emissions Components. Some aspects of the Physics Component will depend on the Chemistry Component (particularly how it handles wet deposition, which is very dependent on the offline met fields or the online GCM used.)

Also, many aspects of the Physics Component will be provided by the NASA/GEOS-5 GCM. The Physics Component will therefore be useful when running GEOS-Chem as a purely ESMF-Compatible standalone model.

--Bob Y. 15:42, 28 March 2012 (EDT)

Programming techniques

We have created a new Programming techniques used for the Grid-Independent GEOS-Chem wiki page. The following discussions have now been moved there:

1. The DEVEL C-preprocessor switch
2. The EXTERNAL_GRID and EXTERNAL_FORCING C-preprocessor switches
3. Restricting screen and log file output to the root CPU
4. Using findFreeLUN to assign logical unit numbers for file I/O

Also please see our new Derived type objects used by the Grid-Independent GEOS-Chem wiki page. This page provides a detailed description of the new derived-type objects that we have introduced to pass data between GEOS-Chem routines. This enhances grid-independent functionality in GEOS-Chem.

--Bob Y. 15:55, 17 December 2012 (EST)