FlexGrid

NOTE: This page is being actively edited right now and the information might be incomplete.

FlexGrid documentation by Jiawei Zhuang (jiaweizhuang@g.harvard.edu)

Overview

FlexGrid is a new functionality in GEOS-Chem v11-02 to unify all metfield I/O routines by HEMCO.

HEMCO's I/O interface and regridding&cropping capability allow users to

• Set up a nested simulation over any custom domains without changing model's source code.
• Explicitly control each meteorological variable in run-time configuration files to perform metfield experiments (an interface resembling GCHP).

The idea of FlexGrid

In theory, setting up a new nested domain simply requires

• New metfields over that domain
• New regional emission data sets if necessary
• Tunning several domain-dependent or resolution-dependent parameters.

There's no reason that one has to touch the model source code. However, in previous versions, all nested domains were hard-coded so users need to do a lot of tedious coding in order to set up a new domain. The major trouble was metfield I/O, as we assumed that the model grid should exactly match the metfield grid. The solution is HEMCO, which has already been used for regridding emission data sets at multiple grids to the same model grid. Here we just do the reverse -- regrid the same metfield data set to different user-specified domains.

FlexGrid implementation plan

The table below shows the implementation plan for FlexGrid:

Stage	Detail	Status
Stage 1: Read metfields by HEMCO	Enable the compile option "MET=flexgrid" to read any NetCDF-format metfields through HEMCO. This unifies "MET=geosfp" and "MET=merra2"	Will be included in v11-02
Stage 2: Implement custom nested simulations	Enable the compile option "NEST=cu" (custom) to allow custom nested simulations.	Will be included in v11-02
Stage 3: Grid-independent GEOS-Chem classic	Unify compile options "GRID=xxx" by moving all grid size parameters and resolution-dependent parameters to run-time configuration files.	No specific plan right now. Need to be traded off against the nested-GCHP development.

FlexGrid for different types of users

According to the availability of meteorological data, users can get different benefits from FlexGrid:

User type	benefits
Only have standard metfields (e.g. global 4x5, 0.25x0.3125 nested NA)	Be able to explicitly control each met variable in HEMCO_Config.rc (like Extdata.rc for GCHP). You can freeze certain variables, apply scaling factors, or read user-generated met data. Be able to set up nested simulations over any sub-domains of the metfield you have. For example, you can set up an east-US simulation in no time if you have the nested North American metfield data. If a new nested domain is needed, you can skip the source code modification and just prepare the metfield input.
Have global native metfields (For example, GCHP users might store them for global high-resolution runs. We can reuse them for nested simulations.)	Be able to run nested simulations over any domains in the world.

FlexGrid Stage 1 User's Guide

Compile

Just change the MET option to "flexgrid". Leave other options as-is. This works for any kinds of simulations (no matter global or nested, UCX or tropchem) that use NetCDF metfields.

For example, the original compile command for the 4x5 standard run is:

make -j4 UCX=yes CHEM=standard GRID=4x5 MET=geosfp

Now simply use

make -j4 UCX=yes CHEM=standard GRID=4x5 MET=flexgrid

Set up run directory

(Need to eventually add new rundirs to UT)

The executable generated in the above section works with the original run directory "geosfp_4x5_standard", with minimal modifications.

(1) Add this new config file FlexGrid_Config.rc to your run directory and include it in your HEMCO_Config.rc, preferably at the NON-EMISSIONS subsection:

###############################################################################
### NON-EMISSIONS DATA (subsection of BASE EMISSIONS SECTION)
...
###############################################################################

#==============================================================================
# --- metfields for FlexGrid ---
#==============================================================================
>>>include FlexGrid_Config.rc  

#==============================================================================
# --- Time zones (offset to UTC) ---
#==============================================================================
...

(2) Add user-defined tokens in HEMCO_Config.rc to complete the file paths in FlexGrid_Config.rc, for example

###############################################################################
### BEGIN SECTION SETTINGS
###############################################################################

ROOT:                        /n/holylfs/EXTERNAL_REPOS/GEOS-CHEM/gcgrid/data/ExtData/HEMCO
METDIR:                      /n/holylfs/EXTERNAL_REPOS/GEOS-CHEM/gcgrid/data/ExtData/GEOS_4x5/GEOS_FP
METMODEL:                    GEOSFP
MET:                         geosfp
METRES:                      4x5 
Logfile:                     HEMCO.log

Note that the $MET token is not for metfield I/O but is needed by FlexGrid. This is for the path of OTD-LIS scaling factor:

103 LIGHTNOX_OTDLIS $ROOT/LIGHTNOX/v2014-07/OTD-LIS-Local-Redist.CTH.v5.$MET.$RES.nc OTD $YYYY/1-12/1/0 C xy unitless NO - 1 1

$MET was once determined by the compile flag, but as long as a single compilation now works for both geosfp and merra2, the model will not know which one we are using unless we specify it.

(3) Add Lightning NOx and dust scaling factor to HEMCO_Config.rc. For 4x5 GEOSFP, they should be:

103     LightNOx             : on    NO
    --> OTD-LIS scaling      :       0.5568 
    --> OTD-LIS factors      :       true
    --> CDF table            :       $ROOT/LIGHTNOX/v2014-07/light_dist.ott2010.dat
104     SoilNOx              : on    NO
    --> Use fertilizer NOx   :       true
105     DustDead             : on    DST1/DST2/DST3/DST4
    --> Mass tuning factor   :       8.3286e-4 

We force users to set scaling factors explicitly because those processes are known to have strong resolution&domain dependence -- you should keep this in mind when building a new nested domain. (same for changing the resolution in GCHP)

The numbers are taken from hcox_dustdead_mod.F and hcox_lightnox_mod.F90. (Need a table summarizing all those factors)

(4) Now you should be able to run the model as usual.

Also note that the original metfield path in input.geos has no effect now:

 => GEOS-FP subdir      : GEOS_FP/YYYY/MM/

Use FlexGrid functionalities

FlexGrid_Config.rc is the place to explicitly control all meteorological input variables (similar to Extdata.rc in GCHP). Each variable is listed in one line, just like other emission data entries in HEMCO_Config.rc:

# --- CN     metfields   ---
* FRLAKE    $METDIR/2011/01/$METMODEL.20110101.CN.$METRES.nc FRLAKE    2011/1/1/0  C xy  1  * -  1 1
* FRLAND    $METDIR/2011/01/$METMODEL.20110101.CN.$METRES.nc FRLAND    2011/1/1/0  C xy  1  * -  1 1
* FRLANDIC  $METDIR/2011/01/$METMODEL.20110101.CN.$METRES.nc FRLANDIC  2011/1/1/0  C xy  1  * -  1 1
* FROCEAN   $METDIR/2011/01/$METMODEL.20110101.CN.$METRES.nc FROCEAN   2011/1/1/0  C xy  1  * -  1 1
* PHIS      $METDIR/2011/01/$METMODEL.20110101.CN.$METRES.nc PHIS      2011/1/1/0  C xy  1  * -  1 1
...
# --- A3dyn  metfields   ---
* DTRAIN    $METDIR/$YYYY/$MM/$METMODEL.$YYYY$MM$DD.A3dyn.$METRES.nc DTRAIN   2011-2014/1-12/1-31/0-23/+90minute C xyz  1  * -  1 1
* OMEGA     $METDIR/$YYYY/$MM/$METMODEL.$YYYY$MM$DD.A3dyn.$METRES.nc OMEGA    2011-2014/1-12/1-31/0-23/+90minute C xyz  1  * -  1 1
* RH        $METDIR/$YYYY/$MM/$METMODEL.$YYYY$MM$DD.A3dyn.$METRES.nc RH       2011-2014/1-12/1-31/0-23/+90minute C xyz  1  * -  1 1
* U         $METDIR/$YYYY/$MM/$METMODEL.$YYYY$MM$DD.A3dyn.$METRES.nc U        2011-2014/1-12/1-31/0-23/+90minute C xyz  1  * -  1 1
* V         $METDIR/$YYYY/$MM/$METMODEL.$YYYY$MM$DD.A3dyn.$METRES.nc V        2011-2014/1-12/1-31/0-23/+90minute C xyz  1  * -  1 1
...

Most of the tricks that you apply to the emission data can also be applied here, such as adding scaling factors.

• Change the metfield path

Say you've modified the default temperature data to test the sensitivity to the temperature, you can just tell HEMCO to read your new file instead of the default file, while keep all other variables untouched:

* TMPU1     path_to_new_file    T        2011-2014/1-12/1-31/0-23          C xyz  1  * -  1 1
* TMPU2     path_to_new_file    T        2011-2014/1-12/1-31/0-23/+3hour   C xyz  1  * -  1 1

• Freeze certain variables

You can force the model to keep reusing the wind in 2011 by restricting the time range: (to see how does the trend of certain meteorological variables affects the simulation)

* U   $METDIR/$YYYY/$MM/$METMODEL.$YYYY$MM$DD.A3dyn.$METRES.nc U   2011/1-12/1-31/0-23/+90minute C xyz  1  * -  1 1 
* V   $METDIR/$YYYY/$MM/$METMODEL.$YYYY$MM$DD.A3dyn.$METRES.nc V   2011/1-12/1-31/0-23/+90minute C xyz  1  * -  1 1

FlexGrid Stage 2 User's Guide

This section describes the steps to set up a custom nested domain, assuming the knowledge of Stage 1.

Issues to discuss

Changing the nested domain becomes trivial once the metfield I/O is handled by HEMCO. However, we should decide how should the new configuration file (e.g. input.geos) look like. We need to read those hard-coded parameters from the configuration file:

• IIPAR, JJPAR in Header/CMN_SIZE_mod.F
• I1_BC, J1_BC, I2_BC, J2_BC in GeosCore/tpcore_bc_mod.F

along with "Global offsets I0, J0" input.geos, they fully describe the range of the nested domain.

FlexGrid implementation details

Source code availability

The source code is currently under development at my own repository. Will be soon merged to the standard repository.

Utilities for compile flags

Cleaning the "#ifdef" C-preprocessor statements in the source can be very tedious. There are some python scripts to check those statements and massively add new codes for FlexGrid.