FlexChem

From Geos-chem
Jump to: navigation, search

On this page we provide information about the FlexChem chemical solver, which was introduced in GEOS-Chem v11-01g.

Overview

The clean and flexible reimplementation of the Kinetic PreProcessor package (aka KPP)—known as FlexChem—is nearing full integration into GEOS-Chem. Most of the remaining FlexChem development work will focus on replacing legacy infrastructure—which prevents GEOS-Chem from operating efficiently in high-performance computing (HPC) environments—with newer, more efficient algorithms.

Check back soon for the latest information!

--Bob Yantosca (talk) 19:58, 19 May 2016 (UTC)

Milestones

The following table shows several milestones that were achieved in the FlexChem implementation, development, as well as ongoing development tasks.

Task Developer Status
Add FlexChem into v11-01c Mike Long Completed 14 Dec 2015
Enable the "Tropchem" mechanism Mike Long Completed 14 Dec 2015
Restore the OH and HO2 diagnostics (ND43) Melissa Sulprizio Completed 18 Dec 2015
Remove CSPEC array and replaced with State_Chm%Species Melissa Sulprizio Completed 22 Dec 2015
Enable a temporary workaround for family tracers (ISOPN, MMN, CFCX, HCFCX) Mike Long Completed 25 Jan 2016
Enable FAST-JX photochemistry Mike Long Completed 25 Jan 2016
Enable the "Benchmark" (aka "Standard") chemistry mechanism Melissa Sulprizio Completed 29 Jan 2016
Restore the broken J-value diagnostic (ND22) Melissa Sulprizio Completed 15 Mar 2016
Parallelize the main KPP driver loop; fixed other minor issues Bob Yantosca Completed 30 Mar 2016
Enable the "SOA" and "SOA-SVPOA" mechanisms Lizzie Lundgren Completed 01 Apr 2016
Merge FlexChem code based on v11-01c with v11-01f Melissa Sulprizio Completed 20 Apr 2016
Add KPP repository to Bitbucket (https://bitbucket.org/gcst/kpp) Mike Long Completed 19 Apr 2016
Create new gckpp_*.F90 files from the updated KPP solver package Melissa Sulprizio Completed 22 Apr 2016
Enable the "UCX" mechanism Melissa Sulprizio Completed 26 Apr 2016
Introduce a new prod/loss diagnostic into the KPP solver package Mike Long Completed 29 Apr 2016
Complete unit tests and 1-month benchmarks for the Standard, Tropchem, UCX, SOA, and SOA-SVPOA simulations Melissa Sulprizio
Lizzie Lundgren
Completed 05 May 2016
Remove tracer indices of Rn, Pb, Be, Hg, and POPs simulations from tracerid_mod.F. (This paves the way for us to retire tracerid_mod.F.) Bob Yantosca Completed 02 May 2016
Add a fast species name lookup algorithm Bob Yantosca Code is in place as of 04 May 2016, but has not been implemented throughout GEOS-Chem yet.
Store the unique list of GEOS-Chem species in the GEOS-Chem species database object. (In other words, we combine the list of advected tracers with the list of KPP chemical species and remove duplicate entries). Bob Yantosca Completed 09 May 2016
Store the indices of each species in the KPP chemical reaction matrix in the GEOS-Chem species database object. Bob Yantosca Completed 09 May 2016
Merge FlexChem with other v11-01g updates and bug fixes Bob Yantosca Completed 17 May 2016
Remove family tracer fields of the Input_Opt object, namely: TRACER_N_CONST, TRACER_CONST, TRACER_COEFF, ID_EMITTED. These were only used by SMVGEAR and are now obsolete. Bob Yantosca Completed 18 May 2016
Create mapping vectors in the State_Chm object to store the ID numbers of species that are advected, dry-deposited, wet-deposited, and/or included in the KPP chemical mechanism Bob Yantosca Completed 19 May 2016
Remove all 1-D loop indexing variables (JLOOP, KLOOP, KTLOOP, JLOP, IXSAVE, IYSAVE, IZSAVE) that were used by SMVGEAR Melissa Sulprizio Completed 19 May 2016
Remove SMVGEAR input files (globchem.dat, mglob.dat) and associated subroutines (readchem.F, reader.F) Melissa Sulprizio Completed 30 Jun 2016
Remove tracerid_mod.F and related tracer ID flags (IDTxxxx, IDxxxx). Instead, use the GEOS-Chem species database to retrieve species index from species name. Jacob Group Code-A-Thon Completed 24 Jun 2016
Remove tracers from restart files and read/write restart file species only Lizzie Lundgren Completed 12 Jul 2016
Create unit conversion routines for species that mimic existing routines for tracers or convert to/from molecules/cm3 Lizzie Lundgren Completed 25 Jul 2016
Add State_Chm variable to track species units throughout GEOS-Chem (State_Chm%Spc_Units) Lizzie Lundgren Completed 25 Jul 2016
Remove the remaining family tracers (ISOPN, MMN, CFCX, HCFCX). Henceforth, FlexChem will only work with individual species. Melissa Sulprizio Completed 28 Jul 2016
Move the non-advected species initial unit conversion of background values from INIT_FLEXCHEM (called during chemistry) to READ_GC_RESTART (called during restart file read) Lizzie Lundgren Completed 4 Aug 2016
Store species background concentrations values previously stored in globchem.dat in the species database Lizzie Lundgren Completed 11 Aug 2016
Rebuild Standard, Tropchem, UCX, SOA, SOA-SVPOA chemistry mechanisms to use the new prod/loss functionality in KPP (i.e. FLUX on) Melissa Sulprizio Completed 13 Jul 2016
Attach the new KPP prod/loss output to the GEOS-Chem prod/loss diagnostic (ND65) Melissa Sulprizio Completed 11 Aug 2016
Remove State_Chm%Tracers and use species concentrations in State_Chm%Species instead GCST Completed 29 Aug 2016
Validate FlexChem with v11-01g benchmarks GCST 1-month benchmark:
Approved 14 Sep 2016

1-year benchmark:
Approved 28 Sep 2016

Fix P/L diagnostics to ignore cycling reactions Mike Long 08 Nov 2016
Rebuild Standard, Tropchem, UCX, SOA, SOA-SVPOA chemistry mechanisms with Kppa GCST TBD
Update the GEOS-Chem Makefiles so that a fresh version of KPP will be built (using a custom chemical mechanism that you specify) each time GEOS-Chem is compiled. GCST TBD

Adding chemical mechanisms in FlexChem

KPP source code

The KPP source code for FlexChem is currently available as a Git repository on Bitbucket. It can be obtained using:

git clone https://bitbucket.org/gcst/kpp
git checkout -b GC_updates origin/GC_updates

Once you have KPP on your system, you will need to build it:

cd kpp/kpp-2.2.3_01
make clean
make

If the build completed successfully, you should see the executable kpp in the kpp/kpp-2.2.3_01/bin/ directory. Next, add KPP to your PATH. For bash users, add the following lines the ~/.bashrc file:

export PATH=$PATH:/PATH_TO_KPP/KPP/kpp-2.2.3_01/bin/
export KPP_HOME=/PATH_TO_KPP/KPP/kpp-2.2.3_01

--Melissa Sulprizio (talk) 21:18, 10 August 2016 (UTC)

Building a custom chemical mechanism

GEOS-Chem v11-01g has five pre-built chemical mechanisms: Standard, Tropchem, SOA, SOA-SVPOA, and UCX. These mechanisms are described on our GEOS-Chem chemistry mechanisms wiki page.

GEOS-Chem will eventually include the option to create a custom chemistry mechanism and automatically build that mechanism with KPP when GEOS-Chem is compiled. For now, users still need to build custom chemistry mechanisms manually. The KPP Git repository includes the following files:

 geos2kpp_parser.pl
 gckpp.kpp
 rename_f90.sh
 copy_gckpp.sh

Note: You'll need to checkout the GC_updates branch to access these files. This can be achieved with git checkout GC_updates.

These files can be used to build a custom chemistry mechanisms with KPP. Follow these steps:

  1. The geos2kpp_parser.pl script can be used to create globchem.eqn, globchem.spc, and globchem.def files from a globchem.dat file. You may also choose to start from the preexisting globchem.eqn, globchem.spc, and globchem.def files found in the KPP subdirectories of the GEOS-Chem source code (e.g. Code.v11-01/KPP/Standard/).
  2. If needed, edit globchem.eqn to add or modify reactions for your mechanism (see below) and edit globchem.spc to add/modify species to your mechanism. Put the final globchem.* files in the same directory as the gckpp.kpp file.
  3. Modify the gckpp.kpp file if needed. This file defines the KPP integrator and tells KPP to use the globchem.spc and globchem.eqn files to build the mechanism. This file also defines the prod/loss families as described below.
    IMPORTANT: The default gckpp.kpp file that is included with KPP does not define any prod/loss families. Therefore, you must make sure the ND65 prod/loss diagnostic is turned off in input.geos when you run GEOS-Chem. If you would like to use prod/loss families in your custom mechanism, you can see the gckpp.kpp files in the KPP subdirectories of the GEOS-Chem source code (e.g. Code.v11-01/KPP/Standard/gckpp.kpp) for examples.
  4. Execute KPP using kpp gckpp.kpp. This will create the necessary gckpp_*.f90 files.
  5. Run ./rename_f90.sh to rename gckpp_*.f90 to gckpp_*.F90.
  6. Open copy_gckpp.sh and modify the line for codedir. Save and run ./copy_gckpp.sh. This will copy the gckpp_*.F90 files to codedir/Custom/.
  7. Compile GEOS-Chem for the new mechanism by including CHEM=Custom in your make command.

--Melissa Sulprizio (talk) 21:18, 10 August 2016 (UTC)

Adding a reaction to the mechanism

No matter what reaction is being added, the general procedure is the same. A new line must be added to globchem.eqn of the following form:

{700} A + B = C + 2.000D : REQN(ARG_A,ARG_B);

The first number (700) is a KPP tag. The specific value has no effect on the reaction, but this should uniquely identify the reaction. The second section denotes the reactants (A and B) as well as the products (C) of the reaction. If exactly one molecule is consumed or produced, then the factor can be omitted; otherwise the number of molecules consumed or produced should be specified with at least 1 decimal place of accuracy. The final section, between the colon and semi-colon, specifies the function and arguments which will be used to calculate the reaction rate constant k. For a two-body reaction, the overall rate at which the reaction proceeds will be calculated as k[A][B].

Two-body reactions

In the case of two-body reactions proceeding according to the Arrhenius equation, calculation of k is straightforwardly implemented by calling the GCARR function. For example, the most recent JPL chemical data evaluation (as of February 2017) specifies that the reaction O3 + NO produces NO2 and O2, and its Arrhenius parameters are A = 3.0x10^-12 and E/R = 1500. The entry for this reaction therefore reads as follows (noting the different sign convention used for E/R)

{1} O3 + NO = NO2 + O2 : GCARR(3.00E-12,0,-1500); 

The exact implementation of the Arrhenius rate calculation can be found in gckpp.kpp, as can other rate functions such as that required for three-body, pressure-dependent reactions. Any rate function which is to be referenced in globchem.eqn must be available in gckpp.kpp prior to building the reaction mechanism.

Photolysis reactions

Similarly, a photolysis reaction can be specified by giving the correct index of the PHOTOL array. This index can be determined by inspecting the file FJX_j2j.dat in your run directory. For example, one branch of the NO3 photolysis reaction is specified in globchem.eqn as

{522} NO3 + hv = NO2 + O : PHOTOL(12) 

Referring back to FJX_j2j.dat shows that reaction 12, as specified by the left-most index, is indeed NO3 = NO2 + O:

 12 NO3       PHOTON    NO2       O                       0.886 /NO3   /

If your reaction is not already in FJX_j2j.dat, you may add it there. You may also need to modify FJX_spec.dat (also found in your run directory) to include cross-sections for your species.

Finally, you will need to update routine PHOTO_JX found in fast_jx_mod.F to archive the J-values for your new reaction to the ND22 diagnostic. The J-values are obtained from array ZPJ which uses the same index found in the FJX_j2j.dat file. For example:

     !---------------------------------------------------------------
     ! Archive J-values for ND22 diagnostic
     !---------------------------------------------------------------
     IF ( ND22 > 0 ) THEN

        ! Save J-values for 2PM diagnostic boxes
        IF ( LTJV(ILON,ILAT) > 0 ) THEN

           DO L = 1, LD22
              ! AD22 IDs 5, 6, and 15 (J-values for O3 and O2)
              ! are handled in routine PHOTRATE_ADJ
              ! Hardcode ZPJ indices based on valued from FJX_j2j.dat
              ! for now (mps, 3/15/16)
              AD22(ILON,ILAT,L, 1) = AD22(ILON,ILAT,L, 1) +  ! JNO2
    &                                ZPJ(L,11,ILON,ILAT)
              ...

Index 11 used for obtaining JNO2 correcponds to the index found FJX_j2j_mod.F90:

 11 NO2       PHOTON    NO        O                       1.000 /NO2   /

Heterogeneous reactions

Implementing new heterogeneous chemistry requires an additional step. For the reaction in question, a reaction should be added as usual, but this time the rate function should be given as an entry in the HET array. A simple example is uptake of HO2, specified as

{285} HO2 = O2 : HET(ind_HO2,1);

Note that the product in this case, O2, is actually a fixed species, so no O2 will actually be produced. O2 is used in this case only as a dummy product to satisfy the KPP requirement that all reactions have at least one product. Here, HET is simply an array of pre-calculated rate constants. The rate constants in HET are actually calculated in gckpp_HetRates.F90. To implement an additional heterogeneous reaction, the rate calculation must be added to this file. The following example illustrates a (fictional) heterogeneous mechanism which converts the species XYZ into CH2O. This reaction is assumed to take place on the surface of all aerosols, but not cloud droplets (this requires additional steps not shown here). Three steps would be required:

  • Add a new line to globchem.eqn, such as {900} XYZ = CH2O : HET(ind_XYZ,1);
  • Add a new function to gckpp_HetRates.F90 designed to calculate the heterogeneous reaction rate. As a simple example, we can copy the function HETNO3 and rename it HETXYZ. This function accepts two arguments: molecular mass of the impinging gas-phase species, in this case XYZ, and the reaction's "sticking coefficient" - the probability that an incoming molecule will stick to the surface and undergo the reaction in question. In the case of HETNO3, it is assumed that all aerosols will have the same sticking coefficient, and the function returns a first-order rate constant based on the total available aerosol surface area and the frequency of collisions.
  • Add a new line to the function SET_HET in gckpp_HetRates.F90 which calls the new function with the appropriate arguments and passes the calculated constant to HET. Example: assuming a molar mass of 93 g/mol, and a sticking coefficient of 0.2, we would write HET(ind_XYZ, 1) = HETXYZ(9.30E1_fp, 2E-1_fp)

The function HETXYZ can then be specialized to distinguish between aerosol types, or extended to provide a second-order reaction rate, or whatever the user desires.

--Sebastian D. Eastham (talk) 22:45, 14 February 2017 (UTC)

Chemical species in FlexChem

A complete list of chemical species included in the pre-built (Standard, Tropchem, SOA, SOA-SVPOA, UCX) mechanisms can be viewed on our Species in GEOS-Chem wiki page.

Adding a chemical species

Adding chemical species to FlexChem can be achieved be following these simple steps:

  1. Add the species name(s) to the globchem.spc file
  2. Add or modify any associated reactions to the globchem.eqn file
  3. Build the chemical mechanism with KPP by following the steps outlined above

NOTE: If a species is defined by KPP, it will automatically be added to the species database with the default settings for non-advected chemical species when GEOS-Chem runs. However, we recommend also adding your species to the species database by following these instructions. Including a species in the species database is required if the new species is to be defined as an advected species in input.geos.

--Melissa Sulprizio (talk) 17:27, 22 November 2016 (UTC)

Removal of family tracers

In GEOS-Chem v9-02 we removed all family tracers from GEOS-Chem. For example, the NOx family was replaced by individual tracers NO, O3, NO3, and HNO2. Similarly, the Ox family was replaced by individual tracers O3, NO2, and NO3.

But certain updates that were made to GEOS-Chem after v9-02 introduced new family tracers into the various chemistry mechanisms:

The implementation of the FlexChem chemical solver package in GEOS-Chem v11-01 forced us to remove these family tracers, because the KPP chemical solver is designed to only include individual species in its mechanism. Removing these families also allows us to avoid some cumbersome sections of code that were used to construct each family from its constituent members.

Therefore, the following families have been removed from GEOS-Chem v11-01g, and replaced with individual species:

Removed Added as individual advected species Mechanisms in which these are used
ISOPN ISOPND ISOPNB Standard, UCX, TropChem, SOA, SOA-SVPOA
MVKN MACRN MVKN Standard, UCX, TropChem, SOA, SOA-SVPOA
CFCX CFC113 CFC114 CFC115 Standard, UCX
HCFCX HCFC22 HCFC141b HCFC142b Standard, UCX

The constituents of each former family are now included as advected species in the various GEOS-Chem mechanisms. To obtain diagnostic output for the former families, you can save out each of the individual species and combine the output in post-processing.

--Bob Yantosca (talk) 16:20, 16 August 2016 (UTC)

Production & loss diagnostic

Mike Long implemented the functionality for chemical prod/loss families in the KPP source code. With this option turned on, KPP will calculate the net prod/loss rates for user-defined chemical families.

Adding production & loss families

KPP prod/loss families are turned on by the #FAMILIES token in the gckpp.kpp file. For example:

  #INTEGRATOR rosenbrock
  #LANGUAGE Fortran90
  #DRIVER none
  
  #INCLUDE globchem.spc
  #INCLUDE globchem.eqn
  
  #FAMILIES
  POx : O3 + NO2 + 2NO3 + PAN + PMN + PPN + HNO4 + 3N2O5 + HNO3 + BrO + HOBr + BrNO2 + 2BrNO3 + MPN + ETHLN + ISN1 + ISOPNB + ISOPND + MACRN + MVKN + PROPNN + R4N2 + INPN + ISNP + INO2 + ISNOOA + ISNOOB + ISNOHOO + MAN2 + PRN1 + PRPN + R4N1 + PMNN + MACRNO2 + ClO + HOCl + ClNO2 + 2ClNO3 + 2Cl2O2 + 2OClO + O + O1D;
  LOx : O3 + NO2 + 2NO3 + PAN + PMN + PPN + HNO4 + 3N2O5 + HNO3 + BrO + HOBr + BrNO2 + 2BrNO3 + MPN + ETHLN + ISN1 + ISOPNB + ISOPND + MACRN + MVKN + PROPNN + R4N2 + INPN + ISNP + INO2 + ISNOOA + ISNOOB + ISNOHOO + MAN2 + PRN1 + PRPN + R4N1 + PMNN + MACRNO2 + ClO + HOCl + ClNO2 + 2ClNO3 + 2Cl2O2 + 2OClO + O + O1D;
  PCO : CO;
  LCO : CO;
  PSO4 : SO4;

To add a new prod/loss family, add a new line to the #FAMILIES section with the format

  FAM_NAME : MEMBER_1 + MEMBER_2 + ... + MEMBER_N;

The family name must start with P or L to indicate whether KPP should calculate a production or a loss rate.

IMPORTANT: When adding a prod/loss family or changing any of the other settings in gckpp.kpp, the chemistry mechanism will need to be rebuilt with KPP as described above.

The pre-built chemistry mechanisms (Standard, Tropchem, SOA, SOA-SVPOA, and UCX) were built with the default prod/loss families listed in the example above. For the Tropchem, SOA, and SOA-SVPOA mechanisms several species (ClO + HOCl + ClNO2 + 2ClNO3 + 2Cl2O2 + 2OClO + O + O1D) were removed from the Ox family because those species are not defined in those mechanisms. The Ox and CO rates are used in GEOS-Chem for computing budgets in the 1-month benchmark simulations and PSO4 is required for simulations using TOMAS aerosol microphysics.

--Melissa Sulprizio (talk) 17:13, 9 November 2016 (UTC)

How it works

In GEOS-Chem v11-01

NOTE: The way that KPP computes production and loss families was greatly simplified in GEOS-Chem v11-02a. See the post below for more information.

The number of prod/loss families is defined by NFAM in gckpp_Parameters.F90. KPP automatically sets this number according to the number of families listed in the gckpp.kpp file used to build the mechanism. The prod/loss family names are defined in array FAM_NAMES in gckpp_Monitor.F90. For example, for the default prod/loss families listed above we have:

  CHARACTER(LEN=15), PARAMETER, DIMENSION(5) :: FAM_NAMES = (/ &
     'POx            ','LOx            ','PCO            ', & ! index 1 - 3
     'LCO            ','PSO4           ' /)

KPP computes the prod/loss families in routine ComputeFamilies (found in gckpp_Util.F90). The FAM array in that routine contains the accumulated prod/loss rates for that family. For example, the POx family (family 1) is computed as:

  FAM(1) = V(13)+V(14)+V(15)+V(16)+V(17)+V(18)+V(19)+V(20)+V(21)+2*V(22)+...

The V array here corresponds to KPP's VAR array containing species concentrations in molec/cm3. The indices passed to V indicate which reaction rates are used to compute the prod/loss rate for that family. Here are the steps you can take to validate the reactions used for a given family:

  1. Select any index number for array V used in the computation
  2. Find the index number in SPC_NAMES (defined in gckpp_Monitor.F90)
  3. Find the dummy RR species in EQN_NAMES (defined in gckpp_Monitor.F90)
  4. Make sure the reaction identified in the previous step includes a family member as a product (for prod families) or a reactant (for loss families)

Following these steps for the POx family example, we have:

  1. Select V(13)
  2. Index 13 corresponds to RR10 in SPC_NAMES
  3. RR10 is found in HO2 + NO --> RR10 + NO2 + OH in EQN_NAMES
  4. NO2 is a member of the POx family and is a product in the above reaction ✔

--Melissa Sulprizio (talk) 17:13, 9 November 2016 (UTC)

In GEOS-Chem v11-02 and later versions

This update was included in v11-02a and approved on 12 May 2017.

KPP has been updated to greatly simplify how the prod/loss rates are computed. In v11-02a, the chemical mechanisms were rebuilt with KPP at commit Fix to add coefficients to *_Monitor.F90 - MSL.. In the updated KPP, prod/loss families now produce one dummy species that is now named with the family name in SPC_NAMES. For example, POx is now computed using KPP species POx in VAR instead of several RR* species. This greatly reduces the number of dummy species down to the number of prod/loss families that are defined in the input file gckpp.kpp.

Routine Compute_Families in gckpp_Util.F90 is now obsolete. We can now obtain the prod/loss rates directly from the VAR array in KPP. The following updates need to be made to routine Do_FlexChem in GeosCore/flexchem_mod.F90 to properly use the updated KPP prod/loss rates. Text green (red) indicates code that was added (removed).

1. Declare new variable KppID and remove obsolete variables for the prod/loss diagnostic
    !
    ! !LOCAL VARIABLES: 
    !
        INTEGER                :: I, J, L, N, F, SpcID, KppID
 
        ...

        ! For prod/loss diagnostic
        REAL(fp)               :: FAM(NFAM)
        INTEGER                :: IND
        INTEGER                :: COEF
        CHARACTER(LEN=14)      :: NAME
2. Declare KppID as !OMP PRIVATE
    !$OMP PRIVATE  ( SpcID, KppID,    F                       ) &
3. Initialize the prod/loss rates to zero in KPP each time Do_FlexChem is called
        !===========================================================
        ! Initialize species concentrations
        !===========================================================
        ! Loop over KPP Species
        DO N=1,NSPEC

           ! GEOS-Chem species ID
           SpcID = State_Chm%Map_KppSpc(N)

           ! Initialize KPP species concentration array
           IF ( SpcID .eq. 0) THEN
              C(N) = 0.0_dp
           ELSE
              C(N) = State_Chm%Species(I,J,L,SpcID)
           ENDIF

        ENDDO

        IF ( Input_Opt%DO_SAVE_PL ) THEN

           ! Loop over # prod/loss families
           DO F = 1, NFAM

              ! Determine dummy species index in KPP
              KppID = Ind_(TRIM(FAM_NAMES(F)),'K')

              ! Initialize prod/loss rates
              IF ( KppID > 0 ) C(KppID) = 0.0_dp

           ENDDO

        ENDIF
4. Update the prod/loss rate diagnostic code
           !==============================================================
           ! Obtain prod/loss rates from KPP [molec/cm3]
           !==============================================================
           IF ( Input_Opt%DO_SAVE_PL ) THEN

              ! Obtain prod/loss rates from KPP [molec/cm3]
              CALL ComputeFamilies( VAR, FAM )

              ! Loop over # prod/loss families
              DO F = 1, NFAM

                 ! Determine dummy species index in KPP
                 KppID = Ind_(TRIM(FAM_NAMES(F)),'K')

                 !--------------------------------------------------------
                 ! Add to AD65 array [molec/cm3/s]
                 !--------------------------------------------------------
                 AD65(I,J,L,F) = AD65(I,J,L,F) + FAM(F) / DT
                 IF ( KppID > 0 ) THEN
                    AD65(I,J,L,F) = AD65(I,J,L,F) + VAR(KppID) / DT
                 ENDIF

                 !--------------------------------------------------------
                 ! Save out P(Ox) and L(Ox) from the fullchem simulation
                 ! for a future tagged O3 run
                 !--------------------------------------------------------
                 IF ( Input_Opt%DO_SAVE_O3 ) THEN
                    IF ( TRIM(FAM_NAMES(F)) == 'POx' ) THEN
                       POx(I,J,L) = FAM(F) / DT
                       POx(I,J,L) = VAR(KppID) / DT
                    ENDIF
                    IF ( TRIM(FAM_NAMES(F)) == 'LOx' ) THEN
                       LOx(I,J,L) = FAM(F) / DT
                       LOx(I,J,L) = VAR(KppID) / DT
                    ENDIF
                 ENDIF

    #if defined( TOMAS )
                 !-------------------------------------------------------
                 ! FOR TOMAS MICROPHYSICS:
                 !
                 ! Obtain P/L with a unit [kg S] for tracing 
                 ! gas-phase sulfur species production (SO2, SO4, MSA)
                 ! (win, 8/4/09)
                 !-------------------------------------------------------

                 ! Calculate H2SO4 production rate [kg s-1] in each
                 ! time step (win, 8/4/09)
                 IF ( TRIM(FAM_NAMES(F)) == 'PSO4' ) THEN 
                    ! Hard-coded MW
                    H2SO4_RATE(I,J,L) = FAM(F) / AVO * 98.e-3_fp * &
                    H2SO4_RATE(I,J,L) = VAR(KppID) / AVO * 98.e-3_fp * &
                                        State_Met%AIRVOL(I,J,L)  * &
                                        1e+6_fp / DT
                 ENDIF
    #endif
              ENDDO

           ENDIF

--Melissa Sulprizio (talk) 20:12, 23 March 2017 (UTC)


The table below shows the speedup that is obtained with the improved prod/loss rate diagnostics.

Run name,
timesteps,
and submitter
Machine or Node
and Compiler
CPU vendor CPU model Speed [MHz] # of
CPUs
CPU time Wall time CPU / Wall
ratio
% of ideal
v11-01-public
(C20T10)
Bob Yantosca
regal16.rc.fas.harvard.edu
ifort 11.1
GenuineIntel Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz 2199.915 8 62554.07 s
17:22:34
9355.80 s
02:35:59
6.6861 83.58
v11-02a-P/L
(C20T10)
Melissa Sulprizio
regal17.rc.fas.harvard.edu
ifort 11.1
GenuineIntel Intel(R) Xeon(R) CPU E5-2660 0 @ 2.20GHz 2199.993 8 55853.62 s
15:30:54
8373.14 s
02:19:44
1.12X faster than v11-01-public
6.6706 83.38

--Bob Yantosca (talk) 15:58, 9 February 2017 (UTC)

Saving out production & loss rates

The prod/loss rates from KPP are saved out in GEOS-Chem via the ND65 diagnostic. To activate this diagnostic, set the option for ND65 (stored in Input_Opt%DO_SAVE_PL) to T in the input.geos file:

  -------------------------------------------------------------------------------
  %%% PROD & LOSS MENU %%%:
  Turn on P/L (ND65) diag?: T
  # of levels for ND65    : 72
  Save O3 P/L (ND20)?     : F
  ------------------------+------------------------------------------------------

When Input_Opt%DO_SAVE_PL is true, FlexChem will call KPP routine ComputeFamilies (found in gckpp_Util.F90) to obtain the prod/loss rates stored in array FAM. The ND65 diagnostic is set up to automatically save out NFAM families to the output file and obtain the family names from FAM_NAMES, so no code modifications are required when adding prod/loss families to the chemical mechanism.

--Melissa Sulprizio (talk) 17:13, 9 November 2016 (UTC)

Validation

Benchmark simulations

FlexChem was implemented in v11-01g and validated with a 1-month benchmark and a 1-year benchmark. Please see the following links for more information:

  1. Approval form for 1-month benchmark simulation v11-01g
  2. Results for 1-year benchmark simulation v11-01g-Run0

--Melissa Sulprizio (talk) 18:38, 9 November 2016 (UTC)

Preliminary timing tests

Seven-day time tests were performed with the FlexChem code in order to evaluate model performance. The time tests all used 4° x 5° GEOS-FP met fields and were assigned 8 CPUs. Two model version were evaluated: v11-01f and v11-01g (which includes FlexChem). The results for the standard mechanism and tropchem mechanism are summarized below.


Standard Chemistry Mechanism

Model Version Wall Time CPU Time CPU Time / Wall Time Memory
v11-01f 1:43:15 10:55:22 6.3474 (79.34% ideal) 6.32 GB
v11-01g 2:10:37 14:19:33 6.5807 (82.26% ideal) 5.00 GB


Tropchem Chemistry Mechanism

Model Version Wall Time CPU Time CPU Time / Wall Time Memory
v11-01f 0:48:23 5:15:25 6.5191 (81.49% ideal) 3.57 GB
v11-01g 0:50:24 5:37:04 6.6878 (83.60% ideal) 2.70 GB

The GEOS-Chem Support Team will continue to evaluate and improve the performance of the FlexChem code for the GEOS-Chem v11-01 release.

--Melissa Sulprizio (talk) 16:26, 7 September 2016 (UTC)

Previous issues that are now resolved

Fix for compiling with CHEM=Custom

This fix was included in GEOS-Chem v11-01 public release

Prasad Kasibhatla wrote:

I am trying to create a custom simulation with some chemistry added to the SOA_SVPOA simulation. I followed all the steps in the wiki to create the gckpp*F90 files and copy them to the Code.v11-01/KPP/Custom directory. I then compile from Code.v11-01 using the command
  make -j4 MET=geosfp GRID=4x5 CHEM=Custom
I noticed that during the compile, the KPP files being compiled are in the Standard KPP directory. So it looks like the CHEM=Custom option is not directing the compiler to the right place.
I solved the problem by adding the following to Makefile_header.mk in the Code directory:
  # %%%%% Test if Custom %%%%%
  REGEXP               :=(^[Cc][Uu][Ss][Tt][Oo][Mm])
  ifeq ($(shell [[ "$(CHEM)" =~ $(REGEXP) ]] && echo true),true)
     KPP_CHEM           :=Custom
     IS_CHEM_SET        :=1
  endif

--Melissa Sulprizio (talk) 22:43, 17 January 2017 (UTC)

Remove calls to UPDATE_SUN, UPDATE_RCONST from gckpp_Integrator.F90

This update was included in GEOS-Chem v11-01 provisional release

A slow down in GEOS-Chem run time was observed following the implementation of FlexChem in v11-01. To resolve this, a temporary workaround was implemented. This fix may be replaced with a more permanent solution in GEOS-Chem v11-01 public release.

Bob Yantosca wrote:

KPP automatically places calls to UPDATE_SUN and UPDATE_RCONST in the routines FunTemplate and JacTemplate (both in the gckpp_Integrator.F90 module). This assumes that you are not interfacing KPP into any other model, and that you will use KPP to compute the sun angles at each timestep.

We now call UPDATE_RCONST once per grid box before calling the KPP integrator. Also, because we use FAST-JX to get the photo rates, we no longer need to call UPDATE_SUN. These duplicate calls were causing a performance bottleneck, as UPDATE_RCONST was being called more than 7 million times per day.

We have removed the duplicate calls from the gckpp_Integrator.F90 modules in each of the chemistry mechanisms. But we will also need to make sure that when building KPP fresh from an equation file, that this step gets automatically added to the build sequence.

--Melissa Sulprizio (talk) 19:53, 9 January 2017 (UTC)

Incorrect species units used in routines UCX_NOX and UCX_H2SO4PHOT

This update was validated with the 1-month benchmark simulation v11-01j and approved on 03 Dec 2016

Seb Eastham reported an error in FlexChem_mod.F90 routine DO_FLEXCHEM, where UCX routines UCX_NOX and UCX_H2SO4PHOT are called when species concentrations are in the wrong units. Both routines expect species concentrations in units of kg but species are not converted from molec/cm3 to kg until after the calls.

Seb Eastham wrote:

The consequences of this error can actually be seen in the v11-01g benchmark zonal mean concentrations. N2O should be uniform in the trop and then lost in the strat, but if you check out the color bar you’ll see that there is a very difficult-to-see (but huge!) maximum in the mesosphere. This is why the concentration in the trop, which should be the maximum, is so low on the color scale.

To correct this issue, the UCX routines are now called after the species unit conversion to kg.

--Lizzie Lundgren (talk) 22:01, 14 November 2016 (UTC)

Known issues

TBD