ISORROPIA II

Overview

Brief description

The ISORROPIA II package performs aerosol thermodynamical equilibrium. It partitions nitrate (HNO3 and NIT) and ammonia (NH3 and NH4) between the gas and aerosol phases. Inputs to the partitioning routine include temperature and RH. ISORROPIA II has significant benefits over previous implementations of ISORROPIA, especially for partitioning of nitrate at low RH.

From Pye et al [2009]:

ISORROPIA II [Fountoukis and Nenes, 2007] is implemented in GEOS-Chem to compute gas-aerosol equilibrium partitioning of nitric acid and ammonia. Particles in this study are not size-resolved; however, they can be generally assumed to represent PM2.5 since formation of sulfate-nitrate-ammonium on coarse mode sea salt and dust is excluded. Submicrometer-sized particles are likely to reach gas-aerosol equilibrium on time-scales less than the 1 hour computational time step used here [Meng and Seinfeld, 1996].

Sodium and chloride from accumulation mode sea salt are considered in the gas-aerosol equilibrium along with sulfate, nitrate, and ammonium. Calcium, magnesium, and potassium concentrations are not considered in the present study because of the issues with dust emissions previously mentioned. All inorganic aerosols are assumed to exist on the upper, metastable branch of the hygroscopic hysterisis curve. Although this assumption may not hold at higher altitudes in the free troposphere [Wang et al., 2008], since the focus of this study is mainly on surface-level concentrations, where humidities reach high values on a daily basis, the metastable assumption is acceptable.

Authors and collaborators

• Thanos Nenes (Georgia Tech) -- Principal Investigator
• Havala O. T. Pye (Caltech)

--Bob Y. 12:48, 2 March 2010 (EST)

Implementation notes

ISORROPIA II has been implemented into GEOS-Chem v8-03-01.

Code structure

The main-level Code directory has now been divided into several subdirectories:

GeosCore/    GEOS-Chem "core" routines
GeosTomas/   Parallel copies of GEOS-Chem routines that reference TOMAS
GeosUtil/    "Utility" modules (e.g. error_mod.f, file_mod.f, time_mod.f, etc.
Headers/     Header files (define.h, CMN_SIZE, CMN_DIAG, etc.)
ISOROPIA/    Directory where ISORROPIA II code resides   
KPP/         KPP solver directory structure
bin/         Directory where executables are placed
doc/         Directory where documentation is created
help/        Directory for GEOS-Chem Help Screen
lib/         Directory where library files are placed
mod/         Directory where module files are placed
obsolete/    Directory where obsolete versions of code are archived

ISORROPIA II consists of the following files:

Files in ISOROPIA/ subdirectory:
---------------------------------
Makefile            Makefile for ISORROPIA II code
isoropiaIIcode.f    Source code file with ISORROPIA II routines
isrpia.inc          ISOROPIA II header file with common blocks

Files in GeosCore/ subdirectory:
--------------------------------
isoropiaII_mod.f    "Interface" code between GEOS-Chem and ISORROPIA II

Additional Documentation

Havala O. T. Pye wrote:

1. Documentation (including a user manual) for ISORROPIAII can be found on the ISORROPIA website.
2. isoropiaIIcode.f is essentially ISOFWD.FOR and ISOCOM.FOR of the ISORROPIA II box model pasted together.
3. For more information on ISORROPIA II, see
   • Fountoukis, C. and Nenes, A.: ISORROPIA II: a computationally efficient thermodynamic equilibrium model for K+–Ca2+–Mg2+–NH4+–Na+–SO42−–NO3−–Cl−–H2O aerosols, Atmos. Chem. Phys., 7, 4639-4659, 2007. pdf
4. The implementation by Pye et al. 2009 JGR did not include Ca, K, or Mg since dust emissions were not used.

--Bob Y. 12:28, 17 March 2010 (EDT)

Validation

See Pye et al [2009]

References

1. Fountoukis, C., and A. Nenes (2007), ISORROPIA II: A computationally efficient thermodynamic equilibrium model for K+-Ca²⁺-Mg²⁺-NH₄⁺-Na⁺-SO₄^2-NO₃^--Cl-H₂O aerosols, Atmos. Chem. Phys., 7(17), 4639-4659.
2. Meng, Z. Y., and J. H. Seinfeld, Time scales to achieve atmospheric gas-aerosol equilibrium for volatile species, Atmos. Environ., 30(16), 28892900, doi:10.1016/1352-2310(95)00493-9., 1996.
3. Pye, H. O. T., H. Liao, S. Wu, L. J. Mickley, D. J.Jacob, D. K. Henze, and J. H. Seinfeld, Effect of changes in climate and emissions on future sulfate-nitrate-ammonium aerosol levels in the United States, J. Geophys. Res., 114, D01205, 2009. PDF
4. Wang, J., A. A. Hoffman, R. J. Park, D. Jacob, and S. T. Martin, Global distribution of solid and aqueous sulfate aerosols: Effect of the hysteresis of particle phase transitions, J. Geophys. Res., 113, D11206, doi:10.1029/2007JD009367, 2008. PDF

--Bob Y. 16:00, 22 February 2010 (EST)

Previous issues now resolved

Patches for v8-03-01

The following issues in ISORROPIA II have now been fixed as of August 2010:

1. Bug fix in ISORROPIA for offline aerosol
2. Patch to fix declaration of IONIC in ISORROPIA II
3. Bug fix for ND42 diagnostic and ISORROPIA II

--Bob Y. 14:44, 3 August 2010 (EDT)

Minor fixes in isorropiaIIcode.f

Havala Pye wrote

There are 2 minor bug fixes for ISORROPIA II. There are two locations in file ISOROPIA/isorropiaIIcode.f where the variable MOLALR(4) should be MOLALR(9). These fixes affect how aerosol phase water is calculated. Since GEOS-Chem typically considers sodium and chloride, these subcases are likely not being used, but the code should be updated in case these subcases are used in the future.

This fix will be implemented in GEOS-Chem v9-01-02.

--Bob Y. 11:10, 1 March 2011 (EST)

Optimization and/or parallelization issues in ISORROPIA II

During the benchmarking process for GEOS-Chem v9-01-02, we discovered that ISORROPIA does not produce exactly the same output on single processor compared to multi-processor:

Daven Henze wrote:

There is an unidentified OpenMP error which contributes to differences shown [in the v9-01-02a 1-month benchmark], as confirmed by independent tests of consecutive evaluations of a single executable compiled with and without parallelization. The impacts of this OpenMP error are typically less than a fraction of a percent, and negligible in terms of absolute value, with the exception of NIT and NH3. This is an issue to investigate further.

Bob Yantosca wrote:

Here is a quick update on the issue in ISORROPIA.

1. If you compile with –O0 (all optimization turned off), you get identical results when using ISORROPIA as single vs. multi processor.
2. When you compile with –O1, then you get different results when using ISORROPIA as single vs. multi processor. This inclines me to believe that the ISORROPIA code is sensitive to optimization.
3. It appears that issue is occurring at the computation of the GNH3, GHNO3, and GHCL variables in function FUNCG5A within file isoropiaIIcode.f. These variables are computed as:

     GNH3      = MAX(CHI4 - PSI4, TINY)              ! Gas NH3
     GHNO3     = MAX(CHI5 - PSI5, TINY)              ! Gas HNO3
     GHCL      = MAX(CHI6 - PSI6, TINY)              ! Gas HCl

Here the CHI* and PSI* are common-block variables and TINY is a small number (1d-20, but I used 1d-28 for testing). The CHI* and PSI* values are held threadprivate for the OpenMP parallelization. It looks as if there is some kind of roundoff error at the limit of precision that is creeping in here that could be caused by the optimization. For example, at grid box (37,21,11) every once and a while we get this roundoff error:

   with single processor
   ### CHI5 :   4.326986976485327E-008
   ### PSI5 :   4.312588684524357E-008
   ### GHNO3:   1.439829196097068E-010

   with multi processor
   ### CHI5 :   4.326986976485327E-008
   ### PSI5 :   4.312588684524356E-008
   ### GHNO3:   1.439829196097134E-010

with similar roundoff errors (not shown here) happening in the computation of GNH3.

GHNO3 gets saved back into the tracer array STT(I,J,L,IDTHNO3) and GNH3 gets saved back into STT(I,J,L,IDTNH3). So that is why these we see these very small (and seemingly random) differences in HNO3, NH3, and NIT, whereas the other tracers are unaffected.

I don’t have a good sense of why precisely this is occurring, other than to say that ISORROPIA uses a lot of internal common blocks and other obsolete coding practices. The ISORROPIA code is extremely confusing to follow. My guess is that either something is not being held private for the parallelization, or that variables in common blocks are not being initialized properly.

Solution

Shannon Capps wrote:

Please see this attached report that documents my replication of the issue with both the GEOS-Chem and ANISORROPIA versions of ISORROPIA II in a box model....I would propose adding a flag (-fp-model source) for the compilation of ISORROPIA only. Add this to the file ISOROPIA/Makefile as follows:

   isoropiaIIcode.o:  isoropiaIIcode.F  isrpia.inc
           $(F90) $(R8) -fp-model source -c $<

Compiling the GEOS-Chem implementation of ISORROPIA with this flag causes the box model to produce results that "diff" finds identical in all combinations (e.g., m0 v. s0, m3 v. s0, etc.).

The -fp-source option of the Intel Fortran Compiler tells the compiler to only do "safe" optimizations (i.e. nothing that would affect the numerical precision of the output). For some reason, ISORROPIA II is very sensitive to the compiler optimization.

Bob Yantosca has done some additional 3-day test runs to assess the impact of using the -fp-model source option. The results are:

Run #	Machine	# CPUs	Wall time [mm:ss]	Scalability	Mean OH [1e5 molec/cm3]
1	terra-03-c.as.harvard.edu	4	18:10	3.5935	13.0384629976933
2	terra-04-c.as.harvard.edu	4	09:58	3.8123	13.0384629976933
3	terra-03-c.as.harvard.edu	4	16:38	3.7546	13.0384629976933
4	terra-04-c.as.harvard.edu	4	11:00	3.5090	13.0384629976933
5	terra-11-c.as.harvard.edu	8	09:29	7.3364	13.0384629976933
6	terra-06-c.as.harvard.edu	8	06:18	7.1421	13.0384629976933
7	terra-11-c.as.harvard.edu	8	09:10	7.5530	13.0384629976933
8	terra-01-c.as.harvard.edu	8	05:52	7.4549	13.0384629976933

All of the output files from runs 1-8 were 100% binary identical to each other. Therefore, using -fp-model source seems to eliminate the random numerical noise generated by ISORROPIA.

Update 8/26/11: In GEOS-Chem v9-01-02 we shall apply the -fp-model source option to all source code files, not just ISORROPIA. This will help us to ensure the precision of the results as we make widespread changes for ESMF compatibility.

Update 9/1/11: Oops, this may not be solved after all. Runs longer than 3 days still manifest the problem. See this web page for more details.

--Bob Y. 15:07, 1 September 2011 (EDT)

Outstanding issues

Aerosol pH

This issue is somewhat resolved.

Havala O. T. Pye wrote:

At the GEOS-Chem User's meeting (2009) Becky Alexander and her student Eric noted that ISORROPIA II sometimes returns a negative H+ concentration. In my tests, the negative values occurred in N. India and Western Russia area during limited times of the year. A quick fix has been implemented. A condition I tracked down was returning a negative [H+] when solving the HSO4 = H + SO4 equilibrium (CALCHS4 in isorropiaIIcode.f) when the actual answer should have been ~1e-27 mol/m3. (I plugged the same input values into a spreadsheet and it returned the same negative answer so it seems like a numerical precision issue.) For my case study, it came down to subtracting two numbers of very similar value (first 8 digits the same) which resulted in a negative. I put in a fix to reset H+ to 1d-30 in CALCHS4 if it goes negative.

--Bob Y. 10:29, 29 January 2010 (EST)

Parallelization issue for nested grid simulations

For those of you who are running one of the GEOS-Chem nested grid simulations, you may encounter a segmentation fault error. This error is caused by GEOS-Chem running up against the memory limits of the machine. One way to solve this problem is to compile ISORROPIA II for single processor only:

Sungshik Patrick Kim wrote:

I tracked down the memory bottlenecks in my [China nested-grid simulation]....The crashes then were caused by memory allocation in ISORROPIA (with all of the private variables that had to be copied), so after turning off the parallelization for the ISORROPIA driver routine, I can actually spin up the model!

To compile ISORROPIA II for single processor, while compiling everything else for multi-processor, you can add an extra comment character ! to the !$OMP commands in GeosCore/isoropiaII_mod.f, such as:

!!$OMP PARALLEL DO
!!$OMP+DEFAULT( SHARED )
!!$OMP+PRIVATE( I,    J,      L,       N,      WI,   WT,  GAS,  TEMPI )
!!$OMP+PRIVATE( RHI,  VOL,    TSO4,    TNH3,   TNA,  TCL, ANO3, GNO3  )
!!$OMP+PRIVATE( TCA,  TMG,    TK,      CNTRL,  SCASI                  )
!!$OMP+PRIVATE( TNO3, AERLIQ, AERSLD,  OTHER,  TNH4, TNIT             )
!!$OMP+PRIVATE( HPLUSTEMP,    NUM_SAV, DEN_SAV, HNO3_DEN              )
!!$OMP+SCHEDULE( DYNAMIC )

...

!!$OMP END PARALLEL DO

--Bob Y. 10:23, 8 April 2011 (EDT)