ISORROPIA II
Overview
Brief description
The ISORROPIA II package performs aerosol thermodynamical equilibrium. It partitions nitrate (HNO3 and NIT) and ammonia (NH3 and NH4) between the gas and aerosol phases. Inputs to the partitioning routine include temperature and RH. ISORROPIA II has significant benefits over previous implementations of ISORROPIA, especially for partitioning of nitrate at low RH.
From Pye et al [2009]:
ISORROPIA II [Fountoukis and Nenes, 2007] is implemented in GEOS-Chem to compute gas-aerosol equilibrium partitioning of nitric acid and ammonia. Particles in this study are not size-resolved; however, they can be generally assumed to represent PM2.5 since formation of sulfate-nitrate-ammonium on coarse mode sea salt and dust is excluded. Submicrometer-sized particles are likely to reach gas-aerosol equilibrium on time-scales less than the 1 hour computational time step used here [Meng and Seinfeld, 1996].
Sodium and chloride from accumulation mode sea salt are considered in the gas-aerosol equilibrium along with sulfate, nitrate, and ammonium. Calcium, magnesium, and potassium concentrations are not considered in the present study because of the issues with dust emissions previously mentioned. All inorganic aerosols are assumed to exist on the upper, metastable branch of the hygroscopic hysterisis curve. Although this assumption may not hold at higher altitudes in the free troposphere [Wang et al., 2008], since the focus of this study is mainly on surface-level concentrations, where humidities reach high values on a daily basis, the metastable assumption is acceptable.
Authors and collaborators
- Thanos Nenes (Georgia Tech) -- Principal Investigator
- Havala O. T. Pye (Caltech)
--Bob Y. 12:48, 2 March 2010 (EST)
Implementation notes
ISORROPIA II has been implemented into GEOS-Chem v8-03-01.
Code structure
The main-level Code directory has now been divided into several subdirectories:
GeosCore/ GEOS-Chem "core" routines GeosTomas/ Parallel copies of GEOS-Chem routines that reference TOMAS GeosUtil/ "Utility" modules (e.g. error_mod.f, file_mod.f, time_mod.f, etc. Headers/ Header files (define.h, CMN_SIZE, CMN_DIAG, etc.) ISOROPIA/ Directory where ISORROPIA II code resides KPP/ KPP solver directory structure bin/ Directory where executables are placed doc/ Directory where documentation is created help/ Directory for GEOS-Chem Help Screen lib/ Directory where library files are placed mod/ Directory where module files are placed obsolete/ Directory where obsolete versions of code are archived
ISORROPIA II consists of the following files:
Files in ISOROPIA/ subdirectory: --------------------------------- Makefile Makefile for ISORROPIA II code isoropiaIIcode.f Source code file with ISORROPIA II routines isrpia.inc ISOROPIA II header file with common blocks Files in GeosCore/ subdirectory: -------------------------------- isoropiaII_mod.f "Interface" code between GEOS-Chem and ISORROPIA II
Additional Documentation
Havala O. T. Pye wrote:
- Documentation (including a user manual) for ISORROPIAII can be found on the ISORROPIA website.
- isoropiaIIcode.f is essentially ISOFWD.FOR and ISOCOM.FOR of the ISORROPIA II box model pasted together.
- For more information on ISORROPIA II, see
- Fountoukis, C. and Nenes, A.: ISORROPIA II: a computationally efficient thermodynamic equilibrium model for K+–Ca2+–Mg2+–NH4+–Na+–SO42−–NO3−–Cl−–H2O aerosols, Atmos. Chem. Phys., 7, 4639-4659, 2007. pdf
- The implementation by Pye et al. 2009 JGR did not include Ca, K, or Mg since dust emissions were not used.
--Bob Y. 12:28, 17 March 2010 (EDT)
Validation
See Pye et al [2009]
References
- Fountoukis, C., and A. Nenes (2007), ISORROPIA II: A computationally efficient thermodynamic equilibrium model for K+-Ca2+-Mg2+-NH4+-Na+-SO42-NO3--Cl-H2O aerosols, Atmos. Chem. Phys., 7(17), 4639-4659.
- Meng, Z. Y., and J. H. Seinfeld, Time scales to achieve atmospheric gas-aerosol equilibrium for volatile species, Atmos. Environ., 30(16), 28892900, doi:10.1016/1352-2310(95)00493-9., 1996.
- Pye, H. O. T., H. Liao, S. Wu, L. J. Mickley, D. J.Jacob, D. K. Henze, and J. H. Seinfeld, Effect of changes in climate and emissions on future sulfate-nitrate-ammonium aerosol levels in the United States, J. Geophys. Res., 114, D01205, 2009. PDF
- Wang, J., A. A. Hoffman, R. J. Park, D. Jacob, and S. T. Martin, Global distribution of solid and aqueous sulfate aerosols: Effect of the hysteresis of particle phase transitions, J. Geophys. Res., 113, D11206, doi:10.1029/2007JD009367, 2008. PDF
--Bob Y. 16:00, 22 February 2010 (EST)
Previous issues now resolved
Patches for v8-03-01
The following issues in ISORROPIA II have now been fixed as of August 2010:
- Bug fix in ISORROPIA for offline aerosol
- Patch to fix declaration of IONIC in ISORROPIA II
- Bug fix for ND42 diagnostic and ISORROPIA II
--Bob Y. 14:44, 3 August 2010 (EDT)
Minor fixes in isorropiaIIcode.f
Havala Pye wrote
- There are 2 minor bug fixes for ISORROPIA II. There are two locations in file ISOROPIA/isorropiaIIcode.f where the variable MOLALR(4) should be MOLALR(9). These fixes affect how aerosol phase water is calculated. Since GEOS-Chem typically considers sodium and chloride, these subcases are likely not being used, but the code should be updated in case these subcases are used in the future.
This fix will be implemented in GEOS-Chem v9-01-02.
--Bob Y. 11:10, 1 March 2011 (EST)
Optimization and/or parallelization issues in ISORROPIA II
During the benchmarking process for GEOS-Chem v9-01-02, we discovered that ISORROPIA does not produce exactly the same output on single processor compared to multi-processor:
Daven Henze wrote:
- There is an unidentified OpenMP error which contributes to differences shown [in the v9-01-02a 1-month benchmark], as confirmed by independent tests of consecutive evaluations of a single executable compiled with and without parallelization. The impacts of this OpenMP error are typically less than a fraction of a percent, and negligible in terms of absolute value, with the exception of NIT and NH3. This is an issue to investigate further.
Bob Yantosca wrote:
- Here is a quick update on the issue in ISORROPIA.
- If you compile with –O0 (all optimization turned off), you get identical results when using ISORROPIA as single vs. multi processor.
- When you compile with –O1, then you get different results when using ISORROPIA as single vs. multi processor. This inclines me to believe that the ISORROPIA code is sensitive to optimization.
- It appears that issue is occurring at the computation of the GNH3, GHNO3, and GHCL variables in function FUNCG5A within file isoropiaIIcode.f. These variables are computed as:
GNH3 = MAX(CHI4 - PSI4, TINY) ! Gas NH3
GHNO3 = MAX(CHI5 - PSI5, TINY) ! Gas HNO3
GHCL = MAX(CHI6 - PSI6, TINY) ! Gas HCl
- Here the CHI* and PSI* are common-block variables and TINY is a small number (1d-20, but I used 1d-28 for testing). The CHI* and PSI* values are held threadprivate for the OpenMP parallelization. It looks as if there is some kind of roundoff error at the limit of precision that is creeping in here that could be caused by the optimization. For example, at grid box (37,21,11) every once and a while we get this roundoff error:
with single processor ### CHI5 : 4.326986976485327E-008 ### PSI5 : 4.312588684524357E-008 ### GHNO3: 1.439829196097068E-010 with multi processor ### CHI5 : 4.326986976485327E-008 ### PSI5 : 4.312588684524356E-008 ### GHNO3: 1.439829196097134E-010
- with similar roundoff errors (not shown here) happening in the computation of GNH3.
- GHNO3 gets saved back into the tracer array STT(I,J,L,IDTHNO3) and GNH3 gets saved back into STT(I,J,L,IDTNH3). So that is why these we see these very small (and seemingly random) differences in HNO3, NH3, and NIT, whereas the other tracers are unaffected.
- I don’t have a good sense of why precisely this is occurring, other than to say that ISORROPIA uses a lot of internal common blocks and other obsolete coding practices. The ISORROPIA code is extremely confusing to follow. My guess is that either something is not being held private for the parallelization, or that variables in common blocks are not being initialized properly.
Solution
Shannon Capps wrote:
- Please see this attached report that documents my replication of the issue with both the GEOS-Chem and ANISORROPIA versions of ISORROPIA II in a box model....I would propose adding a flag (-fp-model source) for the compilation of ISORROPIA only. Add this to the file ISOROPIA/Makefile as follows:
isoropiaIIcode.o: isoropiaIIcode.F isrpia.inc
$(F90) $(R8) -fp-model source -c $<
- Compiling the GEOS-Chem implementation of ISORROPIA with this flag causes the box model to produce results that "diff" finds identical in all combinations (e.g., m0 v. s0, m3 v. s0, etc.).
The -fp-source option of the Intel Fortran Compiler tells the compiler to only do "safe" optimizations (i.e. nothing that would affect the numerical precision of the output). For some reason, ISORROPIA II is very sensitive to the compiler optimization.
Bob Yantosca has done some additional 3-day test runs to assess the impact of using the -fp-model source option. The results are:
| Run # | Machine | # CPUs | Wall time [mm:ss] | Scalability | Mean OH [1e5 molec/cm3] |
|---|---|---|---|---|---|
| 1 | terra-03-c.as.harvard.edu | 4 | 18:10 | 3.5935 | 13.0384629976933 |
| 2 | terra-04-c.as.harvard.edu | 4 | 09:58 | 3.8123 | 13.0384629976933 |
| 3 | terra-03-c.as.harvard.edu | 4 | 16:38 | 3.7546 | 13.0384629976933 |
| 4 | terra-04-c.as.harvard.edu | 4 | 11:00 | 3.5090 | 13.0384629976933 |
| 5 | terra-11-c.as.harvard.edu | 8 | 09:29 | 7.3364 | 13.0384629976933 |
| 6 | terra-06-c.as.harvard.edu | 8 | 06:18 | 7.1421 | 13.0384629976933 |
| 7 | terra-11-c.as.harvard.edu | 8 | 09:10 | 7.5530 | 13.0384629976933 |
| 8 | terra-01-c.as.harvard.edu | 8 | 05:52 | 7.4549 | 13.0384629976933 |
All of the output files from runs 1-8 were 100% binary identical to each other. Therefore, using -fp-model source seems to eliminate the random numerical noise generated by ISORROPIA.
Persistence of problem
Update 8/26/11: In GEOS-Chem v9-01-02 we shall apply the -fp-model source option to all source code files, not just ISORROPIA. This will help us to ensure the precision of the results as we make widespread changes for ESMF compatibility.
Update 9/1/11: Oops, this may not be solved after all. Runs longer than 3 days still manifest the problem. See File:Isorropia problem.pdf for more details.
--Bob Y. 15:32, 23 November 2011 (EST)
Outstanding issues
Aerosol pH
This issue is somewhat resolved.
Havala O. T. Pye wrote:
- At the GEOS-Chem User's meeting (2009) Becky Alexander and her student Eric noted that ISORROPIA II sometimes returns a negative H+ concentration. In my tests, the negative values occurred in N. India and Western Russia area during limited times of the year. A quick fix has been implemented. A condition I tracked down was returning a negative [H+] when solving the HSO4 = H + SO4 equilibrium (CALCHS4 in isorropiaIIcode.f) when the actual answer should have been ~1e-27 mol/m3. (I plugged the same input values into a spreadsheet and it returned the same negative answer so it seems like a numerical precision issue.) For my case study, it came down to subtracting two numbers of very similar value (first 8 digits the same) which resulted in a negative. I put in a fix to reset H+ to 1d-30 in CALCHS4 if it goes negative.
--Bob Y. 10:29, 29 January 2010 (EST)
Parallelization issue for nested grid simulations
For those of you who are running one of the GEOS-Chem nested grid simulations, you may encounter a segmentation fault error. This error is caused by GEOS-Chem running up against the memory limits of the machine. One way to solve this problem is to compile ISORROPIA II for single processor only:
Sungshik Patrick Kim wrote:
- I tracked down the memory bottlenecks in my [China nested-grid simulation]....The crashes then were caused by memory allocation in ISORROPIA (with all of the private variables that had to be copied), so after turning off the parallelization for the ISORROPIA driver routine, I can actually spin up the model!
To compile ISORROPIA II for single processor, while compiling everything else for multi-processor, you can add an extra comment character ! to the !$OMP commands in GeosCore/isoropiaII_mod.f, such as:
!!$OMP PARALLEL DO !!$OMP+DEFAULT( SHARED ) !!$OMP+PRIVATE( I, J, L, N, WI, WT, GAS, TEMPI ) !!$OMP+PRIVATE( RHI, VOL, TSO4, TNH3, TNA, TCL, ANO3, GNO3 ) !!$OMP+PRIVATE( TCA, TMG, TK, CNTRL, SCASI ) !!$OMP+PRIVATE( TNO3, AERLIQ, AERSLD, OTHER, TNH4, TNIT ) !!$OMP+PRIVATE( HPLUSTEMP, NUM_SAV, DEN_SAV, HNO3_DEN ) !!$OMP+SCHEDULE( DYNAMIC ) ... !!$OMP END PARALLEL DO
--Bob Y. 10:23, 8 April 2011 (EDT)