Difference between revisions of "Flexible precision in GEOS-Chem"

From Geos-chem
Jump to: navigation, search
(Previous issues that are now resolved)
 
(60 intermediate revisions by 2 users not shown)
Line 50: Line 50:
 
  !BOP
 
  !BOP
 
  !
 
  !
  ! !MODULE: precision_mod
+
  ! !MODULE: precision_mod.F
 
  !
 
  !
  ! !DESCRIPTION: Module precision\_mod is used to change the precision of
+
  ! !DESCRIPTION: Module PRECISION\_MOD is used to change the precision of
 
  !  many variables throughout GEOS-Chem at compile-time.
 
  !  many variables throughout GEOS-Chem at compile-time.
 
  !\\
 
  !\\
Line 76: Line 76:
 
   
 
   
 
       ! Use 8-byte floating point precision when asked.
 
       ! Use 8-byte floating point precision when asked.
       INTEGER, PARAMETER, PUBLIC :: fp = KIND( 0.d0 )
+
       INTEGER, PARAMETER, PUBLIC :: fp = KIND( REAL( 0.0, 8 ) )
 
   
 
   
 
  #else
 
  #else
 
   
 
   
 
       ! Use 4-byte floating point by default.
 
       ! Use 4-byte floating point by default.
       INTEGER, PARAMETER, PUBLIC :: fp = KIND( 0.0 )
+
       INTEGER, PARAMETER, PUBLIC :: fp = KIND( REAL( 0.0, 4 ) )
 
   
 
   
 
  #endif
 
  #endif
Line 94: Line 94:
 
   
 
   
 
       ! KIND parameter for 4-byte precision
 
       ! KIND parameter for 4-byte precision
       INTEGER, PARAMETER, PUBLIC :: f4 = KIND( 0.0 )
+
       INTEGER, PARAMETER, PUBLIC :: f4 = KIND( REAL( 0.0, 4 ) )
 
        
 
        
 
       ! KIND parameter for 8-byte precision
 
       ! KIND parameter for 8-byte precision
       INTEGER, PARAMETER, PUBLIC :: f8 = KIND( 0.d0 )
+
       INTEGER, PARAMETER, PUBLIC :: f8 = KIND( REAL( 0.0, 8 ) )
 
  !
 
  !
 
  ! !REMARKS:
 
  ! !REMARKS:
Line 104: Line 104:
 
  ! !REVISION HISTORY:
 
  ! !REVISION HISTORY:
 
  !  (1 ) Created. (myannetti, 11/04/14)
 
  !  (1 ) Created. (myannetti, 11/04/14)
 +
!  23 Nov 2016 - R. Yantosca - Now rewrite KIND definitions to prevent 4-byte
 +
!                              and 8-byte variables from being elevated
 +
!                              when using -r8 (or equivalent flags)
 
  !EOP
 
  !EOP
 
  !-----------------------------------------------------------------------------
 
  !-----------------------------------------------------------------------------
Line 112: Line 115:
 
Instead of having to figure out the proper settings with <tt>SELECTED_REAL_KIND</tt>, we can just use the KIND command to return that for us.   
 
Instead of having to figure out the proper settings with <tt>SELECTED_REAL_KIND</tt>, we can just use the KIND command to return that for us.   
  
*<tt>KIND( 0.d0 )</tt> returns the proper "kind" value to define an 8-byte floating point
+
*<tt>KIND( REAL( 0.0, 8 ) )</tt> returns the proper "kind" value to define an 8-byte floating point
*<tt>KIND( 0.0 )</tt> returns the proper "kind" value to define a 4-byte floating point
+
*<tt>KIND( REAL( 0.0, 4 ) )</tt> returns the proper "kind" value to define a 4-byte floating point
  
 
This value returned by the <tt>KIND</tt> function is saved in the constant named <tt>fp</tt>.  (NOTE: This stands for "flexible precision".  It does not have any connection to the [[GEOS-FP]] meteorology.)
 
This value returned by the <tt>KIND</tt> function is saved in the constant named <tt>fp</tt>.  (NOTE: This stands for "flexible precision".  It does not have any connection to the [[GEOS-FP]] meteorology.)
Line 172: Line 175:
  
  
'''5.''' Look for all variables in the module that are declared as REAL*8.  Replace the REAL*8 text with REAL(fp) instead.  So, in the above example, these lines:
+
'''5.''' Look for all variables in the module that are declared as <tt>REAL*8</tt>.  Replace the <tt>REAL*8</tt> text with <tt>REAL(fp)</tt> instead.  So, in the above example, these lines:
  
       REAL*8, ALLOCATABLE :: ANTH_BLKC(:,:,:)
+
       <span style="color:red">REAL*8</span>, ALLOCATABLE :: ANTH_BLKC(:,:,:)
       REAL*8, ALLOCATABLE :: ANTH_ORGC(:,:,:)
+
       <span style="color:red">REAL*8</span>, ALLOCATABLE :: ANTH_ORGC(:,:,:)
       REAL*8, ALLOCATABLE :: BIOB_BLKC(:,:,:)
+
       <span style="color:red">REAL*8</span>, ALLOCATABLE :: BIOB_BLKC(:,:,:)
  
 
would become  
 
would become  
  
       REAL(fp), ALLOCATABLE :: ANTH_BLKC(:,:,:)
+
       <span style="color:green">REAL(fp)</span>, ALLOCATABLE :: ANTH_BLKC(:,:,:)
       REAL(fp), ALLOCATABLE :: ANTH_ORGC(:,:,:)
+
       <span style="color:green">REAL(fp)</span>, ALLOCATABLE :: ANTH_ORGC(:,:,:)
       REAL(fp), ALLOCATABLE :: BIOB_BLKC(:,:,:)
+
       <span style="color:green">REAL(fp)</span>, ALLOCATABLE :: BIOB_BLKC(:,:,:)
  
  
Line 188: Line 191:
  
 
       ! Molecules OH  per kg OH [molec/kg]
 
       ! Molecules OH  per kg OH [molec/kg]
       REAL*8,  PARAMETER  :: XNUMOL_OH  = 6.022d23 / 17d-3
+
       REAL*8,  PARAMETER  :: XNUMOL_OH  = 6.022<span style="color:red">d23</span> / 17<span style="color:red">d-3</span>
       REAL*8,  PARAMETER  :: CM3PERM3  = 1.d6
+
       REAL*8,  PARAMETER  :: CM3PERM3  = 1.<span style="color:red">d6</span>
 
       REAL*8,  PARAMETER  :: TINY      = TINY(1.0)
 
       REAL*8,  PARAMETER  :: TINY      = TINY(1.0)
  
Line 195: Line 198:
  
 
       ! Molecules OH  per kg OH [molec/kg]
 
       ! Molecules OH  per kg OH [molec/kg]
       REAL(fp), PARAMETER :: XNUMOL_OH  = 6.022e+23_fp / 17e-3_fp
+
       REAL(fp), PARAMETER :: XNUMOL_OH  = 6.022<span style="color:green">e+23_fp</span> / 17<span style="color:green">e-3_fp</span>
       REAL(fp), PARAMETER :: CM3PERM3  = 1.e+6_fp
+
       REAL(fp), PARAMETER :: CM3PERM3  = 1.<span style="color:green">e+6_fp</span>
       REAL(fp), PARAMETER :: TINY      = TINY(1.0_fp)
+
       REAL(fp), PARAMETER :: TINY      = TINY(1.0<span style="color:green">_fp</span>)
  
 
'''7.''' We repeated the process in steps 4-6 for each GEOS-Chem source code file.  We typically modify a file or two at a time, and then run a difference test.  A difference test compares the code we are editing to a code with known behavior, such as the last accepted benchmarked version.
 
'''7.''' We repeated the process in steps 4-6 for each GEOS-Chem source code file.  We typically modify a file or two at a time, and then run a difference test.  A difference test compares the code we are editing to a code with known behavior, such as the last accepted benchmarked version.
  
--[[User:Bmy|Bob Y.]] 15:35, 7 November 2014 (EST)
+
--[[User:Bmy|Bob Yantosca]] ([[User talk:Bmy|talk]]) 18:10, 23 November 2016 (UTC)
  
 
== Compiling GEOS-Chem ==
 
== Compiling GEOS-Chem ==
  
To compile GEOS-Chem for 8-byte floating-point precsion, just use the same commands as you always would:
+
To compile GEOS-Chem for 8-byte floating-point precision, just use the same commands as you always would:
  
 
  make -j4 MET=geosfp GRID=4x5 TRACEBACK=y ...
 
  make -j4 MET=geosfp GRID=4x5 TRACEBACK=y ...
  
(The ... denotes other compiler options, as described in [http://acmg.seas.harvard.edu/geos/doc/man/chapter_3.html Ch. 3 of the GEOS-Chem manual].)
+
(The ... denotes other compiler options, as described in [[Compiling GEOS-Chem]].)
  
 
But to compile with 4-byte floating-point precision, you must now use the <tt>PRECISION</tt> keyword:
 
But to compile with 4-byte floating-point precision, you must now use the <tt>PRECISION</tt> keyword:
  
  make -j4 MET=geosfp GRID=4x5 TRACEBACK=y PRECISION=4 ...
+
  make -j4 MET=geosfp GRID=4x5 TRACEBACK=y <span style="color:green">PRECISION=4</span> ...
  
 
Eventually, <tt>PRECISION=4</tt> will be automatically set if you build GEOS-Chem with the <tt>hpc</tt> option.  This option compiles GEOS-Chem for use with the ESMF environment (such as is used in the GEOS-5 GCM).  Then you can just use this command:
 
Eventually, <tt>PRECISION=4</tt> will be automatically set if you build GEOS-Chem with the <tt>hpc</tt> option.  This option compiles GEOS-Chem for use with the ESMF environment (such as is used in the GEOS-5 GCM).  Then you can just use this command:
Line 220: Line 223:
  
 
--[[User:Bmy|Bob Y.]] 15:41, 7 November 2014 (EST)
 
--[[User:Bmy|Bob Y.]] 15:41, 7 November 2014 (EST)
 +
 +
== Validation ==
 +
 +
The following sections summarize our evaluation of GEOS-Chem with the <tt>PRECISION=4</tt> option:
 +
 +
=== Benchmarks ===
 +
 +
Bob Yantosca ([[GCST]]) performed a 1-month benchmark simulation with the <tt>PRECISION=4</tt> option.  The results are posted on
 +
 +
Basic takeaways: The 1-month benchmark simulation with <tt>PRECISION=4</tt>:
 +
 +
# Used about 40% less memory than the [[GEOS-Chem v11-01 benchmark history#v11-01f|v11-01f 1-month benchmark simulation (compiled with <tt>PRECISION=8</tt>)]].
 +
# Finished 20 minutes faster than the v11-01f 1-month benchmark simulation.
 +
# Shows negligible differences w/r/t the v11-01f 1-month benchmark.
 +
 +
--[[User:Bmy|Bob Yantosca]] ([[User talk:Bmy|talk]]) 17:48, 22 February 2017 (UTC)
 +
 +
=== Mass conservation ===
 +
 +
The plots below illustrate the evolution of total CO2 tracer mass in <tt>geosfp_2x25_masscons</tt> simulations (met year 2013) using GEOS-Chem "Classic".  The <span style="color:red">'''RED LINE'''</span> indicates the initial total mass, for reference.
 +
 +
'''Figure 1: Using <tt>PRECISION=4</tt> and [[Boundary_layer_mixing#TURBDAY|TURBDAY]] (aka full) PBL mixing'''
 +
 +
When the <tt>PRECISION=4</tt> option is activated, we observe a consistent loss of mass starting after approximately one month.  Differences start appearing in the 3rd decimal place.
 +
 +
TAU range    [hrs since 1/1/1985] :        245454.00000        254208.00000
 +
Initial mass [kg                ] : 2.87<span style="color:green">55168</span>0000000e+15
 +
Mass range  [kg                ] : 2.87<span style="color:red">48114</span>0000000e+15 2.87<span style="color:blue">56022</span>0000000e+15
 +
 +
[[Image:Mass_cons_prec4_1yr.png]]
 +
 +
'''Figure 2: Using <tt>PRECISION=8</tt> and [[Boundary_layer_mixing#TURBDAY|TURBDAY]] (aka full) PBL mixing'''
 +
 +
With <tt>PRECISION=8</tt>, differences start to appear in the 12th decimal place.  This indicates numerical noise around a constant total tracer mass.
 +
 +
TAU range    [hrs since 1/1/1985] :        245454.00000        254208.00000
 +
Initial mass [kg                ] : 2.87745967221<span style="color:green">413</span>e+15
 +
Mass range  [kg                ] : 2.87745967221<span style="color:red">277</span>e+15 2.87745967221<span style="color:blue">468</span>e+15
 +
 +
[[Image:Mass_cons_prec8_1yr.png]]
 +
 +
'''Figure 3: Using <tt>PRECISION=4</tt> and [[Boundary_layer_mixing#VDIFF|VDIFF]] (aka non-local) PBL mixing'''. 
 +
 +
Using <tt>PRECISION=4</tt> and the non-local boundary layer mixing scheme (VDIFF) results in a similar picture to Figure 1.  There is a consistent, significant mass loss with time.  Again, differences start to appear in the 3rd decimal place.
 +
 +
TAU range    [hrs since 1/1/1985] :        245454.00000        254208.00000
 +
Initial mass [kg                ] : 2.87<span style="color:green">55171</span>0000000e+15
 +
Mass range  [kg                ] : 2.87<span style="color:red">44329</span>0000000e+15 2.87<span style="color:blue">55509</span>0000000e+15
 +
 +
[[Image:Mass_cons_prec4_1yr_vdiff.png]]
 +
 +
'''Figure 4: Using <tt>PRECISION=8</tt> and [[Boundary_layer_mixing#VDIFF|VDIFF]] (aka non-local) PBL mixing'''.
 +
 +
Using <tt>PRECISION=8</tt> and the non-local boundary layer mixing scheme (VDIFF), we see mass conservation to the 8th decimal place.  This indicates numerical noise around a constant tracer mass.
 +
 +
TAU range    [hrs since 1/1/1985] :        245454.00000        254208.00000
 +
Initial mass [kg                ] : 2.87745967<span style="color:green">264367</span>e+15
 +
Mass range  [kg                ] : 2.87745967<span style="color:red">257390</span>e+15 2.87745967<span style="color:blue">694125</span>e+15
 +
 +
[[Image:Mass_cons_prec8_1yr_vdiff.png]]
 +
 +
'''Summary'''
 +
 +
The plots above show that mass conservation is better achieved with the <tt>PRECISION=8</tt> option than with <tt>PRECISION=4</tt>. 
 +
 +
'''''[[User:Sebastian D. Eastham|Seb Eastham]] wrote:'''''
 +
 +
<blockquote>I would also advise that we tell users who want to do a simulation involving long-lived species (think stratosphere or CO2) that they might want to consider using <tt>PRECISION=8</tt>, which is the current GEOS-Chem default setting. One concern with the mass conservation error in <tt>PRECISION=4</tt> (or actually at any precision) is that I think it will be a stationary random variable&mdash;there's no guarantee that the mean of the error over time will tend towards zero. This isn't a problem when the error at each step is ~1e-10, but could be a bigger problem when it's 1e-5 because we have no guarantee that it won't accumulate in one direction out of random bad luck.</blockquote>
 +
 +
'''''Daniel Jacob replied:'''''
 +
 +
<blockquote>The performance gains from using <tt>PRECISION=4</tt> are most relevant for full-chemistry applications, not for CO2 applications.</blockquote>
 +
 +
'''''[[User:bmy|Bob Yantosca]] replied:'''''
 +
 +
<blockquote>We could...set <tt>PRECISION=4</tt> as the default option when you generate any of the full-chemistry run directories (Standard, Tropchem, UCX, SOA, SOA-SVPOA, MarinePOA, Aciduptake).  Then for other specialty simulations with long-lived species (CO2, CH4, maybe Hg, etc.), we could leave those as <tt>PRECISION=8</tt>.</blockquote>
 +
 +
'''''[[User:Sebastian D. Eastham|Seb Eastham]] replied:'''''
 +
 +
<blockquote>I would caution that there are still some species relevant to chemistry simulations which are long-lived and could be affected by mass balance – I’m thinking CFCs and N2O in particular. As long as users interested in these species are aware that there might be a random mass drift of ~0.01% per year, that should be fine.</blockquote>
 +
 +
--[[User:Bmy|Bob Yantosca]] ([[User talk:Bmy|talk]]) 20:43, 24 February 2017 (UTC)
 +
 +
=== Mass conservation implications for GCHP ===
 +
 +
[[User:Sebastian D. Eastham|Seb Eastham]] plotted the evolution of the total tracer mass versus time in [[GEOS-Chem HP|GCHP]] for several configurations, as shown below.
 +
 +
*The <span style="color:blue">BLUE</span> line represents a GCHP simulation using <tt>PRECISION=4</tt>.
 +
*The <span style="color:orange">LIGHT ORANGE</span> line represents a GCHP simulation using the default <tt>PRECISION=8</tt>.
 +
 +
[[Image:Transport_error_v3_gchp.png]]
 +
 +
As you can see, the <span style="color:blue">GCHP simulation using the <tt>PRECISION=4</tt> option</span> causes a significant loss of mass, similar to the GEOS-Chem classic simulations from [[#Mass conservation|the preceding section]].  On the other hand, the <span style="color:orange">simulation using <tt>PRECISION=8</tt></span> results in nearly perfect mass conservation.
 +
 +
Seb writes:
 +
 +
<blockquote>A fairly major structural change is needed to ensure mass conservation (specifically, the [GCHP] internal state must be changed from <code>REAL*4</code> to <code>REAL*8</code>). This fix also invalidates "strong reproducibility"; there will be machine-precision-level differences in output when the code is run with different numbers of cores. This basically comes down to a choice; we can have two of the following:
 +
 +
#Mass conservation
 +
#Strong reproducibility
 +
#Speed
 +
 +
I thought 1 and 3 were most important but I figure that it's useful for us all to be aware.</blockquote>
 +
 +
--[[User:Bmy|Bob Yantosca]] ([[User talk:Bmy|talk]]) 15:17, 24 February 2017 (UTC)
  
 
== Previous issues that are now resolved ==
 
== Previous issues that are now resolved ==
Line 225: Line 333:
 
=== Update KIND parameters to facilitate interface with the Beijing Climate Center model ===
 
=== Update KIND parameters to facilitate interface with the Beijing Climate Center model ===
  
<span style="color:darkorange">'''''These updates will be added to the [[GEOS-Chem v11-01#v11-01 provisional release|v11-01 Provisional release]].'''''</span>
+
<span style="color:green">'''''This update was included in [[GEOS-Chem v11-01 benchmark history#v11-01k|v11-01k]] and approved on 19 Dec 2016'''''</span>
  
 
'''''Mike Long wrote:'''''
 
'''''Mike Long wrote:'''''
Line 231: Line 339:
 
<blockquote>With the flexible precision implementation, GEOS-Chem is unable to compile when the Intel Fortran Compiler flag <tt>-r8</tt> is thrown in the compiler command. The <tt>-r8</tt> flag forces variables declared as <tt>REAL</tt> or <tt>KIND(0.0)</tt> to be double precision by default. I learned this because the Beijing Climate Model uses the <tt>-r8</tt> flag by default, and cannot compile without it.
 
<blockquote>With the flexible precision implementation, GEOS-Chem is unable to compile when the Intel Fortran Compiler flag <tt>-r8</tt> is thrown in the compiler command. The <tt>-r8</tt> flag forces variables declared as <tt>REAL</tt> or <tt>KIND(0.0)</tt> to be double precision by default. I learned this because the Beijing Climate Model uses the <tt>-r8</tt> flag by default, and cannot compile without it.
  
The result is a multitude of errors that seem intractable. The solution I found was simple, but it may not work altogether -- i.e. it's implications haven't been tested yet. By setting the "f4" definition in precision_mod and "sp" definition in hco_error_mod from <tt>KIND( 0.0 )</tt> to
+
The result is a multitude of errors that seem intractable. The solution I found was simple, but it may not work altogether -- i.e. it's implications haven't been tested yet. By setting the "f4" definition in precision_mod and "sp" definition in hco_error_mod from <tt>KIND( 0.0 )</tt> to <tt>KIND( REAL( 0.0, 4 ) )</tt> all variables declared as either f4 or sp are allowed to be single precision.</blockquote>
<tt>KIND( REAL( 0.0, 4 ) )</tt> all variables declared as either f4 or sp are allowed to be single precision.</blockquote>
+
  
 
'''''[[User:Bmy|Bob Yantosca]] replied:'''''
 
'''''[[User:Bmy|Bob Yantosca]] replied:'''''
  
<blockquote>I tested Mike's proposed fix on the [[Intel Fortran Compiler|ifort]], [[PGI Fortran compiler|pgfortran]], and [[GNU Fortran compiler|gfortran]] compilers and confirmed that Mike's proposed fix is the proper one.  We should use:</blockquote>
+
<blockquote>I tested Mike's proposed fix on the [[Intel Fortran Compiler|ifort]], [[PGI Fortran compiler|pgfortran]], and [[GNU Fortran compiler|gfortran]] compilers and confirmed that it is the proper one.  We should use:</blockquote>
  
     KIND( REAL( 0.0, 4 ) ) instead of KIND( 0.0  )  
+
     KIND( REAL( <span style="color:green">0.0, 4</span> ) ) instead of KIND( <span style="color:red">0.0</span>   )  
     KIND( REAL( 0.0, 8 ) ) instead of KIND( 0.0d0 )  
+
     KIND( REAL( <span style="color:green">0.0, 8</span> ) ) instead of KIND( <span style="color:red">0.0d0</span> )  
  
<blockquote>in both <tt>Headers/precision_mod.F</tt> and <tt>HEMCO/Core/hco_error_mod.F90</tt>.  That will prevent variables that you want to keep as 4-byte reals (e.g. for netCDF I/O) from being promoted to 8-byte reals.
+
<blockquote>in both <tt>Headers/precision_mod.F</tt> and <tt>HEMCO/Core/hco_error_mod.F90</tt>.  This will prevent variables that you want to keep as 4-byte reals (e.g. for netCDF I/O) from being promoted to 8-byte reals.
  
 
Also note, for gfortran we need to add the code in <span style="color:green">GREEN</span> to this variable setting in the <tt>Makefile_header.mk</tt>:</blockquote>
 
Also note, for gfortran we need to add the code in <span style="color:green">GREEN</span> to this variable setting in the <tt>Makefile_header.mk</tt>:</blockquote>

Latest revision as of 15:39, 19 June 2019

Overview

Flexible precision was introduced in Fortran-90

In most Fortran codes (including GEOS-Chem) you will see declarations such as:

! Integers
INTEGER   :: I, J   ! 4-byte integer  
INTEGER*4 :: K, L   ! 4-byte integer
INTEGER*8 :: M, N   ! 8-byte integer

! Floating point
REAL      :: A, B   ! 4-byte floating point
REAL*4    :: C, D   ! 4-byte floating point
REAL*8    :: E, F   ! 8-byte floating point 

etc. Note that:

  1. On most compilers, INTEGER refers to a 4-byte integer. You can make this default to an 8-byte integer by compiling with -i8. In most circumstances it is OK to use 4-byte integers, unless you need to point to a memory location or are reading an 8-byte integer from a netCDF file.
  2. On most compilers, REAL refers to a 4-byte floating-point. You can make this default to an 8-byte floating point by compiling with -r8.
  3. In some older Fortran codes, you will see the term DOUBLE PRECISION. This is the same as REAL*8 -- it is an 8-byte floating point.

Fortran 90 introduced a new precision concept. You can replace these fixed data types with declarations of arbitrary precision. This is done with the SELECTED_REAL_KIND and SELECTED_INT_KIND functions, which are described in more detail here.

--Bob Y. 15:01, 7 November 2014 (EST)

Why are we implementing flexible precision in GEOS-Chem?

Long story short: we need to do this in order to interface GEOS-Chem into the NASA GEOS-5 GCM (and other GCM's) more efficiently.

Most of the floating-point variables in GEOS-Chem are declared as REAL*8. We wrote the GEOS-Chem code in this manner, starting many years ago, when we were running on coarser-resolution grids (i.e. 4° x 5°). Back then, memory was generally not an issue.

But many GCMs—including the GEOS-5 GCM—declare floating-point variables as REAL*4. Because GCM's typically operate on very fine horizontal grids, conserving memory is of paramount concern.

When we connect GEOS-Chem to the GEOS-5 GCM, for example, we have to copy REAL*4 data from the GCM (such as the met fields, surface parameters, and other relevant quantities) into GEOS-Chem's REAL*8 arrays. This copying operation is very costly, as it requires extra memory and CPU cycles. But if we can transform GEOS-Chem's REAL*8 arrays into REAL*4 arrays, then we could just point GEOS-Chem's arrays to the GCM arrays without having to do all of the extra operations associated with the copying process.

Therefore, our goal is to recode GEOS-Chem so that you can select the floating-point precision that you want to use (either REAL*4 or REAL*8) at compile time. If you are going to connect GEOS-Chem to the GEOS-5 GCM, you can request all of the floating point variables to be declared as REAL*4, in order to match the variables in the GEOS-5 GCM. But if you are using the "traditional" serial GEOS-Chem, you can request that the floating point varaibles be declared as REAL*8, for backwards compatibility with prior code.

--Bob Y. 15:17, 7 November 2014 (EST)

Methodology

We followed this procedure in order to implement flexible precision into GEOS-Chem:

1. We defined module Headers/precision_mod.F. This module defines a parameter that will be used to specify the precision of variables in other parts of the code.

!------------------------------------------------------------------------------
!                  GEOS-Chem Global Chemical Transport Model                  !
!------------------------------------------------------------------------------
!BOP
!
! !MODULE: precision_mod.F
!
! !DESCRIPTION: Module PRECISION\_MOD is used to change the precision of
!  many variables throughout GEOS-Chem at compile-time.
!\\
!\\
! !INTERFACE:
!
      MODULE PRECISION_MOD
!
! !USES:
! 
      IMPLICIT NONE
      PRIVATE
!
! !DEFINED PARAMETERS:
! 
      !=================================================================
      ! Set parameters for floating precision
      !
      ! FP will be set to either 4-byte or 8-byte precision at compile 
      ! time.  Most variables can now  declared with REAL(fp).
      !=================================================================
#if defined( USE_REAL8 )

      ! Use 8-byte floating point precision when asked.
      INTEGER, PARAMETER, PUBLIC :: fp = KIND( REAL( 0.0, 8 ) )

#else

      ! Use 4-byte floating point by default.
      INTEGER, PARAMETER, PUBLIC :: fp = KIND( REAL( 0.0, 4 ) )

#endif

      !=================================================================
      ! Set parameters for fixed precision
      !
      ! Not all variables can be converted into the flexible precision.  
      ! Some may have to be still declared as either 4-byte or 8-byte 
      ! floating point.  Use these parameters for such variables.
      !=================================================================

      ! KIND parameter for 4-byte precision
      INTEGER, PARAMETER, PUBLIC :: f4 = KIND( REAL( 0.0, 4 ) )
      
      ! KIND parameter for 8-byte precision
      INTEGER, PARAMETER, PUBLIC :: f8 = KIND( REAL( 0.0, 8 ) )
!
! !REMARKS:
!  This module is designed to help avoid hard-coding precision.
!
! !REVISION HISTORY:
!  (1 ) Created. (myannetti, 11/04/14)
!  23 Nov 2016 - R. Yantosca - Now rewrite KIND definitions to prevent 4-byte
!                              and 8-byte variables from being elevated
!                              when using -r8 (or equivalent flags)
!EOP
!-----------------------------------------------------------------------------
!BOC
      END MODULE PRECISION_MOD
!EOC

Instead of having to figure out the proper settings with SELECTED_REAL_KIND, we can just use the KIND command to return that for us.

  • KIND( REAL( 0.0, 8 ) ) returns the proper "kind" value to define an 8-byte floating point
  • KIND( REAL( 0.0, 4 ) ) returns the proper "kind" value to define a 4-byte floating point

This value returned by the KIND function is saved in the constant named fp. (NOTE: This stands for "flexible precision". It does not have any connection to the GEOS-FP meteorology.)

Note that we have used an #if defined block to define the value of fp. If we compile with -DUSE_REAL8, then the fp can be used to declare 8-byte floating-point variables. Otherwise, fp can be used to declare 4-byte variables by default.

2. Add precision_mod.F to the dependencies listing in the Headers/Makefile. Add this line:

precision_mod.o: precision_mod.F

3. In the Makefille_header.mk, we added a new Makefile variable named PRECISION:

# %%%%% Default to 8-byte precision unless specified otherwise %%%%%
ifndef PRECISION
 PRECISION     :=8
endif

This variable is set to 8 by default (because for now, we want to compile the "traditional" serial GEOS-Chem with REAL*8 floating point precision, as it has always been compiled.

PRECISION is used again further down in the Makefile_header.mk to add a C-preprocessor switch:

# Add flexible precision declaration
ifeq ($(PRECISION),8)
USER_DEFS      += -DUSE_REAL8
endif

The -DUSE_REAL8 will define the USE_REAL8 C-preprocessor switch, which in turn will automatically pick 8-byte floating point precision.


4. At the top of each GEOS-Chem module or routine (typically in the !USES: comment section), you can place a reference to precision_mod.F90. For example, at the top of GeosCore/carbon_mod.F, you would add the line in RED:

!------------------------------------------------------------------------------
!                  GEOS-Chem Global Chemical Transport Model                  !
!------------------------------------------------------------------------------
!BOP
!     
! !MODULE: carbon_mod
!     
! !DESCRIPTION: Module CARBON\_MOD contains arrays and routines for performing
!  a carbonaceous aerosol simulation.  Original code taken from Mian Chin's 
!  GOCART model and modified accordingly. (rjp, bmy, 4/2/04, 6/30/10)
!\\   
!\\   
! !INTERFACE: 
!
      MODULE CARBON_MOD
!
! !USES:
!
!
      USE HCO_ERROR_MOD       ! For real precisions (hp)
      USE PRECISION_MOD       ! For GEOS-Chem precision (fp)

      IMPLICIT NONE
      PRIVATE

Note that HEMCO has its own precision parameters. We'll leave those alone, because HEMCO ships as a separate package as well.


5. Look for all variables in the module that are declared as REAL*8. Replace the REAL*8 text with REAL(fp) instead. So, in the above example, these lines:

      REAL*8, ALLOCATABLE :: ANTH_BLKC(:,:,:)
      REAL*8, ALLOCATABLE :: ANTH_ORGC(:,:,:)
      REAL*8, ALLOCATABLE :: BIOB_BLKC(:,:,:)

would become

      REAL(fp), ALLOCATABLE :: ANTH_BLKC(:,:,:)
      REAL(fp), ALLOCATABLE :: ANTH_ORGC(:,:,:)
      REAL(fp), ALLOCATABLE :: BIOB_BLKC(:,:,:)


6. IMPORTANT NOTE! Any literal constants in scientific notation made with the Fortran d exponents have to be changed to e. Also, the text _fp has to be appended to the exponent. This tells Fortran that we are using a customized precision definition. For example, the code:

      ! Molecules OH  per kg OH [molec/kg]
      REAL*8,  PARAMETER  :: XNUMOL_OH  = 6.022d23 / 17d-3
      REAL*8,  PARAMETER  :: CM3PERM3   = 1.d6
      REAL*8,  PARAMETER  :: TINY       = TINY(1.0)

would become instead:

      ! Molecules OH  per kg OH [molec/kg]
      REAL(fp), PARAMETER :: XNUMOL_OH  = 6.022e+23_fp / 17e-3_fp
      REAL(fp), PARAMETER :: CM3PERM3   = 1.e+6_fp
      REAL(fp), PARAMETER :: TINY       = TINY(1.0_fp)

7. We repeated the process in steps 4-6 for each GEOS-Chem source code file. We typically modify a file or two at a time, and then run a difference test. A difference test compares the code we are editing to a code with known behavior, such as the last accepted benchmarked version.

--Bob Yantosca (talk) 18:10, 23 November 2016 (UTC)

Compiling GEOS-Chem

To compile GEOS-Chem for 8-byte floating-point precision, just use the same commands as you always would:

make -j4 MET=geosfp GRID=4x5 TRACEBACK=y ...

(The ... denotes other compiler options, as described in Compiling GEOS-Chem.)

But to compile with 4-byte floating-point precision, you must now use the PRECISION keyword:

make -j4 MET=geosfp GRID=4x5 TRACEBACK=y PRECISION=4 ...

Eventually, PRECISION=4 will be automatically set if you build GEOS-Chem with the hpc option. This option compiles GEOS-Chem for use with the ESMF environment (such as is used in the GEOS-5 GCM). Then you can just use this command:

make -j4 MET=geosfp GRID=4x5 TRACEBACK=y ... hpc

--Bob Y. 15:41, 7 November 2014 (EST)

Validation

The following sections summarize our evaluation of GEOS-Chem with the PRECISION=4 option:

Benchmarks

Bob Yantosca (GCST) performed a 1-month benchmark simulation with the PRECISION=4 option. The results are posted on

Basic takeaways: The 1-month benchmark simulation with PRECISION=4:

  1. Used about 40% less memory than the v11-01f 1-month benchmark simulation (compiled with PRECISION=8).
  2. Finished 20 minutes faster than the v11-01f 1-month benchmark simulation.
  3. Shows negligible differences w/r/t the v11-01f 1-month benchmark.

--Bob Yantosca (talk) 17:48, 22 February 2017 (UTC)

Mass conservation

The plots below illustrate the evolution of total CO2 tracer mass in geosfp_2x25_masscons simulations (met year 2013) using GEOS-Chem "Classic". The RED LINE indicates the initial total mass, for reference.

Figure 1: Using PRECISION=4 and TURBDAY (aka full) PBL mixing

When the PRECISION=4 option is activated, we observe a consistent loss of mass starting after approximately one month. Differences start appearing in the 3rd decimal place.

TAU range    [hrs since 1/1/1985] :         245454.00000        254208.00000
Initial mass [kg                ] : 2.87551680000000e+15
Mass range   [kg                ] : 2.87481140000000e+15 2.87560220000000e+15

Mass cons prec4 1yr.png

Figure 2: Using PRECISION=8 and TURBDAY (aka full) PBL mixing

With PRECISION=8, differences start to appear in the 12th decimal place. This indicates numerical noise around a constant total tracer mass.

TAU range    [hrs since 1/1/1985] :         245454.00000        254208.00000
Initial mass [kg                ] : 2.87745967221413e+15
Mass range   [kg                ] : 2.87745967221277e+15 2.87745967221468e+15

Mass cons prec8 1yr.png

Figure 3: Using PRECISION=4 and VDIFF (aka non-local) PBL mixing.

Using PRECISION=4 and the non-local boundary layer mixing scheme (VDIFF) results in a similar picture to Figure 1. There is a consistent, significant mass loss with time. Again, differences start to appear in the 3rd decimal place.

TAU range    [hrs since 1/1/1985] :         245454.00000        254208.00000
Initial mass [kg                ] : 2.87551710000000e+15
Mass range   [kg                ] : 2.87443290000000e+15 2.87555090000000e+15

Mass cons prec4 1yr vdiff.png

Figure 4: Using PRECISION=8 and VDIFF (aka non-local) PBL mixing.

Using PRECISION=8 and the non-local boundary layer mixing scheme (VDIFF), we see mass conservation to the 8th decimal place. This indicates numerical noise around a constant tracer mass.

TAU range    [hrs since 1/1/1985] :         245454.00000        254208.00000
Initial mass [kg                ] : 2.87745967264367e+15
Mass range   [kg                ] : 2.87745967257390e+15 2.87745967694125e+15

Mass cons prec8 1yr vdiff.png

Summary

The plots above show that mass conservation is better achieved with the PRECISION=8 option than with PRECISION=4.

Seb Eastham wrote:

I would also advise that we tell users who want to do a simulation involving long-lived species (think stratosphere or CO2) that they might want to consider using PRECISION=8, which is the current GEOS-Chem default setting. One concern with the mass conservation error in PRECISION=4 (or actually at any precision) is that I think it will be a stationary random variable—there's no guarantee that the mean of the error over time will tend towards zero. This isn't a problem when the error at each step is ~1e-10, but could be a bigger problem when it's 1e-5 because we have no guarantee that it won't accumulate in one direction out of random bad luck.

Daniel Jacob replied:

The performance gains from using PRECISION=4 are most relevant for full-chemistry applications, not for CO2 applications.

Bob Yantosca replied:

We could...set PRECISION=4 as the default option when you generate any of the full-chemistry run directories (Standard, Tropchem, UCX, SOA, SOA-SVPOA, MarinePOA, Aciduptake). Then for other specialty simulations with long-lived species (CO2, CH4, maybe Hg, etc.), we could leave those as PRECISION=8.

Seb Eastham replied:

I would caution that there are still some species relevant to chemistry simulations which are long-lived and could be affected by mass balance – I’m thinking CFCs and N2O in particular. As long as users interested in these species are aware that there might be a random mass drift of ~0.01% per year, that should be fine.

--Bob Yantosca (talk) 20:43, 24 February 2017 (UTC)

Mass conservation implications for GCHP

Seb Eastham plotted the evolution of the total tracer mass versus time in GCHP for several configurations, as shown below.

  • The BLUE line represents a GCHP simulation using PRECISION=4.
  • The LIGHT ORANGE line represents a GCHP simulation using the default PRECISION=8.

Transport error v3 gchp.png

As you can see, the GCHP simulation using the PRECISION=4 option causes a significant loss of mass, similar to the GEOS-Chem classic simulations from the preceding section. On the other hand, the simulation using PRECISION=8 results in nearly perfect mass conservation.

Seb writes:

A fairly major structural change is needed to ensure mass conservation (specifically, the [GCHP] internal state must be changed from REAL*4 to REAL*8). This fix also invalidates "strong reproducibility"; there will be machine-precision-level differences in output when the code is run with different numbers of cores. This basically comes down to a choice; we can have two of the following:
  1. Mass conservation
  2. Strong reproducibility
  3. Speed
I thought 1 and 3 were most important but I figure that it's useful for us all to be aware.

--Bob Yantosca (talk) 15:17, 24 February 2017 (UTC)

Previous issues that are now resolved

Update KIND parameters to facilitate interface with the Beijing Climate Center model

This update was included in v11-01k and approved on 19 Dec 2016

Mike Long wrote:

With the flexible precision implementation, GEOS-Chem is unable to compile when the Intel Fortran Compiler flag -r8 is thrown in the compiler command. The -r8 flag forces variables declared as REAL or KIND(0.0) to be double precision by default. I learned this because the Beijing Climate Model uses the -r8 flag by default, and cannot compile without it. The result is a multitude of errors that seem intractable. The solution I found was simple, but it may not work altogether -- i.e. it's implications haven't been tested yet. By setting the "f4" definition in precision_mod and "sp" definition in hco_error_mod from KIND( 0.0 ) to KIND( REAL( 0.0, 4 ) ) all variables declared as either f4 or sp are allowed to be single precision.

Bob Yantosca replied:

I tested Mike's proposed fix on the ifort, pgfortran, and gfortran compilers and confirmed that it is the proper one. We should use:
    KIND( REAL( 0.0, 4 ) ) instead of KIND( 0.0   ) 
    KIND( REAL( 0.0, 8 ) ) instead of KIND( 0.0d0 ) 
in both Headers/precision_mod.F and HEMCO/Core/hco_error_mod.F90. This will prevent variables that you want to keep as 4-byte reals (e.g. for netCDF I/O) from being promoted to 8-byte reals. Also note, for gfortran we need to add the code in GREEN to this variable setting in the Makefile_header.mk:
    R8                 := -fdefault-real-8 -fdouble-real-8
This will prevent the default double precision size (which in GEOS-Chem should be 8 bytes) from inadvertently being elevated to 16 bytes.

--Bob Yantosca (talk) 17:33, 23 November 2016 (UTC)