Flexible precision in GEOS-Chem
Contents
Overview
Flexible precision was introduced in Fortran-90
In most Fortran codes (including GEOS-Chem) you will see declarations such as:
! Integers INTEGER :: I, J ! 4-byte integer INTEGER*4 :: K, L ! 4-byte integer INTEGER*8 :: M, N ! 8-byte integer ! Floating point REAL :: A, B ! 4-byte floating point REAL*4 :: C, D ! 4-byte floating point REAL*8 :: E, F ! 8-byte floating point
etc. Note that:
- On most compilers, INTEGER refers to a 4-byte integer. You can make this default to an 8-byte integer by compiling with -i8. In most circumstances it is OK to use 4-byte integers, unless you need to point to a memory location or are reading an 8-byte integer from a netCDF file.
- On most compilers, REAL refers to a 4-byte floating-point. You can make this default to an 8-byte floating point by compiling with -r8.
- In some older Fortran codes, you will see the term DOUBLE PRECISION. This is the same as REAL*8 -- it is an 8-byte floating point.
Fortran 90 introduced a new precision concept. You can replace these fixed data types with declarations of arbitrary precision. This is done with the SELECTED_REAL_KIND and SELECTED_INT_KIND functions, which are described in more detail here.
--Bob Y. 15:01, 7 November 2014 (EST)
Why are we implementing flexible precision in GEOS-Chem?
Long story short: we need to do this in order to interface GEOS-Chem into the NASA GEOS-5 GCM (and other GCM's) more efficiently.
Most of the floating-point variables in GEOS-Chem are declared as REAL*8. We wrote the GEOS-Chem code in this manner, starting many years ago, when we were running on coarser-resolution grids (i.e. 4° x 5°). Back then, memory was generally not an issue.
But many GCMs—including the GEOS-5 GCM—declare floating-point variables as REAL*4. Because GCM's typically operate on very fine horizontal grids, conserving memory is of paramount concern.
When we connect GEOS-Chem to the GEOS-5 GCM, for example, we have to copy REAL*4 data from the GCM (such as the met fields, surface parameters, and other relevant quantities) into GEOS-Chem's REAL*8 arrays. This copying operation is very costly, as it requires extra memory and CPU cycles. But if we can transform GEOS-Chem's REAL*8 arrays into REAL*4 arrays, then we could just point GEOS-Chem's arrays to the GCM arrays without having to do all of the extra operations associated with the copying process.
Therefore, our goal is to recode GEOS-Chem so that you can select the floating-point precision that you want to use (either REAL*4 or REAL*8) at compile time. If you are going to connect GEOS-Chem to the GEOS-5 GCM, you can request all of the floating point variables to be declared as REAL*4, in order to match the variables in the GEOS-5 GCM. But if you are using the "traditional" serial GEOS-Chem, you can request that the floating point varaibles be declared as REAL*8, for backwards compatibility with prior code.
--Bob Y. 15:17, 7 November 2014 (EST)
Methodology
We followed this procedure in order to implement flexible precision into GEOS-Chem:
1. We defined module Headers/precision_mod.F. This module defines a parameter that will be used to specify the precision of variables in other parts of the code.
!------------------------------------------------------------------------------
! GEOS-Chem Global Chemical Transport Model !
!------------------------------------------------------------------------------
!BOP
!
! !MODULE: precision_mod.F
!
! !DESCRIPTION: Module PRECISION\_MOD is used to change the precision of
! many variables throughout GEOS-Chem at compile-time.
!\\
!\\
! !INTERFACE:
!
MODULE PRECISION_MOD
!
! !USES:
!
IMPLICIT NONE
PRIVATE
!
! !DEFINED PARAMETERS:
!
!=================================================================
! Set parameters for floating precision
!
! FP will be set to either 4-byte or 8-byte precision at compile
! time. Most variables can now declared with REAL(fp).
!=================================================================
#if defined( USE_REAL8 )
! Use 8-byte floating point precision when asked.
INTEGER, PARAMETER, PUBLIC :: fp = KIND( REAL( 0.0, 8 ) )
#else
! Use 4-byte floating point by default.
INTEGER, PARAMETER, PUBLIC :: fp = KIND( REAL( 0.0, 4 ) )
#endif
!=================================================================
! Set parameters for fixed precision
!
! Not all variables can be converted into the flexible precision.
! Some may have to be still declared as either 4-byte or 8-byte
! floating point. Use these parameters for such variables.
!=================================================================
! KIND parameter for 4-byte precision
INTEGER, PARAMETER, PUBLIC :: f4 = KIND( REAL( 0.0, 4 ) )
! KIND parameter for 8-byte precision
INTEGER, PARAMETER, PUBLIC :: f8 = KIND( REAL( 0.0, 8 ) )
!
! !REMARKS:
! This module is designed to help avoid hard-coding precision.
!
! !REVISION HISTORY:
! (1 ) Created. (myannetti, 11/04/14)
! 23 Nov 2016 - R. Yantosca - Now rewrite KIND definitions to prevent 4-byte
! and 8-byte variables from being elevated
! when using -r8 (or equivalent flags)
!EOP
!-----------------------------------------------------------------------------
!BOC
END MODULE PRECISION_MOD
!EOC
Instead of having to figure out the proper settings with SELECTED_REAL_KIND, we can just use the KIND command to return that for us.
- KIND( REAL( 0.0, 8 ) ) returns the proper "kind" value to define an 8-byte floating point
- KIND( REAL( 0.0, 4 ) ) returns the proper "kind" value to define a 4-byte floating point
This value returned by the KIND function is saved in the constant named fp. (NOTE: This stands for "flexible precision". It does not have any connection to the GEOS-FP meteorology.)
Note that we have used an #if defined block to define the value of fp. If we compile with -DUSE_REAL8, then the fp can be used to declare 8-byte floating-point variables. Otherwise, fp can be used to declare 4-byte variables by default.
2. Add precision_mod.F to the dependencies listing in the Headers/Makefile. Add this line:
precision_mod.o: precision_mod.F
3. In the Makefille_header.mk, we added a new Makefile variable named PRECISION:
# %%%%% Default to 8-byte precision unless specified otherwise %%%%% ifndef PRECISION PRECISION :=8 endif
This variable is set to 8 by default (because for now, we want to compile the "traditional" serial GEOS-Chem with REAL*8 floating point precision, as it has always been compiled.
PRECISION is used again further down in the Makefile_header.mk to add a C-preprocessor switch:
# Add flexible precision declaration ifeq ($(PRECISION),8) USER_DEFS += -DUSE_REAL8 endif
The -DUSE_REAL8 will define the USE_REAL8 C-preprocessor switch, which in turn will automatically pick 8-byte floating point precision.
4. At the top of each GEOS-Chem module or routine (typically in the !USES: comment section), you can place a reference to precision_mod.F90. For example, at the top of GeosCore/carbon_mod.F, you would add the line in RED:
!------------------------------------------------------------------------------
! GEOS-Chem Global Chemical Transport Model !
!------------------------------------------------------------------------------
!BOP
!
! !MODULE: carbon_mod
!
! !DESCRIPTION: Module CARBON\_MOD contains arrays and routines for performing
! a carbonaceous aerosol simulation. Original code taken from Mian Chin's
! GOCART model and modified accordingly. (rjp, bmy, 4/2/04, 6/30/10)
!\\
!\\
! !INTERFACE:
!
MODULE CARBON_MOD
!
! !USES:
!
!
USE HCO_ERROR_MOD ! For real precisions (hp)
USE PRECISION_MOD ! For GEOS-Chem precision (fp)
IMPLICIT NONE
PRIVATE
Note that HEMCO has its own precision parameters. We'll leave those alone, because HEMCO ships as a separate package as well.
5. Look for all variables in the module that are declared as REAL*8. Replace the REAL*8 text with REAL(fp) instead. So, in the above example, these lines:
REAL*8, ALLOCATABLE :: ANTH_BLKC(:,:,:) REAL*8, ALLOCATABLE :: ANTH_ORGC(:,:,:) REAL*8, ALLOCATABLE :: BIOB_BLKC(:,:,:)
would become
REAL(fp), ALLOCATABLE :: ANTH_BLKC(:,:,:) REAL(fp), ALLOCATABLE :: ANTH_ORGC(:,:,:) REAL(fp), ALLOCATABLE :: BIOB_BLKC(:,:,:)
6. IMPORTANT NOTE! Any literal constants in scientific notation made with the Fortran d exponents have to be changed to e. Also, the text _fp has to be appended to the exponent. This tells Fortran that we are using a customized precision definition. For example, the code:
! Molecules OH per kg OH [molec/kg]
REAL*8, PARAMETER :: XNUMOL_OH = 6.022d23 / 17d-3
REAL*8, PARAMETER :: CM3PERM3 = 1.d6
REAL*8, PARAMETER :: TINY = TINY(1.0)
would become instead:
! Molecules OH per kg OH [molec/kg]
REAL(fp), PARAMETER :: XNUMOL_OH = 6.022e+23_fp / 17e-3_fp
REAL(fp), PARAMETER :: CM3PERM3 = 1.e+6_fp
REAL(fp), PARAMETER :: TINY = TINY(1.0_fp)
7. We repeated the process in steps 4-6 for each GEOS-Chem source code file. We typically modify a file or two at a time, and then run a difference test. A difference test compares the code we are editing to a code with known behavior, such as the last accepted benchmarked version.
--Bob Yantosca (talk) 18:10, 23 November 2016 (UTC)
Compiling GEOS-Chem
To compile GEOS-Chem for 8-byte floating-point precsion, just use the same commands as you always would:
make -j4 MET=geosfp GRID=4x5 TRACEBACK=y ...
(The ... denotes other compiler options, as described in Ch. 3 of the GEOS-Chem manual.)
But to compile with 4-byte floating-point precision, you must now use the PRECISION keyword:
make -j4 MET=geosfp GRID=4x5 TRACEBACK=y PRECISION=4 ...
Eventually, PRECISION=4 will be automatically set if you build GEOS-Chem with the hpc option. This option compiles GEOS-Chem for use with the ESMF environment (such as is used in the GEOS-5 GCM). Then you can just use this command:
make -j4 MET=geosfp GRID=4x5 TRACEBACK=y ... hpc
--Bob Y. 15:41, 7 November 2014 (EST)
Validation
The following sections summarize our evaluation of GEOS-Chem with the PRECISION=4 option:
Benchmarks
Bob Yantosca (GCST) performed a 1-month benchmark simulation with the PRECISION=4 option. The results are posted on
Basic takeaways: The 1-month benchmark simulation with PRECISION=4:
- Used about 40% less memory than the v11-01f 1-month benchmark simulation )(compiled with PRECISION=8).
- Finished 20 minutes faster than the v11-01f 1-month benchmark simulation.
- Shows negligible differences w/r/t the v11-01f 1-month benchmark.
--Bob Yantosca (talk) 17:48, 22 February 2017 (UTC)
Mass conservation
The plots below illustrate the evolution of total CO2 tracer mass for 1 year (2013) in geosfp_2x25_masscons simulations, using GEOS-Chem classic. The RED line indicates the initial total mass, for reference.
Figure 1: Uses PRECISION=4 and TURBDAY (aka full) PBL mixing
When the PRECISION=4 option is activated, we observe a consistent loss of mass starting after approximately one month. Differences start appearing in the 3rd decimal place.
TAU range [hrs since 1/1/1985] : 245454.00000 254208.00000 Initial mass [kg ] : 2.87551680000000e+15 Mass range [kg ] : 2.87481140000000e+15 2.87560220000000e+15
Figure 2: Uses PRECISION=8 and TURBDAY (aka full) PBL mixing
With PRECISION=8, differences start to appear in the 12th decimal place. This indicates numerical noise around a constant total tracer mass.
TAU range [hrs since 1/1/1985] : 245454.00000 254208.00000 Initial mass [kg ] : 2.87745967221413e+15 Mass range [kg ] : 2.87745967221277e+15 2.87745967221468e+15
Figure 3: Uses PRECISION=4 and VDIFF (aka non-local) PBL mixing.
Using PRECISION=4 and the non-local boundary layer mixing scheme (VDIFF) results in a similar picture to Figure 1. There is a consistent, significant mass loss with time. Again, differences start to appear in the 3rd decimal place.
TAU range [hrs since 1/1/1985] : 245454.00000 254208.00000 Initial mass [kg ] : 2.87551710000000e+15 Mass range [kg ] : 2.87443290000000e+15 2.87555090000000e+15
Figure 4: Uses PRECISION=8 and VDIFF (aka non-local) PBL mixing.
Using PRECISION=8 and the non-local boundary layer mixing scheme (VDIFF), we see mass conservation to the 8th decimal place. This indicates numerical noise around a constant tracer mass.
TAU range [hrs since 1/1/1985] : 245454.00000 254208.00000 Initial mass [kg ] : 2.87745967264367e+15 Mass range [kg ] : 2.87745967257390e+15 2.87745967694125e+15
Summary
The plots above show that mass conservation is better achieved with the PRECISION=8 option than with PRECISION=4.
Seb Eastham wrote:
I would also advise that we tell users who want to do a simulation involving long-lived species (think stratosphere or CO2) that they might want to consider using PRECISION=8, which is the current GEOS-Chem default setting. One concern with the mass conservation error in PRECISION=4 (or actually at any precision) is that I think it will be a stationary random variable—there's no guarantee that the mean of the error over time will tend towards zero. This isn't a problem when the error at each step is ~1e-10, but could be a bigger problem when it's 1e-5 because we have no guarantee that it won't accumulate in one direction out of random bad luck.
--Bob Yantosca (talk) 20:59, 23 February 2017 (UTC)
Mass conservation implications for GCHP
Seb Eastham plotted the evolution of the total tracer mass versus time in GCHP for several configurations, as shown below.
- The BLUE line represents a GCHP simulation using PRECISION=4.
- The LIGHT ORANGE line represents a GCHP simulation using the default PRECISION=8.
As you can see, the GCHP simulation using the PRECISION=4 option causes a significant loss of mass, similar to the GEOS-Chem classic simulations from the preceding section. On the other hand, the PRECISION=8 option results in almost perfect mass conservation.
Seb writes:
A fairly major structural change is needed to ensure mass conservation (specifically, the [GCHP] internal state must be changed fromREAL*4toREAL*8). This fix also invalidates "strong reproducibility"; there will be machine-precision-level differences in output when the code is run with different numbers of cores. This basically comes down to a choice; we can have two of the following:I thought 1 and 3 were most important but I figure that it's useful for us all to be aware.
- Mass conservation
- Strong reproducibility
- Speed
--Bob Yantosca (talk) 20:57, 23 February 2017 (UTC)
Previous issues that are now resolved
Update KIND parameters to facilitate interface with the Beijing Climate Center model
This update was included in v11-01k and approved on 19 Dec 2016
Mike Long wrote:
With the flexible precision implementation, GEOS-Chem is unable to compile when the Intel Fortran Compiler flag -r8 is thrown in the compiler command. The -r8 flag forces variables declared as REAL or KIND(0.0) to be double precision by default. I learned this because the Beijing Climate Model uses the -r8 flag by default, and cannot compile without it. The result is a multitude of errors that seem intractable. The solution I found was simple, but it may not work altogether -- i.e. it's implications haven't been tested yet. By setting the "f4" definition in precision_mod and "sp" definition in hco_error_mod from KIND( 0.0 ) to KIND( REAL( 0.0, 4 ) ) all variables declared as either f4 or sp are allowed to be single precision.
Bob Yantosca replied:
I tested Mike's proposed fix on the ifort, pgfortran, and gfortran compilers and confirmed that it is the proper one. We should use:
KIND( REAL( 0.0, 4 ) ) instead of KIND( 0.0 ) KIND( REAL( 0.0, 8 ) ) instead of KIND( 0.0d0 )
in both Headers/precision_mod.F and HEMCO/Core/hco_error_mod.F90. This will prevent variables that you want to keep as 4-byte reals (e.g. for netCDF I/O) from being promoted to 8-byte reals. Also note, for gfortran we need to add the code in GREEN to this variable setting in the Makefile_header.mk:
R8 := -fdefault-real-8 -fdouble-real-8
This will prevent the default double precision size (which in GEOS-Chem should be 8 bytes) from inadvertently being elevated to 16 bytes.
--Bob Yantosca (talk) 17:33, 23 November 2016 (UTC)
