Flexible precision in GEOS-Chem
Contents
Overview
Flexible precision was introduced in Fortran-90
In most Fortran codes (including GEOS-Chem) you will see declarations such as:
! Integers INTEGER :: I, J ! 4-byte integer INTEGER*4 :: K, L ! 4-byte integer INTEGER*8 :: M, N ! 8-byte integer ! Floating point REAL :: A, B ! 4-byte floating point REAL*4 :: C, D ! 4-byte floating point REAL*8 :: E, F ! 8-byte floating point
etc. Note that:
- On most compilers, INTEGER refers to a 4-byte integer. You can make this default to an 8-byte integer by compiling with -i8. In most circumstances it is OK to use 4-byte integers, unless you need to point to a memory location or are reading an 8-byte integer from a netCDF file.
- On most compilers, REAL refers to a 4-byte floating-point. You can make this default to an 8-byte floating point by compiling with -r8.
- In some older Fortran codes, you will see the term DOUBLE PRECISION. This is the same as REAL*8 -- it is an 8-byte floating point.
Fortran 90 introduced a new precision concept. You can replace these fixed data types with declarations of arbitrary precision. This is done with the SELECTED_REAL_KIND and SELECTED_INT_KIND functions, which are described in more detail here.
--Bob Y. 15:01, 7 November 2014 (EST)
Why are we implementing flexible precision in GEOS-Chem?
Long story short: we need to do this in order to interface GEOS-Chem into the NASA GEOS-5 GCM (and other GCM's) more efficiently.
Most of the floating-point variables in GEOS-Chem are declared as REAL*8. We wrote the GEOS-Chem code in this manner, starting many years ago, when we were running on coarser-resolution grids (i.e. 4° x 5°). Back then, memory was generally not an issue.
But many GCMs—including the GEOS-5 GCM—declare floating-point variables as REAL*4. Because GCM's typically operate on very fine horizontal grids, conserving memory is of paramount concern.
When we connect GEOS-Chem to the GEOS-5 GCM, for example, we have to copy REAL*4 data from the GCM (such as the met fields, surface parameters, and other relevant quantities) into GEOS-Chem's REAL*8 arrays. This copying operation is very costly, as it requires extra memory and CPU cycles. But if we can transform GEOS-Chem's REAL*8 arrays into REAL*4 arrays, then we could just point GEOS-Chem's arrays to the GCM arrays without having to do all of the extra operations associated with the copying process.
Therefore, our goal is to recode GEOS-Chem so that you can select the floating-point precision that you want to use (either REAL*4 or REAL*8) at compile time. If you are going to connect GEOS-Chem to the GEOS-5 GCM, you can request all of the floating point variables to be declared as REAL*4, in order to match the variables in the GEOS-5 GCM. But if you are using the "traditional" serial GEOS-Chem, you can request that the floating point varaibles be declared as REAL*8, for backwards compatibility with prior code.
--Bob Y. 15:17, 7 November 2014 (EST)
Methodology
We followed this procedure in order to implement flexible precision into GEOS-Chem:
1. We defined module Headers/precision_mod.F. This module defines a parameter that will be used to specify the precision of variables in other parts of the code.
!------------------------------------------------------------------------------
! GEOS-Chem Global Chemical Transport Model !
!------------------------------------------------------------------------------
!BOP
!
! !MODULE: precision_mod.F
!
! !DESCRIPTION: Module PRECISION\_MOD is used to change the precision of
! many variables throughout GEOS-Chem at compile-time.
!\\
!\\
! !INTERFACE:
!
MODULE PRECISION_MOD
!
! !USES:
!
IMPLICIT NONE
PRIVATE
!
! !DEFINED PARAMETERS:
!
!=================================================================
! Set parameters for floating precision
!
! FP will be set to either 4-byte or 8-byte precision at compile
! time. Most variables can now declared with REAL(fp).
!=================================================================
#if defined( USE_REAL8 )
! Use 8-byte floating point precision when asked.
INTEGER, PARAMETER, PUBLIC :: fp = KIND( REAL( 0.0, 8 ) )
#else
! Use 4-byte floating point by default.
INTEGER, PARAMETER, PUBLIC :: fp = KIND( REAL( 0.0, 4 ) )
#endif
!=================================================================
! Set parameters for fixed precision
!
! Not all variables can be converted into the flexible precision.
! Some may have to be still declared as either 4-byte or 8-byte
! floating point. Use these parameters for such variables.
!=================================================================
! KIND parameter for 4-byte precision
INTEGER, PARAMETER, PUBLIC :: f4 = KIND( REAL( 0.0, 4 ) )
! KIND parameter for 8-byte precision
INTEGER, PARAMETER, PUBLIC :: f8 = KIND( REAL( 0.0, 8 ) )
!
! !REMARKS:
! This module is designed to help avoid hard-coding precision.
!
! !REVISION HISTORY:
! (1 ) Created. (myannetti, 11/04/14)
! 23 Nov 2016 - R. Yantosca - Now rewrite KIND definitions to prevent 4-byte
! and 8-byte variables from being elevated
! when using -r8 (or equivalent flags)
!EOP
!-----------------------------------------------------------------------------
!BOC
END MODULE PRECISION_MOD
!EOC
Instead of having to figure out the proper settings with SELECTED_REAL_KIND, we can just use the KIND command to return that for us.
- KIND( REAL( 0.0, 8 ) ) returns the proper "kind" value to define an 8-byte floating point
- KIND( REAL( 0.0, 4 ) ) returns the proper "kind" value to define a 4-byte floating point
This value returned by the KIND function is saved in the constant named fp. (NOTE: This stands for "flexible precision". It does not have any connection to the GEOS-FP meteorology.)
Note that we have used an #if defined block to define the value of fp. If we compile with -DUSE_REAL8, then the fp can be used to declare 8-byte floating-point variables. Otherwise, fp can be used to declare 4-byte variables by default.
2. Add precision_mod.F to the dependencies listing in the Headers/Makefile. Add this line:
precision_mod.o: precision_mod.F
3. In the Makefille_header.mk, we added a new Makefile variable named PRECISION:
# %%%%% Default to 8-byte precision unless specified otherwise %%%%% ifndef PRECISION PRECISION :=8 endif
This variable is set to 8 by default (because for now, we want to compile the "traditional" serial GEOS-Chem with REAL*8 floating point precision, as it has always been compiled.
PRECISION is used again further down in the Makefile_header.mk to add a C-preprocessor switch:
# Add flexible precision declaration ifeq ($(PRECISION),8) USER_DEFS += -DUSE_REAL8 endif
The -DUSE_REAL8 will define the USE_REAL8 C-preprocessor switch, which in turn will automatically pick 8-byte floating point precision.
4. At the top of each GEOS-Chem module or routine (typically in the !USES: comment section), you can place a reference to precision_mod.F90. For example, at the top of GeosCore/carbon_mod.F, you would add the line in RED:
!------------------------------------------------------------------------------
! GEOS-Chem Global Chemical Transport Model !
!------------------------------------------------------------------------------
!BOP
!
! !MODULE: carbon_mod
!
! !DESCRIPTION: Module CARBON\_MOD contains arrays and routines for performing
! a carbonaceous aerosol simulation. Original code taken from Mian Chin's
! GOCART model and modified accordingly. (rjp, bmy, 4/2/04, 6/30/10)
!\\
!\\
! !INTERFACE:
!
MODULE CARBON_MOD
!
! !USES:
!
!
USE HCO_ERROR_MOD ! For real precisions (hp)
USE PRECISION_MOD ! For GEOS-Chem precision (fp)
IMPLICIT NONE
PRIVATE
Note that HEMCO has its own precision parameters. We'll leave those alone, because HEMCO ships as a separate package as well.
5. Look for all variables in the module that are declared as REAL*8. Replace the REAL*8 text with REAL(fp) instead. So, in the above example, these lines:
REAL*8, ALLOCATABLE :: ANTH_BLKC(:,:,:) REAL*8, ALLOCATABLE :: ANTH_ORGC(:,:,:) REAL*8, ALLOCATABLE :: BIOB_BLKC(:,:,:)
would become
REAL(fp), ALLOCATABLE :: ANTH_BLKC(:,:,:) REAL(fp), ALLOCATABLE :: ANTH_ORGC(:,:,:) REAL(fp), ALLOCATABLE :: BIOB_BLKC(:,:,:)
6. IMPORTANT NOTE! Any literal constants in scientific notation made with the Fortran d exponents have to be changed to e. Also, the text _fp has to be appended to the exponent. This tells Fortran that we are using a customized precision definition. For example, the code:
! Molecules OH per kg OH [molec/kg]
REAL*8, PARAMETER :: XNUMOL_OH = 6.022d23 / 17d-3
REAL*8, PARAMETER :: CM3PERM3 = 1.d6
REAL*8, PARAMETER :: TINY = TINY(1.0)
would become instead:
! Molecules OH per kg OH [molec/kg]
REAL(fp), PARAMETER :: XNUMOL_OH = 6.022e+23_fp / 17e-3_fp
REAL(fp), PARAMETER :: CM3PERM3 = 1.e+6_fp
REAL(fp), PARAMETER :: TINY = TINY(1.0_fp)
7. We repeated the process in steps 4-6 for each GEOS-Chem source code file. We typically modify a file or two at a time, and then run a difference test. A difference test compares the code we are editing to a code with known behavior, such as the last accepted benchmarked version.
--Bob Yantosca (talk) 18:10, 23 November 2016 (UTC)
Compiling GEOS-Chem
To compile GEOS-Chem for 8-byte floating-point precsion, just use the same commands as you always would:
make -j4 MET=geosfp GRID=4x5 TRACEBACK=y ...
(The ... denotes other compiler options, as described in Ch. 3 of the GEOS-Chem manual.)
But to compile with 4-byte floating-point precision, you must now use the PRECISION keyword:
make -j4 MET=geosfp GRID=4x5 TRACEBACK=y PRECISION=4 ...
Eventually, PRECISION=4 will be automatically set if you build GEOS-Chem with the hpc option. This option compiles GEOS-Chem for use with the ESMF environment (such as is used in the GEOS-5 GCM). Then you can just use this command:
make -j4 MET=geosfp GRID=4x5 TRACEBACK=y ... hpc
--Bob Y. 15:41, 7 November 2014 (EST)
Validation
The following sections summarize our evaluation of GEOS-Chem with the PRECISION=4 option:
Benchmarks
Bob Yantosca (GCST) performed a 1-month benchmark simulation with the PRECISION=4 option. The results are posted on
Basic takeaways: The 1-month benchmark simulation with PRECISION=4:
- Used about 40% less memory than the v11-01f 1-month benchmark simulation )(compiled with PRECISION=8).
- Finished 20 minutes faster than the v11-01f 1-month benchmark simulation.
- Shows negligible differences w/r/t the v11-01f 1-month benchmark.
--Bob Yantosca (talk) 17:48, 22 February 2017 (UTC)
Mass conservation
The plots below illustrate the evolution of total mass for 1 year (2013) in geosfp_2x25_masscons simulations, using GEOS-Chem classic. The RED line indicates the initial total mass, for reference.
Figure 1: Uses PRECISION=4 and TURBDAY (aka full) PBL mixing.
NOTE: The plot contains data for only 8 months, we will re-run the full year shortly.
As you can see, a significant loss of mass occurs approximately after the first month when the PRECISION=4 option is activated. Differences start appearing in the 3rd decimal place.
TAU range [hrs since 1/1/1985] : 245454.00000 251274.00000 Initial mass [kg ] : 2.87551680000000e+15 Mass range [kg ] : 2.87487610000000e+15 2.87560220000000e+15
Figure 2: Uses PRECISION=8 and TURBDAY (aka full) PBL mixing.
With PRECISION=8, differences start to appear in the 12th decimal place. This indicates numerical noise around a constant total tracer mass.
TAU range [hrs since 1/1/1985] : 245454.00000 254208.00000 Initial mass [kg ] : 2.87745967221413e+15 Mass range [kg ] : 2.87745967221277e+15 2.87745967221468e+15
Figure 3: Uses PRECISION=4 and VDIFF (aka non-local) PBL mixing.
TBD
Figure 4: Uses PRECISION=8 and VDIFF (aka non-local) PBL mixing.
TBD
Mass conservation in GCHP
Previous issues that are now resolved
Update KIND parameters to facilitate interface with the Beijing Climate Center model
This update was included in v11-01k and approved on 19 Dec 2016
Mike Long wrote:
With the flexible precision implementation, GEOS-Chem is unable to compile when the Intel Fortran Compiler flag -r8 is thrown in the compiler command. The -r8 flag forces variables declared as REAL or KIND(0.0) to be double precision by default. I learned this because the Beijing Climate Model uses the -r8 flag by default, and cannot compile without it. The result is a multitude of errors that seem intractable. The solution I found was simple, but it may not work altogether -- i.e. it's implications haven't been tested yet. By setting the "f4" definition in precision_mod and "sp" definition in hco_error_mod from KIND( 0.0 ) to KIND( REAL( 0.0, 4 ) ) all variables declared as either f4 or sp are allowed to be single precision.
Bob Yantosca replied:
I tested Mike's proposed fix on the ifort, pgfortran, and gfortran compilers and confirmed that it is the proper one. We should use:
KIND( REAL( 0.0, 4 ) ) instead of KIND( 0.0 ) KIND( REAL( 0.0, 8 ) ) instead of KIND( 0.0d0 )
in both Headers/precision_mod.F and HEMCO/Core/hco_error_mod.F90. This will prevent variables that you want to keep as 4-byte reals (e.g. for netCDF I/O) from being promoted to 8-byte reals. Also note, for gfortran we need to add the code in GREEN to this variable setting in the Makefile_header.mk:
R8 := -fdefault-real-8 -fdouble-real-8
This will prevent the default double precision size (which in GEOS-Chem should be 8 bytes) from inadvertently being elevated to 16 bytes.
--Bob Yantosca (talk) 17:33, 23 November 2016 (UTC)
