Coupling GEOS-Chem with RRTMG
On this page we provide information about the coupling of GEOS-Chem with the RRTMG radiative transfer model (by AER, Inc.).
- 1 Overview
- 2 netCDF inputs via HEMCO
- 3 Running GCRT
- 4 Example Output
- 5 Previous issues that are now resolved
- 6 Known issues
- 7 References
The GEOS-Chem model with online radiative transfer calculations (referred to as GCRT) was developed to allow GEOS-Chem users to produce gas and aerosol direct radiative effect (DRE) output for both the longwave and shortwave. This alternative to offline coupling allows better temporal resolution in the RT calculations and provides a consist platform for GEOS-Chem users with the widely used radiative transfer package RRTMG.
Most of the added code is 'transparent', therefore this version of the GEOS-Chem model can still be run with the radiation code switched off. The optical properties are calculated at multiple wavelengths so that the user is no longer restricted to 550nm as default, so there are associated changes regardless of whether the radiative code is invoked. However, these cause negligible slow down (the default model is actually slightly faster than the standard v9-01-03). Compiling with RRTMG=yes requires approximately double the amount of RAM (~15Gb for a 2x2.5 simulation) and takes between 40% and 100% longer depending on the settings used.
This wiki explains the key changes and requirements. If you want to dig deeper into the changes and assumptions made, or are looking for a quick-start guide to running the model, download the GCRT User Guide
GCRT User Groups
|Atmospheric Chemistry, Massachusetts Institute of Technology||Colette Heald
|Add yours here|
Obtaining the source code
GCRT was originally added into GEOS-Chem v10-01i, which was tested with 1-month benchmark simulations approved on 30 April 2015 and a 1-year benchmark simulation approved on 01 May 2015. It is included in the GEOS-Chem v10-01 public release.
Authors and collaborators
- David Ridley (Civil and Environmental Engineering, MIT) -- Lead Developer
- Colette Heald (Civil and Environmental Engineering, MIT) -- Principal Investigator
- Steven Barrett (Aeronautics and Astronautics, MIT)
- Karen Cady-Pereira (AER)
- Matthew Alvarado (AER)
Questions regarding GCRT can be directed at David (e-mail linked above).
--David R. 18:00, 08 October 2013 (EST)
- A new menu for Radiation exists in input.geos (see below). This also includes a wavelength selection for optical depth output that is independent of whether RRTMG is switched on (up to three optical depths can be output in the ND21 (OD-MAP-$) diagnostic, but only the first is used for the timeseries diagnostics ND48, ND49, ND50, ND51, and ND51b).
- A new code folder, GeosRad/ is required for the RRTMG code. The module rrtmg_rad_transfer_mod.F is the driver code (found in GeosCore/) that interfaces with RRTMG.
- The optics look-up tables are updated, containing multiple wavelengths and separated into files for each species (soot.dat, so4.dat, org.dat, dust.dat, ssa.dat, ssc.dat) that are stored in the run directory. To prevent discrepancies jv_spec.dat no longer contains optical properties for aerosol, Fast-J uses aerosol optics from from the same speciated look-up tables. The new files mean that jv_spec_aod.dat is obsolete and has been removed.
- Several new input files are required. Surface albedo and emissivity climatologies have been generated and must be stored in modis_surf_201210/ within the root data directory (e.g. ExtData/CHEM_INPUTS/). Climatologies of gases (CH4, N2O, CFC-11, CFC-12, CFC-22 and CCl4) must be stored within the root data directory.
- A new set of diagnostics (ND72) are available providing the change in radiative flux (DRE) for gases and aerosol, LW/SW, top of atmosphere (TOA) and surface, and clear-sky and all-sky conditions. These also include AOD, SSA and asymmetry parameter for each aerosol species at the requested wavelengths.
netCDF inputs via HEMCO
This update was included in GEOS-Chem 12.1.0, which was released on 26 Nov 2018.
Several of the inputs to RRTMG have been stored in the binary punch (aka bpch) data format. These include:
- Diffuse surface albedoes in visible and near-IR
- Direct surface albedoes in visible and near-IR
- Surface emissivity in 16 different wavelength bands
- Concentrations of CCl4, CFC11, CFC12, CFC22, CH4, N2O from TES in ppb
We are planning on removing all bpch data from GEOS-Chem. As such we have ported these bpch inuts to netCDF so that they can be read in via HEMCO. The HEMCO_Config.rc file that ships with the GEOS-Chem Unit Tester will now have these additional lines.
#============================================================================== # --- Inputs for the RRTMG radiative transfer model --- # # These fields will only be read if the +RRTMG+ toggle is activated. # +RRTMG+ can be set explicitly above as a setting of the base extension # (--> +RRTMG+: true/false). If not defined, it will be set automatically # based on the RRTMG toggle in input.geos (recommended). # # NOTE: The 2 x 2.5 albedo fields and emissivity fields will produce # differences at the level of numerical noise when comparing output to # simulations from prior versions (esp. when running at 4 x 5 resolution). # You might see larger differences w/r/t prior verisons for a few grid boxes # along the coastline of Antarctica, where the difference in resolution # and regridding will be more apparent in the sharp transition from ice to # ocean. If this is a problem, you can use the data files at 4x5 resolution # for 4x5 RRTMG simulations. # # ALSO NOTE: The algorithm that HEMCO uses to select each time slice is # likely different than what was implemented when reading the old bpch # data from disk. This can also cause differences when comparing to # prior versions. #============================================================================== (((+RRTMG+ * MODIS_ALBDFNIR $ROOT/RRTMG/v2018-11/modis_surf_albedo.2x25.nc ALBDFNIR 2002/1-12/1-31/0 C xy 1 * - 1 1 * MODIS_ALBDFVIS $ROOT/RRTMG/v2018-11/modis_surf_albedo.2x25.nc ALBDFVIS 2002/1-12/1-31/0 C xy 1 * - 1 1 * MODIS_ALBDRNIR $ROOT/RRTMG/v2018-11/modis_surf_albedo.2x25.nc ALBDRNIR 2002/1-12/1-31/0 C xy 1 * - 1 1 * MODIS_ALBDRVIS $ROOT/RRTMG/v2018-11/modis_surf_albedo.2x25.nc ALBDRVIS 2002/1-12/1-31/0 C xy 1 * - 1 1 * MODIS_EMISSIVITY_01 $ROOT/RRTMG/v2018-11/modis_emissivity.2x25.nc RTEMISS_band01 2002/1-12/1-31/0 C xy 1 * - 1 1 * MODIS_EMISSIVITY_02 $ROOT/RRTMG/v2018-11/modis_emissivity.2x25.nc RTEMISS_band02 2002/1-12/1-31/0 C xy 1 * - 1 1 * MODIS_EMISSIVITY_03 $ROOT/RRTMG/v2018-11/modis_emissivity.2x25.nc RTEMISS_band03 2002/1-12/1-31/0 C xy 1 * - 1 1 * MODIS_EMISSIVITY_04 $ROOT/RRTMG/v2018-11/modis_emissivity.2x25.nc RTEMISS_band04 2002/1-12/1-31/0 C xy 1 * - 1 1 * MODIS_EMISSIVITY_05 $ROOT/RRTMG/v2018-11/modis_emissivity.2x25.nc RTEMISS_band05 2002/1-12/1-31/0 C xy 1 * - 1 1 * MODIS_EMISSIVITY_06 $ROOT/RRTMG/v2018-11/modis_emissivity.2x25.nc RTEMISS_band06 2002/1-12/1-31/0 C xy 1 * - 1 1 * MODIS_EMISSIVITY_07 $ROOT/RRTMG/v2018-11/modis_emissivity.2x25.nc RTEMISS_band07 2002/1-12/1-31/0 C xy 1 * - 1 1 * MODIS_EMISSIVITY_08 $ROOT/RRTMG/v2018-11/modis_emissivity.2x25.nc RTEMISS_band08 2002/1-12/1-31/0 C xy 1 * - 1 1 * MODIS_EMISSIVITY_09 $ROOT/RRTMG/v2018-11/modis_emissivity.2x25.nc RTEMISS_band09 2002/1-12/1-31/0 C xy 1 * - 1 1 * MODIS_EMISSIVITY_10 $ROOT/RRTMG/v2018-11/modis_emissivity.2x25.nc RTEMISS_band10 2002/1-12/1-31/0 C xy 1 * - 1 1 * MODIS_EMISSIVITY_11 $ROOT/RRTMG/v2018-11/modis_emissivity.2x25.nc RTEMISS_band11 2002/1-12/1-31/0 C xy 1 * - 1 1 * MODIS_EMISSIVITY_12 $ROOT/RRTMG/v2018-11/modis_emissivity.2x25.nc RTEMISS_band12 2002/1-12/1-31/0 C xy 1 * - 1 1 * MODIS_EMISSIVITY_13 $ROOT/RRTMG/v2018-11/modis_emissivity.2x25.nc RTEMISS_band13 2002/1-12/1-31/0 C xy 1 * - 1 1 * MODIS_EMISSIVITY_14 $ROOT/RRTMG/v2018-11/modis_emissivity.2x25.nc RTEMISS_band14 2002/1-12/1-31/0 C xy 1 * - 1 1 * MODIS_EMISSIVITY_15 $ROOT/RRTMG/v2018-11/modis_emissivity.2x25.nc RTEMISS_band15 2002/1-12/1-31/0 C xy 1 * - 1 1 * MODIS_EMISSIVITY_16 $ROOT/RRTMG/v2018-11/modis_emissivity.2x25.nc RTEMISS_band16 2002/1-12/1-31/0 C xy 1 * - 1 1 * TES_CLIM_CCL4 $ROOT/RRTMG/v2018-11/species_clim_profiles.2x25.nc CCl4 2000/1/1/0 C xyz ppbv * - 1 1 * TES_CLIM_CFC11 $ROOT/RRTMG/v2018-11/species_clim_profiles.2x25.nc CFC11 2000/1/1/0 C xyz ppbv * - 1 1 * TES_CLIM_CFC12 $ROOT/RRTMG/v2018-11/species_clim_profiles.2x25.nc CFC12 2000/1/1/0 C xyz ppbv * - 1 1 * TES_CLIM_CFC22 $ROOT/RRTMG/v2018-11/species_clim_profiles.2x25.nc CFC22 2000/1/1/0 C xyz ppbv * - 1 1 * TES_CLIM_CH4 $ROOT/RRTMG/v2018-11/species_clim_profiles.2x25.nc CH4 2000/1/1/0 C xyz ppbv * - 1 1 * TES_CLIM_N2O $ROOT/RRTMG/v2018-11/species_clim_profiles.2x25.nc N2O 2000/1/1/0 C xyz ppbv * - 1 1 )))+RRTMG+
Therefore you will need to obtain the netCDF files in the HEMCO/RRTMG/v2018-11 data directory. For more information, please see our HEMCO data directories wiki page.
As with the latest version of GEOS-Chem, NetCDF libraries must be installed (see Installing libraries for GEOS-Chem). The code should be compiled using the compile switch RRTMG, i.e. make RRTMG=yes. A fresh compilation after a make realclean will take 15-20 min (subsequent compiles are much faster).
Once the code is compiled the input.geos file will need configuring:
%%% RADIATION MENU %%% : AOD Wavelength (nm) : 550 Turn on RRTMG? : T Calculate LW fluxes? : T Calculate SW fluxes? : T Clear-sky flux? : T All-sky flux? : F Radiation Timestep [sec]: 10800 Species fluxes : 0 0 0 0 0 0 0 0 0 1 0 ---[O3,ME,SU,NI,AM,BC,OA,SS,DU,PM,ST]
The species for which DRE output is currently available are listed with two-letter identifiers. In the example the DRE is calculated for sulfate (SU) and all particulate matter (PM) - see the table below for all identifier descriptions. The recommended radiation time step is 180 min but can be set for any multiple of the chemistry time step. The baseline (BA) flux is always output by default and is the flux (W/m^2) with all aerosol and species included.
In this set up all timeseries diagnostics (ND48, ND49, ND50, ND51, ND51b) produce optical depth output at 550nm, but the main ND21 diagnostic output contains three copies of the optical depths (and SSA and asymmetry parameters if requested). The output variables have the wavelength appended to the variable name e.g. OPBC550.
The species available for output from the flux calculations are as follows:
|10||PM||All particulate matter|
|11||ST||Stratospheric aerosol (UCX simulation only)|
- The radiative impacts of gases (ozone and methane) are relatively untested at this stage and should be interpreted with caution.
- If you are using the UCX chemistry mechanism, then you will need to add an extra entry (0 or 1) at the end of the species fluxes line for stratospheric aerosols for a total of 11 possible species.
Finally, the ND72 diagnostic must be set to 1 if RRTMG is switched on.
Once all the auxiliary files are in place (new optics data files, surface albedo climatologies, trace gas profile climatologies) and the input.geos set up, the model can be run in the usual way.
For more information please consult the GCRT User Guide.
ND72 diagnostic in binary punch format
NOTE: The binary punch (aka bpch) file format will be discontinued in a future GEOS-Chem version. For the time being, GEOS-Chem 12 will allow you to save diagnostics in either bpch or netCDF file formats.
Below, example output fluxes and direct radiative effect (DRE) from GCRT are displayed for March 2013. The top row show the total flux with all aerosol and gases included, subsequent rows show the DRE for all particulate matter, black carbon, and sulfate. The left column shows top of atmosphere (TOA) and the right column shows surface DRE. All results are for all-sky conditions (i.e. including clouds). The identifier for each figure is |F|SW|PM|T|A| (F - denotes a flux output, SW/LW - shortwave or longwave, PM - species identifier, T/S - TOA or surface, A/C - All-sky or clear sky)
netCDF diagnostic output
Previous issues that are now resolved
Replace nonstandard DLOG function with LOG
This fix was included in v11-01j and approved on 03 Dec 2016
GeosRad/rrtmg_lw_setcoef.F, the GNU Fortran compiler detected an error using the specific REAL*8
DLOG function. We have now converted this to the standard
We commented out the code in RED and added the code in GREEN:
!------------------------------------------------------------------------------ ! Prior to 10/14/16: ! Use the F90 standard LOG function instead of ALOG or DLOG. (bmy, 10/14/16) !! plog = alog(pavel(lay)) ! plog = dlog(pavel(lay)) !------------------------------------------------------------------------------ plog = LOG(pavel(lay))
Bug fixes in rrtmg_rad_transfer_mod.F
David Ridley wrote:
- Seb has kindly found and fixed a couple of bugs in the RRTMG code that have crept in recently. These do mess up the results and so will need implementing in the model as soon as possible. I’ll let anyone that I know currently using RRTMG to re-download the code once it has been fixed.
In GeosCore/rrtmg_rad_transfer_mod.F, around line 1011 change the following lines in red to the lines in green:
IF (ASMAERDIAG(IJLOOP,L,IB_SW).GT.0.999d0) THEN ASMAER(IJLOOP,L,IB_SW) = 0.999d0 ASMAERDIAG(IJLOOP,L,IB_SW) = 0.999d0 ENDIF ! IF (ASMAERDIAG(IJLOOP,L,IB_SW).LT.0.001d0) THEN ! ASMAER(IJLOOP,L,IB_SW) = 0.001 ! ASMAERDIAG(IJLOOP,L,IB_SW) = 0.001 ! ENDIF IF ((SSAAERDIAG(IJLOOP,L,IB_SW).LT.0.001d0).OR. & (SSAAERDIAG(IJLOOP,L,IB_SW).GT.1.0d0)) THEN SSAAER(IJLOOP,L,IB_SW) = 0.99 SSAAERDIAG(IJLOOP,L,IB_SW) = 0.99 ENDIF IF (TAUAERDIAG(IJLOOP,L,IB_SW).GT.1.0) THEN TAUAER_SW(IJLOOP,L,IB_SW) = 1.0 TAUAERDIAG(IJLOOP,L,IB_SW) = 1.0 ENDIF
In the same routine around line 1370, change the following line from:
p_SSAAER = TAUAER_LW ( IL,:,:)
p_SSAAER = SSAAER ( IL,:,:)
Fix bug that prevents storage of O3 and CH4 RFs
This fix was validated in the 1-month benchmark simulation v11-01a, which was approved on 07 Jul 2015.
Sebastian Eastham wrote:
- (GEOS-Chem v10-01 contains) an issue in the RRTMG interface where there is a single-character typo in SET_SPECMASK. This typo results in dust, ozone and methane RFs being incorrectly stored - in fact it prevents O3 and CH4 RFs from being stored at all, instead storing dust RFs in their place (which in turn miss the last two dust bins and instead use either sea salt or stratospheric aerosol). This is because line 1609 in rrtmg_rad_transfer_mod reads
- When it should read
- (note second character). This bug affects both the tropospheric and UCX version of the code. Since NXTRA is the offset used to account for whether or not stratospheric aerosols are inserted mid-way through the indexing, and NASPECRAD is always 2 smaller than NSPECRAD, everything index above 7 is set to a value 2 smaller than it should be in the mask.
- This bug occurs with or without UCX. The offset is always incorrectly calculated, so dust, ozone and methane are never output correctly. The easiest test is to check the ozone or methane output (they will both look like large-diameter dust RF). Dust will look *almost* right, but will actually include stratospheric aerosols (if UCX=yes) or sea-salt (if UCX is not on).
Bug in RRTMG array sizes
Sebastian Eastham wrote:
- I came across a bug in the RRTMG code, which looks like it originated with AER. In GeosRad/mcica_subcol_gen_lw.F90 and GeosRad/rrtmg_lw_rad.F90, a few arrays are incorrectly declared to be size (nbndlw,ncol,nlay) when they should instead by (ngptlw,ncol,nlay), or vice versa. This results in NaNs sometimes creeping into the arrays when OpenMP is turned on, or just outright failure.
- The comments show that the arrays should have been declared differently in the first place, so I’ll be getting in touch with AER shortly to make sure they’re aware of the issue.
In GeosRad/mcica_subcol_gen_lw.F90, change the following code (in bold) to:
real(kind=rb), intent(in) :: tauc(nbndlw,ncol,nlay) ! in-cloud optical depth ! Dimensions: (nbndlw,ncol,nlay)
In GeosRad/rrtmg_lw_rad.F90, change the following code (in bold) to:
real(kind=rb), intent(in) :: cldfmcl(ngptlw,ncol,nlay) ! Cloud fraction ! Dimensions: (ngptlw,ncol,nlay) real(kind=rb), intent(in) :: ciwpmcl(ngptlw,ncol,nlay) ! In-cloud ice water path (g/m2) ! Dimensions: (ngptlw,ncol,nlay) real(kind=rb), intent(in) :: clwpmcl(ngptlw,ncol,nlay) ! In-cloud liquid water path (g/m2) ! Dimensions: (ngptlw,ncol,nlay) ... real(kind=rb), intent(in) :: taucmcl(ngptlw,ncol,nlay) ! In-cloud optical depth ! Dimensions: (ngptlw,ncol,nlay)
Bug fixes for RRTMG in v11-02
This fix was included in GEOS-Chem 12.0.0.
Christopher Holmes wrote:
The RRTMG calculations have a bug in v11-01. The problem was probably introduced with the unit changes in v11-01. O3 and CH4 concentrations are specified incorrectly, so their radiative effects are badly wrong. Aerosol radiative effects are also affected due to spectral overlap with gases (whose concentrations are wrong), but the aerosol error is modest. Xinyi Dong (U Tennessee) tested my fix and found that the error in BC radiative effect is <1% in the global mean for a 1 month average, but locally up to 5%. Other aerosols likely have similar magnitude errors.
RRTMG does not compile with GNU Fortran
The GEOS-Chem Support Team has recently discovered that the RRTMG code in GEOS-Chem does not compile with the gfortran compiler. The compile error is included below for reference. While users still have the option to use the IFORT compilers, this is somewhat problematic for our movement towards using open-source software. For example, if users run GEOS-Chem on the Amazon Web Services cloud computing environment, we recommend that they use gfortran. We’ve also been using gfortran 7.1.0 for our unit tests because that compiler tends to be more strict.
We wanted to alert you to this unresolved issue in case anyone is interested in pursuing a fix with the folks at AER and/or updating RRTMG in GEOS-Chem. The GCST typically tries to remain hands-off with third party code (e.g. RRTMG, ISORROPIA) in GEOS-Chem. It appears that RRTMG v3.0 was implemented in GEOS-Chem. The latest RRTMG version is 3.3, so it may be that the issue has been resolved on AER’s end but that’s not clear.
gfortran -cpp -w -std=legacy -fautomatic -fno-align-commons -fconvert=big-endian -fno-range-check -O1 -override-limits -fopenmp -mcmodel=medium -fbacktrace -g -DLINUX_GFORTRAN -DBPCH_DIAG -DBPCH_TIMESER -DBPCH_TPBC -DRRTMG -DGEOS_FP -DGRIDREDUCED -DGRID4x5 -DNC_HAS_COMPRESSION -DUSE_REAL8 -J../mod -I/n/helmod/apps/centos7/MPI/gcc/7.1.0-fasrc01/openmpi/2.1.0-fasrc02/netcdf/4.3.2-fasrc05/include -I/n/helmod/apps/centos7/MPI/gcc/7.1.0-fasrc01/openmpi/2.1.0-fasrc02/netcdf-fortran/4.4.0-fasrc03/include -fdefault-real-8 -fdefault-double-8 -c -O1 -override-limits rrtmg_sw_rad.F90 rrtmg_lw_k_g.F90:4298:0: subroutine lw_kgb03 note: variable tracking size limit exceeded with -fvar-tracking-assignments, retrying without ar crs librad.a rrsw_cld.o rrlw_kg01.o mcica_random_numbers.o rrlw_kg09.o mcica_subcol_gen_sw.o rrlw_wvn.o rrsw_kg28.o rrtmg_sw_setcoef.o rrsw_kg23.o parkind.o rrlw_cld.o rrlw_ncpar.o rrsw_kg29.o rrlw_tbl.o rrsw_kg17.o rrlw_kg11.o rrtmg_sw_spcvmc.o rrsw_tbl.o rrtmg_sw_init.o parrrtm.o rrtmg_sw_k_g.o rrlw_kg04.o rrsw_kg18.o rrlw_kg02.o rrtmg_lw_k_g.o rrtmg_lw_init.o rrtmg_lw_taumol.o rrsw_con.o rrtmg_lw_rtrnmc.o rrsw_kg19.o rrtmg_sw_rad.o rrsw_kg21.o rrsw_kg27.o rrlw_kg16.o rrsw_kg26.o rrsw_ref.o rrtmg_lw_setcoef.o rrtmg_sw_reftra.o rrsw_aer.o rrsw_vsn.o rrlw_kg10.o rrlw_ref.o test_arr_mult.o rrlw_kg12.o rrlw_con.o parrrsw.o rrsw_kg22.o rrsw_kg20.o rrlw_vsn.o rrlw_kg15.o rrtmg_sw_taumol.o rrsw_kg25.o rrsw_kg16.o rrlw_kg03.o rrtmg_lw_cldprmc.o rrlw_kg05.o rrlw_kg14.o rrtmg_sw_cldprmc.o rrlw_kg06.o rrtmg_sw_vrtqdr.o rrlw_kg13.o rrlw_kg07.o rrlw_kg08.o rrsw_wvn.o rrtmg_lw_rad.o rrsw_kg24.o mcica_subcol_gen_lw.o ar: rrsw_cld.o: No such file or directory make: *** [lib] Error 1 make: Leaving directory `/n/home05/msulprizio/GC/Code.Dev/GeosRad' make: *** [librad] Error 2 make: Leaving directory `/n/home05/msulprizio/GC/Code.Dev/GeosCore' make: *** [lib] Error 2 make: Leaving directory `/n/home05/msulprizio/GC/Code.Dev/GeosCore' make: *** [all] Error 2 make: Leaving directory `/n/home05/msulprizio/GC/Code.Dev/GeosCore' make: *** [all] Error 2
- Heald, C.L., D.A. Ridley, J.H. Kroll, S.R.H. Barrett, K.E. Cady-Pereira, M.J. Alvarado, C.D. Holmes, Beyond Direct Radiative Forcing: The Case for Characterizing the Direct Radiative Effect of Aerosols, Atmos. Chem. Phys., 14, 5513-5527, doi:10.5194/acp-14-5513-2014, 2014. (Article)