GCHP v11-02

From Geos-chem
Jump to: navigation, search

GEOS-Chem Main Page | GCHP Main Page

Contents

Overview

This page contains the version history for GEOS-Chem High Performance (GCHP) v11-02. GCHP features the same science as GEOS-Chem using the standard "classic" capability (GCC) but operates on a cubed sphere grid and is parallelized using a message-passing interface (MPI) implementation. GCHP improves upon GCC by (1) enabling more accurate transport, and (2) providing efficient scaling making finer resolution global simulations possible.

Running GEOS-Chem with the high performance option requires use of source code from two repositories and run directory files from one repository maintained under version control. This page documents the mapping between the repositories for v11-02 as well as the features and bug fixes added specifically to the GCHP-only repository called GCHP. In v11-02, updates to the GCHP repository are documented separately from the primary GEOS-Chem v11-02 wiki page. Starting in GEOS-Chem 12, GCHP updates are listed alongside updates to GEOS-Chem Classic for each version.

Version Numbering System

GCHP v11-02 represents the first series of GCHP versions that are validated against GEOS-Chem Classic. Throughout the development of v11-02 we maintained a version numbering system for GCHP that matched GEOS-Chem Classic. This system used lettered version numbering for preliminary v11-02 releases containing collections of updates (e.g. v11-02a, v11-02b, etc). The most recent preliminary v11-02 release is v11-02-Release-Candidate and the final release will be v11-02-Final available late summer 2018.

This system originated when we only made new versions public every year or so, and used lettered version internally leading up to the public release. Now that all GEOS-Chem repositories are publicly available we are changing our GEOS-Chem Classic versioning system starting at the v11-02-Final release. We will be adopting a new version numbering system based loosely on semantic versioning. You can read more about the new system at the GEOS-Chem Classic version numbering system wiki page.

We are also rethinking how we will be versioning GCHP. Adopting the same version numbering system as GEOS-Chem Classic is not necessarily the best way to reflect major updates to GCHP since more often than not the major GCHP updates do not impact GEOS-Chem Classic. Stay tuned for more information soon on our new version numbering system. If you have ideas that you would like us to consider, please send them to the GEOS-Chem Support Team.

Version Mapping and Status

Each GCHP preliminary version of v11-02 (e.g. v11-02c) is tagged in the master branch of the repository. For the most part the GCHP and GEOS-Chem Classic tags are the same; however, there are a few exceptions that you should be on the lookout for by inspecting tags available in each repo with 'git tag'. This is similarly the case with the GEOS-Chem Unit Tester which we use to generate run directories for both GEOS-Chem Classic and GCHP. The final v11-02 release is also named 12.0.0 as we transition to a new GEOS-Chem version numbering system for the main GEOS-Chem source code repository.

The v11-02 system relies on checking out git tags within each repository such that all three required repositories are compatible. In a future version of GEOS-Chem we may use git submodules instead. This will allow easy checkout of repository versions that are compatible with each other.

The table below lists GCHP v11-02 preliminary version names, tags, status, and validation (if any) for each benchmarked release of GEOS-Chem Classic, as well as the final v11-02 version which is also called 12.0.0. The initial version, v1.0.0, which is not compatible with v11-02, is included for a complete history.

GCHP version Git Tag Status Validation
12.0.0 12.0.0 Complete Validation in progress
v11-02-release-candidate v11-02-release-candidate Complete No benchmark is planned for this version
v11-02e v11-02e Complete No benchmark was performed for this version
v11-02d v11-02d Complete No benchmark was performed for this version
v11-02c v11-02c Complete No benchmark was performed for this version
v11-02b v11-02b Complete

Benchmarks approved by GCSC:

GCHP_v1.0.0 none Complete No benchmark was performed for this version

Version Features

12.0.0

See also GEOS-Chem Classic 12.0.0 and the GCHP repository git history.

Feature Submitted by Type
Force usage of GCC libstdc++ rather than system version if using gfortran Lizzie Lundgren (GCST) Structural
Update ESMF build rules for compatibility with OpenMPI3 Lizzie Lundgren (GCST) Structural

v11-02 Release Candidate

See also GEOS-Chem Classic v11-02 Release Candidate and the GCHP repository git history.

Feature Submitted by Type
Bug fix to not rely on deprecated mpi_f77 library when compiling with gfortran and OpenMPI Lizzie Lundgren (GCST) Bug fix
Bug fix to turn off debug flags by default when compiling with gfortran Lizzie Lundgren (GCST) Bug fix
Updates for compatibility with GEOS-Chem v11-02 Release Candidate core Lizzie Lundgren (GCST) Structural

--Lizzie Lundgren (talk) 21:06, 28 June 2018 (UTC))

v11-02e

See also GEOS-Chem Classic v11-02e.

Feature Submitted by Type
Enable compatibility with GNU compiler collection version 6 and higher Yanko Davilo (University of Colorado) Bug fix
Assorted diagnostic improvements Lizzie Lundgren (GCST) Structural
Updates for compatibility with GEOS-Chem v11-02e core Lizzie Lundgren (GCST) Structural

--Lizzie Lundgren (talk) 22:50, 5 March 2018 (UTC)

v11-02d

See also GEOS-Chem Classic v11-02d.

Feature Submitted by Type
Set land/water/ice mask (LWI) from surface type fraction imports Lizzie Lundgren (GCST)

Sebastian Eastham (MIT)

Bug fix
Enable usage of MPICH for users without Infiniband Sebastian Eastham (MIT) Structural
Enable diagnostic output of State_Chm, State_Met, State_Diag, and HEMCO diagnostic arrays Lizzie Lundgren (GCST) Structural
Turn off warnings triggered by missing pressure-related grid variables in input file Sebastian Eastham (MIT) Structural

--Lizzie Lundgren (talk) 15:54, 17 October 2017 (UTC)

v11-02c

This benchmark was not formally benchmarked. See also GEOS-Chem Classic v11-02c.

Feature Submitted by Type
Set State_Met arrays previously set to all zeroes Lizzie Lundgren (GCST) Bug fix
Update initial restart file to reflect 2D initial mixing ratios applied at GCC runtime Lizzie Lundgren (GCST)

Sebastian Eastham (Harvard)

Restart
Structural updates for compatibility with v11-02c modifications Lizzie Lundgren (GCST) Structural
Store HEMCO restart variables in the internal state to enable use of HEMCO restart variables across continuous runs Sebastian Eastham (Harvard) Structural / Bug fix
Speed up file read time by consolidating LAI per land type variables in XLAI file Sebastian Eastham (Harvard) Structural
Structural updates for implementation of diagnostics (ongoing project) Lizzie Lundgren (GCST) Structural

--Lizzie Lundgren (talk) 15:55, 17 October 2017 (UTC)

v11-02b

Please see the following resources for complete information on validation of this version:

See also GEOS-Chem Classic v11-02b. Please note that diagnostics are not yet available in this version and GCHP benchmark materials are therefore limited to concentration comparisons.

Feature Submitted by Type
Enable use of GFortran compiler with GCHP Seb Eastham (Harvard) Structural
Update handling of the species restart file
  1. Allow omission of advected species from the restart file
  2. Add option to skip use of a species restart file and initialize all species concentrations from the species database
Lizzie Lundgren (GCST) Structural
Enable read-in of day-of-week scale factors in ExtData Seb Eastham (Harvard) Structural
Minor structural updates for compatibility with v11-02a modifications Lizzie Lundgren (GCST) Structural
Correct handling of monthly climatology in December Lizzie Lundgren (GCST) Bug fix
Enable starting simulation in early January Seb Eastham (Harvard) Bug fix
Correct compile failure if using certain versions of gmake Seb Eastham (Harvard) Bug fix
Make the Rn-Pb-Be7 simulation functional in GCHP Lizzie Lundgren (GCST) Structural

v1.0.0

GEOS-Chem with HP v1.0.0 is a preliminary version of the HP option in GEOS-Chem that is compatible with a version of GEOS-Chem base code built off of v11-01. As an initial version of GCHP it is included on this page for completeness even though it is not compatible with GEOS-Chem v11-02. Note that v1.0.0 did not undergo a formal benchmark review by the GEOS-Chem Steering Committee. Please see GEOS-Chem v11-01 HP validation wiki page for information about the internal benchmark of this version performed by the GCHP Development Team.

Feature Submitted by Type
Initial version for validation of scientific fidelity Mike Long (GCST)
Seb Eastham (Harvard)
Jiawei Zhuang (Harvard)
Bob Yantosca (GCST)
Lizzie Lundgren (GCST)
General

Validation

In this section we provide information about the benchmarks and tests that we have done to validate GCHP v11-02.

1-month and 1-year benchmarks

For complete information about the benchmark simulations used to validate GEOS-Chem v11-02, please see our GEOS-Chem v11-02 benchmark history wiki page.

Timing tests

We did not perform formal timing tests for GCHP v11-02. The next major GCHP version release will include performance improvements. We will perform timing tests when it is available.

Issues resolved in GCHP v11-02

GCHP is incompatible with GNU compiler v6 due to use of ESMF v5

This update was included in GCHP v11-02e.

Yanko Davila (University of Colorado) reported the following problem encountered while compiling GCHP with GNU Compiler Collection (GCC) version 6.1.0. He wrote:

I’m trying to compile the latest version of GCHP as per the wiki and I’m finding something interesting with gfortran. On file ESMCI_WebServNetEsmfServer.C there is a problem with line 1798 with the equal sign. For some reason gcc version 6.1.0 complains about a wrong function overloading. The following change works for me.
@@ -1795,7 +1795,7 @@ void  ESMCI_WebServNetEsmfServer::copyFile(
        fstream fin(srcFilename, ios::in | ios::binary);
        fstream fout(destFilename, ios::out | ios::binary);

-       if ((fin == NULL)  ||  (fout == NULL))
+       if ((!fin.is_open())  ||  (!fout.is_open()))

Sebastian Eastham (Harvard) explains the fix as follows:

The GMAO MAPL infrastructure in GCHP uses ESMF v5, which unfortunately is known to be incompatible with GCC v6 specifically because GCC v6 does not support the if (NULL == file) construct (https://trac.macports.org/ticket/52427). However, ESMF v7 - which GMAO are currently in the process of testing in an unstable branch of MAPL - does support GCC v6.

Due to the dependency of GCHP on ESMF v5 at this time and the expected update to ESMF v7 in the near future, we are not applying this fix into the GCHP source code for distribution. However, if you encounter this issue you may modify your local code using this fix.

--Lizzie Lundgren (talk) 21:29, 14 July 2017 (UTC)

First and last daily output files missing hourly data

This update was included in GEOS-Chem v11-02d Unit Tester.

File output is incomplete if outputting 1-hr time-averaged data to daily files using filename format %y4%m2%d2.nc in HISTORY.rc. For example, if running GCHP with 1-hr output frequency and 1 day duration for a 1 day simulation the output file should contain 24 time values. The first time should be 30 minutes after the start time, e.g. 00:30:00 if starting the simulation at midnight. However, if HISTORY.rc is configured to show time format string YYYYMMDD in the filename then only one time is in the file and it corresponds to the last of the 24 averaged taken over the day (e.g. 23:30:00).

This problem will also appear for other combinations of file duration, frequency, and filename timestamp string. You can avoid this problem by including a finer time unit in the filename. For the case above in which files are output daily, the problem is solved by including hour (%h2) in the filename timestamp string. If outputting files hourly then you should include minutes. In general, always include the time unit smaller than the file frequency time unit.

--Lizzie Lundgren (talk) 20:55, 14 December 2017 (UTC)

LWI import incorrectly regridded in GCHP

This update was included in GCHP v11-02d.

Prior to v11-02d State_Met%LWI was read in as a met-field import and regridded from lat/lon to cubed sphere in the process. This caused incorrect land/water/ice indexes which are supposed to be 0, 1, or 2 although they are stored as real numbers. Since the source of the LWI met-field is pre-processing using the surface fraction types from GMAO, we now use a similar approach within GCHP to create a 0/1/2 LWI mask from surface type fraction imports.

--Lizzie Lundgren (talk) 20:55, 12 December 2017 (UTC)

MPICH MPI implementation not readily usable with GCHP

This update was included in GCHP v11-02d.

MPICH is an open-source and widely portable implementation of MPI (MPICH). However, GCHP does not work out-of-the-box with MPICH prior to v11-02d. It instead works with MVAPICH2 which is a derivation of MPICH for use with the Infiniband interconnect. This was a problem for users without Infiniband, such as those using Ethernet interconnect.

Using MPICH with GCHP out-of-the-box is possible starting in v11-02d thanks to an update submitted by Sebastian Eastham (MIT). Seb writes:

I have now successfully built and run GCHP using MPICH, and have confirmed that it runs as quickly with MPICH as it does with MVAPICH2 (at least on a single machine and with 6 cores). Some notes:

  1. ESMF_COMM must be set to “mpich2”, NOT mpich. This reflects the fact that, when ESMF_COMM=mpich, some very old (and no longer relevant) compiler flags get set.
  2. The run command should now simply be “srun -n $SLURM_NTASKS ./geos” (i.e. do not use --mpi=pmi2).
  3. This has only been tested for 6 cores, with the current stable build of MPICH v3.2. I anticipate it could be slower than MVAPICH2 on very large jobs, due to the lack of Infiniband support.
  4. For reference, I configured MPICH with “./configure --prefix=$PREFIX --with-pm=no --with-pmi=slurm --with-slurm=/path/to/slurm”

We recommend that users who do not have access to an Infiniband interconnect use of MPICH over MVAPICH2.

--Lizzie Lundgren (talk) 16:49, 6 November 2017 (UTC)

MPI_Wait error when running at coarse resolution with high-resolution input data

This update was included in GCHP v11-02d.

MAPL crashes when regridding wind vector data from very high resolutions (e.g. 0.25x0.3125 to C24) if running on a small number of cores (e.g. 6). The problem appears to be the production of invalid MPI instructions within the scatter and gather routines. Sebastian Eastham (MIT) determined how to correct the problem if using MVAPICH2 or MPICH:

MVAPICH2

When using MVAPICH2, it’s critical that ch3:mrail interface is used. This is the preferred MVAPICH2 interface, but it only works if MVAPICH2 can use the Infiniband connector. Situations where problems will occur include when:

  • MVAPICH2 was explicitly configured using the deprecated ch3:nemesis interface. This was the case on Odyssey.
  • A (very) old version of MVAPICH2 is being used, which lacks the ch3:mrail interface.
  • MVAPICH2 is being run on a system where the default network fabric is ethernet, causing MVAPICH2 to default to the ch3:nemesis interface. When running using mpirun (i.e. using the MVAPICH2 dedicated process manager), this can be resolved by adding the argument “-iface ib0”, replacing ib0 with the Infiniband adapter name. I have not yet found a working solution if srun must be used.

This reflects a flaw in the MVAPICH2 ch3:nemesis interface, which is NOT present in MPICH. To that end…

MPICH

I have now successfully built and run GCHP using MPICH, and have confirmed that it runs as quickly with MPICH as it does with MVAPICH2 (at least on a single machine and with 6 cores). I have attached a git patch which allows a successful build. Some notes:

  • ESMF_COMM must be set to “mpich2”, NOT mpich. This reflects the fact that, when ESMF_COMM=mpich, some very old (and no longer relevant) compiler flags get set.
  • The run command should now simply be “srun -n $SLURM_NTASKS ./geos” (i.e. do not use --mpi=pmi2).
  • This has only been tested for 6 cores, with the current stable build of MPICH v3.2. I anticipate it could be slower than MVAPICH2 on very large jobs, due to the lack of Infiniband support.
  • For users who do not have access to an Infiniband interconnect, I think that we should be recommending the use of MPICH over MVAPICH2. This is on the basis that MVAPICH2 is essentially “a branch of MPICH + infiniband support”, and the MVAPICH2 ethernet interface is broken (for the purposes of GCHP).
  • For reference, I configured MPICH with “./configure --prefix=$PREFIX --with-pm=no --with-pmi=slurm --with-slurm=/path/to/slurm”

--Sebastian D. Eastham (talk) 16 October 2017 (UTC)

HEMCO restart variables are not read into ExtData

This update was included in GEOS-Chem v11-02c.

GCHP is not currently compatible with the HEMCO restart file and all HEMCO restart variables are set to /dev/null in ExtData.rc. This is an open issue which will be addressed in v11-02.

--Lizzie Lundgren (talk) 21:02, 13 June 2017 (UTC)

Some State_Met arrays never set beyond initialization to zero

This update was included in GCHP v11-02c.</span

Meteorology variables EFLUX, U, V, and OMEGA were all zeros in GCHP prior to v11-02c. EFLUX was imported via ExtData.rc but setting State_Met%EFLUX to the import values were omitted in Includes_Before_Run.H. OMEGA, U, and V (orthogonal lat-lon U and V) were not included in Chem_Registry.rc and therefore were not imported. State_Met%OMEGA, State_Met%U, and State_Met%V were initialized in GEOS-Chem but no values other than zero were set. OMEGA is used in sulfate chemistry and EFLUX, U, and V are all used in non-local PBL mixing.

Starting in v11-02c, all of these are variables are imported and the associated State_Met arrays are set. Variables U and V are imported as ULL and VLL to distinguish them from imports U and V that are rotated for the cubed-sphere.

--Lizzie Lundgren (talk) 16:22, 17 October 2017 (UTC)

Day-of-week scale factors not read in by ExtData

This update was included in GCHP v11-02b and approved on 14 August 2017.

Seb Eastham (Harvard) implemented an update in MAPL to allow use of day-of-week scale factors with emissions. He wrote:

Data which varies by day-of-the-week can now be read in. Such data must be contained within a single file, and must have exactly 7 or 84 time entries. If 7 entries, these must correspond to each day of the week, starting with Sunday. If 84, this must be 12 7-day sets, one set for each month. Time information will be largely ignored. This read-in mode can be selected by specifying "D" for the "Cyclic" flag in ExtData.

--Lizzie Lundgren (talk) 20:26, 12 June 2017 (UTC)

GFortran compiler not compatible with GCHP

This update was included in GCHP v11-02b and approved on 14 August 2017.

Seb Eastham (Harvard) implemented updates to enable compilation of the high performance option of GEOS-Chem with GFortran. He wrote:

Modifications to makefiles required for the build process to recognize, and correctly handle, the use of GCC as the only compiler. This set up was tested with GCC 5.4.0. The code was successfully compiled and run with MVAPICH2, and included the bug fixes in the next few commits:
  • The species identification routine in gchp_utils fails if "EOF" is reported from read_one_line, but EOF is never set. This has been fixed.
  • GC_F_INCLUDE is now only added to the "INC_NETCDF" variable if it is defined. Previously it was added once, and then again if defined, resulting in it being added twice.
  • The State_Met field MOLENGTH was being populated even when the array had not been allocated. This behavior resulted in a segfault with GFortran.
  • The default LUN for log writing is 700 in the code, but can be overwritten from the input directory. However, if the user selects LUN 6, this can cause problems, as units 5 and 6 are reserved for STDIN and STDOUT. ifort seemed to be able to cope but GFortran would die with mysterious errors. To prevent this, the code now stops if the user tries to select either of these LUNs.

--Lizzie Lundgren (talk) 20:31, 12 June 2017 (UTC)

Restart file must contain all advected species

This update was included in GCHP v11-02b and approved on 14 August 2017.

Lizzie Lundgren (GCST) implemented two updates to allow more flexibility in the use of restart files.

  1. Remove the requirement of including all advected species in the restart file: MAPL now changes the internal state RESTART attribute per species from initial category MAPL_RestartOptional to new category MAPL_RestartBootstrap if species are bootstrapped (default values used) rather than initialized from the restart file. This enables a later step of over-writing only the bootstrapped species with background concentration values stored in the species database, ultimately allowing the user to omit species from the restart file.
  2. Remove the requirement of using a restart file: If the user now enters '+none' for GIGCchem_INTERNAL_RESTART_FILE in the GCHP.rc config file, then new RESTART category MAPL_RestartSkipInitial is set for all species during initialization of the internal state. This has the same effect as using existing category MAPL_RestartSkip except that (1) background values from the species database are retrieved to overwrite default values set in MAPL, and (2) species are written to the output checkout point.

--Lizzie Lundgren (talk) 20:53, 12 June 2017 (UTC)

Bug in handling of monthly climatology in December

This update was included in GCHP v11-02b and approved on 14 August 2017.

Lizzie Lundgren (GCST) discovered a MAPL bug where the right time bracket is not correctly handled for climatology files during December simulations. This bug does not affect GCHP simulations at other times of the year.

Special climatology file handling is needed for edge months of the year (December and January) due to the year change and month number wrap-around. Special handling exists if the current month is January to properly get the climatology for the left bracket (previous) month. Similar handling should occur if the current month is December to properly get the climatology for the right bracket (next) month. Instead, the right bracket special handling is applied only if the current time is January and also improperly sets the right bracket month to December. This results in treating December as any other month for calculation of the right time bracket, thereby iterating the month number by one and subsequently attempting to access data for month 13.

The fix is to apply the right bracket special handling if the current month is December and to set variable imm to January. Below is the updated code in GCHP/Shared/MAPL_Base/MAPL_ExtDataGridCompMod.F90 subroutine UpdateBracketTime, with the bug fix shown in red:

          if (bSide == "R") then
             found=.false.
             newFile=.true.
             status = ESMF_SUCCESS
             If (MAPL_Am_I_Root().and.(Ext_Debug > 19)) Write(*,'(a,a,a,I5,x,2L1)') ' DEBUG: Sanity check on file ', trim(file_processed), ' with flags: ', status, status==ESMF_SUCCESS,found
             do while ((status==ESMF_SUCCESS).and.(.not.found))
                if (trim(cyclic)=='y') then
                   call ESMF_TimeGet(cTime,yy=iyr,mm=imm,dd=idd,h=ihr,m=imn,s=isc,__RC__)
                   if (imm == 12) then
                      cYear = iyr + 1
                      ! change year you will read from
                      iyr = climYear-1
                      call ESMF_TimeSet(readTime,yy=iyr,mm=imm,dd=idd,h=ihr,m=imn,s=isc,__RC__)
                      iyr = climYear
                      ! change month of file
                      imm = 1

--Lizzie Lundgren (talk) 15:58, 21 June 2017 (UTC)

Compile failure if using certain versions of gmake

This update was included in GCHP v11-02b approved on 14 August 2017.

Seb Eastham (Harvard) reported a bug that causes compile failure with certain versions of gmake. There is a typo in Line 72 of GCHP/Shared/GFDL_fms/GNUmakefile where the SUFFIXES command is called as follows:

  SUFFIXES: += .inc. 

The SUFFIXES command automatically adds its arguments to the existing list and therefore the extra "+=" is unnecessary and may get misinterpreted or rejected by some versions of gmake. If that happens then gmake will print the following message immediately after starting the GFDL_fms build and then skip the GFDL_fms build entirely:

  GNUmakefile:72: *** empty variable name.  Stop.

Note that compilation does not immediately fail and continues until when code needs to use libGFDL_fms.a. As a result, the signature of this error would probably be an error like this:

  File not found: ….libGFDL_fms.a

The solution is to remove the "+/=" from Line 72 of GCHP/Shared/GFDL_fms/GNUmakefile.

--Lizzie Lundgren (talk) 18:38, 12 July 2017 (UTC)

Simulation crashes if started prior to the first MMDD in climatology files

This update was included in GCHP v11-02b and approved on 14 August 2017.

Prior to GCHP v1.1.0 there was a bug where MAPL subroutine UpdateBracketTime would exit if the current datetime preceded the first datetime of a climatology dataset. This occurred if starting a simulation in January prior to January 15th which is the first date in the monthly XLAI 2008 files used as climatology. The bug resulted in run failure.

Seb Eastham (Harvard) corrected this issue by removing the forced exit and replacing it with returning ESMF_FAILURE if both the time precedes the earliest climatology file time and no file is found. For XLAI, a file is found and thus no failure is triggered. Updating bracket time to successfully look for the December file proceeds later in the routine as part of the existing handling of climatology edge months.

--Lizzie Lundgren (talk) 20:25, 12 July 2017 (UTC)

Rn-Pb-Be7 simulation not functional in GCHP in v1.0.0

This update was included in GCHP v11-02b and approved on 14 August 2017.

Only the standard simulation is functional in GCHP v1.0.0. The radon simulation will be implemented in v1.1.0 and a corresponding run directory will be available from the GEOS-Chem Unit tester.

--Lizzie Lundgren (talk) 21:49, 24 July 2017 (UTC)

Known issues not resolved in GCHP v11-02

Segmentation fault if using volcano emissions with NetCDF 4.2 and higher versions on some systems

Sebastian Eastham (MIT) and Jimmie Lin (PKU) have both reported a segmentation fault in GCHP if running a full chemistry simulation with VOLCANO emissions turned on. Both users get the error if using recent NetCDF libraries in which the C and Fortran compilers are split. These libraries are split in all NetCDF versions starting 4.2. However, we have not been able to replicate the problem using a split NetCDF version on the Harvard compute cluster.

The error message expected is as follows:

forrtl: severe (174): SIGSEGV, segmentation fault occurred
Image              PC                Routine            Line        Source             
libintlc.so.5      00002B6B08267961  Unknown               Unknown  Unknown
libintlc.so.5      00002B6B082660B7  Unknown               Unknown  Unknown
libmpi.so.12       00002B6B028154B2  Unknown               Unknown  Unknown
libmpi.so.12       00002B6B02815306  Unknown               Unknown  Unknown
libmpi.so.12       00002B6B02803B5C  Unknown               Unknown  Unknown
libmpi.so.12       00002B6B027E54D8  Unknown               Unknown  Unknown
libpthread.so.0    00002B6B084C6330  Unknown               Unknown  Unknown
libnetcdff.so.6    00002B6B02D6FE44  Unknown               Unknown  Unknown
libnetcdff.so.6    00002B6B02DF4BB5  Unknown               Unknown  Unknown
geos               0000000000C7DC7E  Unknown               Unknown  Unknown
geos               0000000000C62A8F  Unknown               Unknown  Unknown
geos               0000000000A8412A  Unknown               Unknown  Unknown
geos               0000000000A7E167  Unknown               Unknown  Unknown
geos               0000000000A71CA8  Unknown               Unknown  Unknown
libesmf.so         00002B6B01A23190  Unknown               Unknown  Unknown
libesmf.so         00002B6B01A232BA  Unknown               Unknown  Unknown
libesmf.so         00002B6B01B767B1  Unknown               Unknown  Unknown
libesmf.so         00002B6B01B6F95F  Unknown               Unknown  Unknown
libesmf.so         00002B6B01A2486E  Unknown               Unknown  Unknown
libesmf.so         00002B6B01C699F5  esmf_compmod_mp_e         991  ESMF_Comp.F90
libesmf.so         00002B6B01E3EA27  esmf_gridcompmod_        1444  ESMF_GridComp.F90
geos               0000000000A6451B  Unknown               Unknown  Unknown
geos               00000000008B264C  MAIN__                     38  GEOSChem.F90
geos               000000000040F6FE  Unknown               Unknown  Unknown
libc.so.6          00002B6B088F9F45  Unknown               Unknown  Unknown
geos               000000000040F609  Unknown               Unknown  Unknown

It will directly follow a series of warning messages that are triggered by the volcano emissions file:

problem in getting akid in NF90_INQ_VARID 

Until we identify and correct the problem we recommend that users who encounter this error either turn off volcano emissions or try compiling and running with a pre-4.2 version of NetCDF.

--Lizzie Lundgren (talk) 17:22, 14 December 2017 (UTC)

Resolution dependency of emissions

Resolution dependencies within GEOS-Chem prevent users from full utilizing the resolution-independence feature of GCHP. This is particularly an issue for lightning and dust emissions. On this topic, Seb Eastham writes:

The existing approach in GEOS-Chem Classic results in inconsistency between simulations at different resolutions. Any emissions extension which included a non-linear function of meteorological data would result in different spatial and temporal emissions distributions over time, depending on the simulation resolution. In the case of lightning emissions, the distribution is constrained in GC Classic by application of redistribution factors, but these factors must be regenerated for each simulation year, resolution, and met product. For both dust and lightning, the total domain emissions mass is further constrained by a single fudge factor, calculated once for each met product, resolution, and subgrid, meaning a new fudge factor is required for every possible nested domain. In the case of other emissions extensions, such as MEGAN and sea salt, no constraint at all is applied, meaning that the emissions are not guaranteed or even likely to be consistent between different resolutions. While this was already becoming a problem in GEOS-Chem Classic, it is untenable in a system like GCHP where one of the unique selling points is flexibility of resolution.

--Lizzie Lundgren (talk) 17:57, 9 August 2017 (UTC)

Time can only flow forward in MAPL

Time must be able to flow backwards in all components of GEOS-Chem in order for the adjoint to be compatible with GCHP. Seb Eastham (Harvard) writes:

To run GCHP backwards, I think there are two phases of work that will be needed.

The first is just reconditioning MAPL to be able to deal with backwards flow of time. This means disabling some of the hardcoded error handling, such as in the case Melissa pointed out where we ran afoul of the line "ASSERT_(HEARTBEAT_DT>=0" in MAPL_Cap.F90. Some more significant plumbing work will also be needed to deal with the way that modules are called. Routines in ESMF/MAPL run when "alarms" are triggered, meaning that the time has progressed past a certain point. My expectation is that the majority of the changes will just involve adding logic for a negative timestep which flips inequalities, i.e. "If time > alarm_time" becomes "If time < alarm_time". In these instances the absolute value of "time" doesn't matter, just how much has passed since the last alarm. Most of the rest of the code is pretty agnostic about the flow of time, caring only about what the current value is.

The second phase will be modifying the exceptions to this agnosticism, by far the biggest of which is ExtData. This is because a) the time dimensions in all input files are required to increase with index, which complicates the idea of just reversing everything, and b) ExtData must at all times know what file is coming next so that it can perform interpolation. Again, this doesn't seem too difficult to solve, but it will require either a creative solution or a lot of logical tests. A lesser but related case will be the modifications needed to the output module, History.

--Lizzie Lundgren (talk) 21:12, 13 June 2017 (UTC)

Timestep midpoint cos(SZA) is set to cos(SZA) at start of timestep

State_Met variables SUNCOS and SUNCOSmid are both set to internal state variable zenith which is extracted at the start of each chemistry timestep in MAPL. This causes out-of-the-box differences with GEOS-Chem "classic" (GCC). This is an open issue which will be addressed in v11-02 as part of our efforts to make GCHP and GCC out-of-the-box standard simulations more comparable.

GFortran + MVAPICH2 runs hang if using multiple nodes

While GCHP successfully compiles with GFortran starting in v1.1.0 (compatible with GEOS-Chem v11-02b), runs hang with an MPI_WAIT error if using cores across multiple nodes. This issue is not guaranteed; it has been observed on the Harvard Odyssey cluster with some builds, but there have also been successful GCHP simulations across multiple nodes when using a different set of GFortran and MVAPICH2 versions. The root cause is not yet known.

--Lizzie Lundgren (talk) 21:35, 15 June 2017 (UTC)

C preprocessor inserting GNU compiler version 6.1.0 information text into MAPL_COMMS___.f90

Yanko Davila (University of Colorado) reported that GNU C library license information was inserted in GCHP file MAPL_COMMS____.f90 when using GCC v6.1.0 causing compile failure. Seb Eastham (Harvard) recommended the following fix which is currently being tested. We will incorporate this fix into the standard GCHP source code once it is more thoroughly tested.

In GCHP/Shared/Config/ESMA_base.mk, change line 302 (or thereabouts), specifically the CPPANSIX line for gfortran.

Change this line of code:

CPPANSI="-std=gnu11 -C" 

to this:

CPPANSI="-std=gnu11 -nostdinc -C"

--Lizzie Lundgren (talk) 21:43, 14 July 2017 (UTC)


GEOS-Chem Main Page | GCHP Main Page