Difference between revisions of "Setting Up the GCHP Environment"

From Geos-chem
Jump to: navigation, search
(→‎Recent Changes)
m
 
(13 intermediate revisions by one other user not shown)
Line 1: Line 1:
 +
----
 +
<span style="color:crimson;font-size:120%">'''The GCHP documentation has moved to https://gchp.readthedocs.io/.''' The GCHP documentation on http://wiki.seas.harvard.edu/ will stay online for several months, but it is outdated and no longer active!</span>
 +
----
 +
 
__FORCETOC__
 
__FORCETOC__
−
'''''[[Obtaining_a_GCHP_Run_Directory|Previous]] | [[Compiling_GCHP|Next]] |  [[Getting Started With GCHP]] | [[GCHP Main Page]]'''''
+
'''''[[GCHP_Hardware_and_Software_Requirements|Previous]] | [[Downloading_GCHP|Next]] |  [[Getting Started with GCHP]] | [[GCHP Main Page]]'''''
 
#[[GCHP_Hardware_and_Software_Requirements|Hardware and Software Requirements]]
 
#[[GCHP_Hardware_and_Software_Requirements|Hardware and Software Requirements]]
−
#[[Downloading_GCHP|Downloading Source Code and Data Directories]]
 
−
#[[Obtaining_a_GCHP_Run_Directory|Obtaining a Run Directory]]
 
 
#<span style="color:blue">'''Setting Up the GCHP Environment'''</span>
 
#<span style="color:blue">'''Setting Up the GCHP Environment'''</span>
 +
#[[Downloading_GCHP|Downloading Source Code and Data Directories]]
 
#[[Compiling_GCHP|Compiling]]
 
#[[Compiling_GCHP|Compiling]]
 +
#[[Obtaining_a_GCHP_Run_Directory|Obtaining a Run Directory]]
 
#[[Running_GCHP:_Basics|Running GCHP: Basics]]
 
#[[Running_GCHP:_Basics|Running GCHP: Basics]]
 
#[[Running_GCHP:_Configuration|Running GCHP: Configuration]]
 
#[[Running_GCHP:_Configuration|Running GCHP: Configuration]]
Line 12: Line 16:
 
#[[GCHP_Run_Configuration_Files|Run Configuration Files]]
 
#[[GCHP_Run_Configuration_Files|Run Configuration Files]]
 
<br>
 
<br>
−
 
−
== Recent Changes ==
 
−
 
−
Please note that starting in GCHP 12.5.0 the environment file must include defining environment variable gFTL. If you have an existing environment file, please add the following when upgrading to GCHP 12.5.0:
 
−
 
−
# Set path to GMAO Fortran template library (gFTL)
 
−
export gFTL=$(readlink -f ./gFTL)
 
  
 
== Create an Environment File ==
 
== Create an Environment File ==
  
−
You must load all necessary libraries and export certain environment variables before compiling GCHP. The GCHP environment is different from GEOS-Chem Classic and is often considered the largest obstacle to getting GCHP up and running for the first time. We have tried to make setting libraries and variables as automatic as possible to minimize problems. However, libraries will always be specific to your local compute cluster which presents challenges for compatibility. We recommend simplifying the environment setup process by customizing a GCHP-specific environment file that works on your system and saving it for future work.
+
You must load all necessary libraries and export certain environment variables before compiling and running GCHP. We recommend simplifying the environment setup process by customizing one or more GCHP-specific environment files that work on your system and storing them in a central location for repeated use.
−
 
+
−
Sample environment files are included in the run directory, several for the [https://www.rc.fas.harvard.edu/resources/odyssey-architecture/ Harvard University Odyssey cluster] and one for a more generic Linux system. These are located in the <code>environmentFileSamples</code> subdirectory. You can use these to develop one compatible with your system. Each sample environment file is customized for a specific combination of Fortran compiler, MPI implementation, netCDF libraries, and compute cluster. For clarity we recommend using the naming format <tt>gchp.compiler_mpi_cluster.env</tt>. For example, <code>gchp.ifort17_openmpi3_computecanada.env</code>. Open several of the sample environment files and getting familiar with the environment variables set.
+
  
−
An example of environment variables needed for GCHP are as follows. In this example a version of netCDF is used that does not break up the C and Fortran libraries. Setting environment variables for netCDF-Fortran is therefore commented out. For more discussion about this, and whether you need to do it based on your netCDF library, see the [[GCHP_Hardware_and_Software_Requirements#NetCDF_Libraries|netCDF libraries section of this guide]].
+
An example of an environment file is below. This particular file was also used to build ESMF and therefore includes ESMF-only environment variables as well as those required by GCHP. It also sets certain system limits to avoid potential problems with memory.
  
 
  if [[ $- = *i* ]] ; then
 
  if [[ $- = *i* ]] ; then
−
   echo "Loading modules for GCHP on Odyssey, please wait ..."
+
   echo "Loading modules, please wait ..."
 
  fi
 
  fi
 
   
 
   
 
  #==============================================================================
 
  #==============================================================================
−
  # %%%%% Clear existing environment variables %%%%%
+
  # Load modules
−
#==============================================================================
+
−
unset GC_BIN
+
−
unset GC_INCLUDE
+
−
unset GC_LIB
+
−
unset GC_F_BIN
+
−
unset GC_F_INCLUDE
+
−
unset GC_F_LIB
+
−
+
−
#==============================================================================
+
−
# Modules (specific to compute cluster)
+
 
  #==============================================================================
 
  #==============================================================================
 
   
 
   
 
  module purge
 
  module purge
−
  module load git/2.17.0-fasrc01  
+
  module load git/2.17.0-fasrc01
−
+
  module load intel/18.0.5-fasrc01
−
# Modules for CentOS7
+
  module load openmpi/4.0.1-fasrc01
−
  module load intel/17.0.4-fasrc01
+
  module load netcdf-fortran/4.5.2-fasrc01
−
  module load openmpi/3.1.1-fasrc01
+
module load cmake/3.16.1-fasrc01
−
  module load netcdf/4.1.3-fasrc03
+
 
   
 
   
 
  #==============================================================================
 
  #==============================================================================
 
  # Environment variables
 
  # Environment variables
−
  #==============================================================================  
+
  #==============================================================================
 
   
 
   
 
  # Make all files world-readable by default
 
  # Make all files world-readable by default
−
  umask 022  
+
  umask 022
 
   
 
   
 
  # Specify compilers
 
  # Specify compilers
 
  export CC=gcc
 
  export CC=gcc
−
export OMPI_CC=$CC
 
−
 
 
  export CXX=g++
 
  export CXX=g++
−
  export OMPI_CXX=$CXX
+
  export FC=ifort
 
   
 
   
−
  export FC=ifort
+
  # MPI
−
  export F77=$FC
+
  export MPI_ROOT=${MPI_HOME}
−
  export F90=$FC
+
   
−
  export OMPI_FC=$FC
+
  # For ESMF
−
export COMPILER=$FC
+
 
  export ESMF_COMPILER=intel
 
  export ESMF_COMPILER=intel
−
 
−
# MPI Communication
 
 
  export ESMF_COMM=openmpi
 
  export ESMF_COMM=openmpi
−
  export MPI_ROOT=$MPI_HOME
+
  export ESMF_DIR=/n/jacob_lab/Lab/ESMF/ESMF_8_0_1
−
+
  export ESMF_INSTALL_PREFIX=${ESMF_DIR}/INSTALL_ifort18_openmpi4
−
# Base paths
+
−
export GC_BIN="$NETCDF_HOME/bin"
+
−
export GC_INCLUDE="$NETCDF_HOME/include"
+
−
export GC_LIB="$NETCDF_HOME/lib"
+
−
+
−
# Add to primary path
+
−
export PATH=${NETCDF_HOME}/bin:$PATH
+
−
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${NETCDF_HOME}/lib
+
−
   
+
−
# If using NetCDF after the C/Fortran split (4.3+), then you will need to
+
−
# specify the following additional environment variables
+
−
#export GC_F_BIN="$NETCDF_FORTRAN_HOME/bin"
+
−
#export GC_F_INCLUDE="$NETCDF_FORTRAN_HOME/include"
+
−
#export GC_F_LIB="$NETCDF_FORTRAN_HOME/lib"
+
−
#export PATH=${NETCDF_FORTRAN_HOME}/bin:$PATH
+
−
#export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${NETCDF_FORTRAN_HOME}/lib
+
−
+
−
# Set ESMF optimization (g=debugging, O=optimized (capital o))
+
−
export ESMF_BOPT=O
+
−
+
−
# Set path to GMAO Fortran template library (gFTL)
+
−
export gFTL=$(readlink -f ./gFTL)
+
 
   
 
   
−
  # Specify number number of job slots for build
+
  # For GCHP
−
  export NUM_JOB_SLOTS=8
+
  export ESMF_ROOT=${ESMF_INSTALL_PREFIX}
 
   
 
   
 
  #==============================================================================
 
  #==============================================================================
−
  # Raise memory limits
+
  # Set limits
 
  #==============================================================================
 
  #==============================================================================
 
   
 
   
−
  ulimit -c unlimited              # coredumpsize
+
  ulimit -c 0                      # coredumpsize
 
  ulimit -l unlimited              # memorylocked
 
  ulimit -l unlimited              # memorylocked
 
  ulimit -u 50000                  # maxproc
 
  ulimit -u 50000                  # maxproc
Line 116: Line 73:
 
   
 
   
 
  #==============================================================================
 
  #==============================================================================
−
  # Print information for clarity
+
  # Print information
 
  #==============================================================================
 
  #==============================================================================
 
   
 
   
 
  module list
 
  module list
 
  echo ""
 
  echo ""
−
  echo "Environment variables set:"
+
  echo "Environment:"
−
echo ""
+
−
echo "LD_LIBRARY_PATH: ${LD_LIBRARY_PATH}"
+
−
echo ""
+
−
echo "ESMF_COMM: ${ESMF_COMM}"
+
−
echo "ESMP_BOPT: ${ESMF_BOPT}"
+
−
echo "MPI_ROOT: ${MPI_ROOT}"
+
 
  echo ""
 
  echo ""
 
  echo "CC: ${CC}"
 
  echo "CC: ${CC}"
−
echo "OMPI_CC: ${OMPI_CC}"
 
−
echo ""
 
 
  echo "CXX: ${CXX}"
 
  echo "CXX: ${CXX}"
−
echo "OMPI_CXX: ${OMPI_CXX}"
 
−
echo ""
 
 
  echo "FC: ${FC}"
 
  echo "FC: ${FC}"
−
  echo "F77: ${F77}"
+
  echo "ESMF_COMM: ${ESMF_COMM}"
−
echo "F90: ${F90}"
+
−
echo "OMPI_FC: ${OMPI_FC}"
+
−
echo "COMPILER: ${COMPILER}"
+
 
  echo "ESMF_COMPILER: ${ESMF_COMPILER}"
 
  echo "ESMF_COMPILER: ${ESMF_COMPILER}"
 +
echo "ESMF_DIR: ${ESMF_DIR}"
 +
echo "ESMF_INSTALL_PREFIX: ${ESMF_INSTALL_PREFIX}"
 +
echo "ESMF_ROOT: ${ESMF_ROOT}"
 +
echo "MPI_HOME: ${MPI_HOME}"
 +
echo "NetCDF C: $(nc-config --prefix)"
 +
echo "NetCDF Fortran: $(nf-config --prefix)"
 
  echo ""
 
  echo ""
−
echo "GC_BIN: ${GC_BIN}"
 
−
echo "GC_INCLUDE: ${GC_INCLUDE}"
 
−
echo "GC_LIB: ${GC_LIB}"
 
−
echo ""
 
−
#echo "GC_F_BIN: ${GC_F_BIN}"
 
−
#echo "GC_F_INCLUDE: ${GC_F_INCLUDE}"
 
−
#echo "GC_F_LIB: ${GC_F_LIB}"
 
−
#echo ""
 
 
  echo "Done sourcing ${BASH_SOURCE[0]}"
 
  echo "Done sourcing ${BASH_SOURCE[0]}"
  
Line 174: Line 116:
 
  virtual memory          (kbytes, -v) unlimited
 
  virtual memory          (kbytes, -v) unlimited
 
  file locks                      (-x) unlimited
 
  file locks                      (-x) unlimited
−
 
−
All example environment files for GCHP explicitly set several of these to unlimited, as shown above in the example. If you run into a memory issue be sure to check your limits against the list above to see if anything may be limiting your run.
 
−
 
−
== Expanding MPI Options (Advanced) ==
 
−
 
−
GCHP is currently not tested with MPI implementations other than OpenMPI3. However, we encourage users to experiment with other MPI implementations. To do this, follow the steps below. You may need to make tweaks and it is possible it may fail. Please report what you are trying and your results by opening a [https://github.com/geoschem/gchp/issues GitHub GCHP issue] whether you fail or succeed. Before you begin, check the [[GCHP_Hardware_and_Software_Requirements#MPI|MPI section of this guide]] to see if there is any news about known issues.
 
−
 
−
#Specify environment variable <tt>ESMF_COMM</tt> to match the MPI implementation. Options are currently in place for MVAPICH2 (ESMF_COMM=mvapich2), OpenMPI (ESMF_COMM=openmpi), and a generic MPI implementation. (ESMF_COMM=mpi). The generic option is sufficient when, for example, running with the SGI MPI implementation on NASA's Pleiades servers. If you are using a new MPI implementation not covered by one of these options, we recommend running first with ESMF_COMM=mpi.
 
−
#Check that ${MPI_HOME} exists and contains the path to your MPI library. It should be set automatically when you load the library.
 
−
#Ensure that you have valid <tt>mpif90</tt> and <tt>mpifort</tt> executables. These almost always perform the same role but both names are invoked in the build sequence. If you have one but not the other, we strongly recommend that you make a symbolic link to the working binary with the name of the missing binary in a dedicated folder, and then add that folder to your path at the command line and in your <tt>.bashrc</tt>. For example, if you have a <tt>mpifort</tt> binary but not an <tt>mpif90</tt> binary, run the following commands:
 
−
mkdir $HOME/mpi_extra
 
−
cd $HOME/mpi_extra
 
−
ln -s $( which mpifort ) mpif90
 
−
export PATH=${PATH}:${HOME}/mpi_extra
 
−
 
−
You should now try to compile GCHP. If the generic option does not work then you will need to implement a new option. This involves updating GCHP source code. An example of how to do this for Intel MPI is as follows:
 
−
#Decide a new name, such as <tt>ESMF_COMM=intel</tt> for the Intel MPI implementation.
 
−
#Determine the relevant include path and linker commands for your MPI implementation. In this example for Intel MPI they are <tt>$(MPI_ROOT)/include</tt> and <tt>-L$(MPI_ROOT)/lib -lmpi -lmpi++</tt> respectively.
 
−
#Update source code files <tt>CodeDir/GCHP/GIGC.mk</tt> and <tt>CodeDir/GCHP/Shared/Config/ESMA_base.mk</tt>. In both files, search for environment variable <tt>ESMF_COMM</tt> in the file. You should find a small set of occurrences in a single "if..else.." block. Add a new clause below the one for mvapich2 as follows.
 
−
 
−
In <tt>GIGC.mk</tt>:
 
−
else ifeq ($(ESMF_COMM),intel)
 
−
    # %%%%% Intel MPI %%%%%
 
−
    MPI_LIB    := -L$(MPI_ROOT)/lib -lmpi -lmpi++
 
−
 
−
In <tt>ESMA_base.mk</tt>:
 
−
else ifeq ($(ESMF_COMM),intel)
 
−
    INC_MPI := $(MPI_ROOT)/include
 
−
    LIB_MPI := -L$(MPI_ROOT)/lib -lmpi -lmpi++
 
  
 
--------------------------------------
 
--------------------------------------
  
−
'''''[[Obtaining_a_GCHP_Run_Directory|Previous]] | [[Compiling_GCHP|Next]] | [[Getting Started With GCHP]] | [[GCHP Main Page]]'''''
+
'''''[[GCHP_Hardware_and_Software_Requirements|Previous]] | [[Downloading_GCHP|Next]] | [[Getting Started with GCHP]] | [[GCHP Main Page]]'''''

Latest revision as of 15:40, 8 December 2020

The GCHP documentation has moved to https://gchp.readthedocs.io/. The GCHP documentation on http://wiki.seas.harvard.edu/ will stay online for several months, but it is outdated and no longer active!


Previous | Next | Getting Started with GCHP | GCHP Main Page

  1. Hardware and Software Requirements
  2. Setting Up the GCHP Environment
  3. Downloading Source Code and Data Directories
  4. Compiling
  5. Obtaining a Run Directory
  6. Running GCHP: Basics
  7. Running GCHP: Configuration
  8. Output Data
  9. Developing GCHP
  10. Run Configuration Files


Create an Environment File

You must load all necessary libraries and export certain environment variables before compiling and running GCHP. We recommend simplifying the environment setup process by customizing one or more GCHP-specific environment files that work on your system and storing them in a central location for repeated use.

An example of an environment file is below. This particular file was also used to build ESMF and therefore includes ESMF-only environment variables as well as those required by GCHP. It also sets certain system limits to avoid potential problems with memory. 

if  $- = *i*  ; then
  echo "Loading modules, please wait ..."
fi

#==============================================================================
# Load modules
#==============================================================================

module purge
module load git/2.17.0-fasrc01
module load intel/18.0.5-fasrc01
module load openmpi/4.0.1-fasrc01
module load netcdf-fortran/4.5.2-fasrc01
module load cmake/3.16.1-fasrc01

#==============================================================================
# Environment variables
#==============================================================================

# Make all files world-readable by default
umask 022

# Specify compilers
export CC=gcc
export CXX=g++
export FC=ifort

# MPI
export MPI_ROOT=${MPI_HOME}

# For ESMF
export ESMF_COMPILER=intel
export ESMF_COMM=openmpi
export ESMF_DIR=/n/jacob_lab/Lab/ESMF/ESMF_8_0_1
export ESMF_INSTALL_PREFIX=${ESMF_DIR}/INSTALL_ifort18_openmpi4

# For GCHP
export ESMF_ROOT=${ESMF_INSTALL_PREFIX}

#==============================================================================
# Set limits
#==============================================================================

ulimit -c 0                      # coredumpsize
ulimit -l unlimited              # memorylocked
ulimit -u 50000                  # maxproc
ulimit -v unlimited              # vmemoryuse
ulimit -s unlimited              # stacksize

#==============================================================================
# Print information
#==============================================================================

module list
echo ""
echo "Environment:"
echo ""
echo "CC: ${CC}"
echo "CXX: ${CXX}"
echo "FC: ${FC}"
echo "ESMF_COMM: ${ESMF_COMM}"
echo "ESMF_COMPILER: ${ESMF_COMPILER}"
echo "ESMF_DIR: ${ESMF_DIR}"
echo "ESMF_INSTALL_PREFIX: ${ESMF_INSTALL_PREFIX}"
echo "ESMF_ROOT: ${ESMF_ROOT}"
echo "MPI_HOME: ${MPI_HOME}"
echo "NetCDF C: $(nc-config --prefix)"
echo "NetCDF Fortran: $(nf-config --prefix)"
echo ""
echo "Done sourcing ${BASH_SOURCE[0]}"

System memory limits and stack size should be set to unlimited to avoid memory problems. Such problems would manifest as sudden termination upon file read or a segmentation fault during advection. You can find out what you system limits are by typing the following at the command prompt: 

ulimit -a

You will see something like this: 

core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 1030083
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 100000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) unlimited
cpu time               (seconds, -t) unlimited
max user processes              (-u) 4096
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Previous | Next | Getting Started with GCHP | GCHP Main Page

Retrieved from "https://wiki.seas.harvard.edu/geos-chem/index.php?title=Setting_Up_the_GCHP_Environment&oldid=48669"