Difference between revisions of "Setting Up the GCHP Environment"

From Geos-chem
Jump to: navigation, search
(Create an Environment File)
Line 15: Line 15:
 
== Create an Environment File ==
 
== Create an Environment File ==
  
You must load all necessary libraries and export certain environment variables before compiling GCHP. The GCHP environment is different from GEOS-Chem Classic and is often considered the largest obstacle to getting GCHP up and running for the first time. We have tried to make setting libraries and variables as automatic as possible to minimize problems. However, libraries will always be specific to your local compute cluster which presents challenges for compatibility. We recommend simplifying the environment setup process by customizing a GCHP-specific environment file that works on your system and saving it for future work.
+
You must load all necessary libraries and export certain environment variables before compiling GCHP. We recommend simplifying the environment setup process by customizing one or more GCHP-specific environment files that work on your system and storing them in a central location for repeated use.
  
Sample environment files are included in the run directory, several for the [https://www.rc.fas.harvard.edu/resources/odyssey-architecture/ Harvard University Odyssey cluster] and one for a more generic Linux system. These are located in the <code>environmentFileSamples</code> subdirectory. You can use these to develop one compatible with your system. Each sample environment file is customized for a specific combination of Fortran compiler, MPI implementation, netCDF libraries, and compute cluster. For clarity we recommend using the naming format <tt>gchp.compiler_mpi_cluster.env</tt>. For example, <code>gchp.ifort17_openmpi3_computecanada.env</code>. Open several of the sample environment files and getting familiar with the environment variables set.
+
An example of an environment file is below. This particular file was also used to build ESMF and therefore includes ESMF-only environment variables as well as those required by GCHP. It also sets certain system limits to avoid potential problems with memory.
 
+
An example of environment variables needed for GCHP are as follows. In this example a version of netCDF is used that does not break up the C and Fortran libraries. Setting environment variables for netCDF-Fortran is therefore commented out. For more discussion about this, and whether you need to do it based on your netCDF library, see the [[GCHP_Hardware_and_Software_Requirements#NetCDF_Libraries|netCDF libraries section of this guide]].
+
  
 
  if [[ $- = *i* ]] ; then
 
  if [[ $- = *i* ]] ; then
   echo "Loading modules for GCHP on Odyssey, please wait ..."
+
   echo "Loading modules, please wait ..."
 
  fi
 
  fi
 
   
 
   
 
  #==============================================================================
 
  #==============================================================================
  # %%%%% Clear existing environment variables %%%%%
+
  # Load modules
#==============================================================================
+
unset GC_BIN
+
unset GC_INCLUDE
+
unset GC_LIB
+
unset GC_F_BIN
+
unset GC_F_INCLUDE
+
unset GC_F_LIB
+
+
#==============================================================================
+
# Modules (specific to compute cluster)
+
 
  #==============================================================================
 
  #==============================================================================
 
   
 
   
 
  module purge
 
  module purge
  module load git/2.17.0-fasrc01  
+
  module load git/2.17.0-fasrc01
+
  module load intel/18.0.5-fasrc01
# Modules for CentOS7
+
  module load openmpi/4.0.1-fasrc01
  module load intel/17.0.4-fasrc01
+
  module load netcdf-fortran/4.5.2-fasrc01
  module load openmpi/3.1.1-fasrc01
+
module load cmake/3.16.1-fasrc01
  module load netcdf/4.1.3-fasrc03
+
 
   
 
   
 
  #==============================================================================
 
  #==============================================================================
 
  # Environment variables
 
  # Environment variables
  #==============================================================================  
+
  #==============================================================================
 
   
 
   
 
  # Make all files world-readable by default
 
  # Make all files world-readable by default
  umask 022  
+
  umask 022
 
   
 
   
 
  # Specify compilers
 
  # Specify compilers
 
  export CC=gcc
 
  export CC=gcc
export OMPI_CC=$CC
 
 
 
  export CXX=g++
 
  export CXX=g++
  export OMPI_CXX=$CXX
+
  export FC=ifort
 
   
 
   
  export FC=ifort
+
  # MPI
  export F77=$FC
+
  export MPI_ROOT=${MPI_HOME}
  export F90=$FC
+
   
  export OMPI_FC=$FC
+
  # For ESMF
export COMPILER=$FC
+
 
  export ESMF_COMPILER=intel
 
  export ESMF_COMPILER=intel
 
# MPI Communication
 
 
  export ESMF_COMM=openmpi
 
  export ESMF_COMM=openmpi
  export MPI_ROOT=$MPI_HOME
+
  export ESMF_DIR=/n/jacob_lab/Lab/ESMF/ESMF_8_0_1
+
  export ESMF_INSTALL_PREFIX=${ESMF_DIR}/INSTALL_ifort18_openmpi4
# Base paths
+
export GC_BIN="$NETCDF_HOME/bin"
+
export GC_INCLUDE="$NETCDF_HOME/include"
+
export GC_LIB="$NETCDF_HOME/lib"
+
+
# Add to primary path
+
export PATH=${NETCDF_HOME}/bin:$PATH
+
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${NETCDF_HOME}/lib
+
   
+
# If using NetCDF after the C/Fortran split (4.3+), then you will need to
+
# specify the following additional environment variables
+
#export GC_F_BIN="$NETCDF_FORTRAN_HOME/bin"
+
#export GC_F_INCLUDE="$NETCDF_FORTRAN_HOME/include"
+
#export GC_F_LIB="$NETCDF_FORTRAN_HOME/lib"
+
#export PATH=${NETCDF_FORTRAN_HOME}/bin:$PATH
+
#export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:${NETCDF_FORTRAN_HOME}/lib
+
+
# Set ESMF optimization (g=debugging, O=optimized (capital o))
+
export ESMF_BOPT=O
+
+
# Set path to GMAO Fortran template library (gFTL)
+
export gFTL=$(readlink -f ./gFTL)
+
 
   
 
   
  # Specify number number of job slots for build
+
  # For GCHP
  export NUM_JOB_SLOTS=8
+
  export ESMF_ROOT=${ESMF_INSTALL_PREFIX}
 
   
 
   
 
  #==============================================================================
 
  #==============================================================================
  # Raise memory limits
+
  # Set limits
 
  #==============================================================================
 
  #==============================================================================
 
   
 
   
Line 109: Line 69:
 
   
 
   
 
  #==============================================================================
 
  #==============================================================================
  # Print information for clarity
+
  # Print information
 
  #==============================================================================
 
  #==============================================================================
 
   
 
   
 
  module list
 
  module list
 
  echo ""
 
  echo ""
  echo "Environment variables set:"
+
  echo "Environment:"
echo ""
+
echo "LD_LIBRARY_PATH: ${LD_LIBRARY_PATH}"
+
echo ""
+
echo "ESMF_COMM: ${ESMF_COMM}"
+
echo "ESMP_BOPT: ${ESMF_BOPT}"
+
echo "MPI_ROOT: ${MPI_ROOT}"
+
 
  echo ""
 
  echo ""
 
  echo "CC: ${CC}"
 
  echo "CC: ${CC}"
echo "OMPI_CC: ${OMPI_CC}"
 
echo ""
 
 
  echo "CXX: ${CXX}"
 
  echo "CXX: ${CXX}"
echo "OMPI_CXX: ${OMPI_CXX}"
 
echo ""
 
 
  echo "FC: ${FC}"
 
  echo "FC: ${FC}"
  echo "F77: ${F77}"
+
  echo "ESMF_COMM: ${ESMF_COMM}"
echo "F90: ${F90}"
+
echo "OMPI_FC: ${OMPI_FC}"
+
echo "COMPILER: ${COMPILER}"
+
 
  echo "ESMF_COMPILER: ${ESMF_COMPILER}"
 
  echo "ESMF_COMPILER: ${ESMF_COMPILER}"
 +
echo "ESMF_DIR: ${ESMF_DIR}"
 +
echo "ESMF_INSTALL_PREFIX: ${ESMF_INSTALL_PREFIX}"
 +
echo "ESMF_ROOT: ${ESMF_ROOT}"
 +
echo "MPI_HOME: ${MPI_HOME}"
 +
echo "NetCDF C: $(nc-config --prefix)"
 +
echo "NetCDF Fortran: $(nf-config --prefix)"
 
  echo ""
 
  echo ""
echo "GC_BIN: ${GC_BIN}"
 
echo "GC_INCLUDE: ${GC_INCLUDE}"
 
echo "GC_LIB: ${GC_LIB}"
 
echo ""
 
#echo "GC_F_BIN: ${GC_F_BIN}"
 
#echo "GC_F_INCLUDE: ${GC_F_INCLUDE}"
 
#echo "GC_F_LIB: ${GC_F_LIB}"
 
#echo ""
 
 
  echo "Done sourcing ${BASH_SOURCE[0]}"
 
  echo "Done sourcing ${BASH_SOURCE[0]}"
  
Line 167: Line 112:
 
  virtual memory          (kbytes, -v) unlimited
 
  virtual memory          (kbytes, -v) unlimited
 
  file locks                      (-x) unlimited
 
  file locks                      (-x) unlimited
 
All example environment files for GCHP explicitly set several of these to unlimited, as shown above in the example. If you run into a memory issue be sure to check your limits against the list above to see if anything may be limiting your run.
 
  
 
--------------------------------------
 
--------------------------------------
  
 
'''''[[Obtaining_a_GCHP_Run_Directory|Previous]] | [[Compiling_GCHP|Next]] | [[Getting Started with GCHP]] | [[GCHP Main Page]]'''''
 
'''''[[Obtaining_a_GCHP_Run_Directory|Previous]] | [[Compiling_GCHP|Next]] | [[Getting Started with GCHP]] | [[GCHP Main Page]]'''''

Revision as of 23:28, 2 November 2020

Previous | Next | Getting Started with GCHP | GCHP Main Page

  1. Hardware and Software Requirements
  2. Downloading Source Code and Data Directories
  3. Obtaining a Run Directory
  4. Setting Up the GCHP Environment
  5. Compiling
  6. Running GCHP: Basics
  7. Running GCHP: Configuration
  8. Output Data
  9. Developing GCHP
  10. Run Configuration Files


Create an Environment File

You must load all necessary libraries and export certain environment variables before compiling GCHP. We recommend simplifying the environment setup process by customizing one or more GCHP-specific environment files that work on your system and storing them in a central location for repeated use.

An example of an environment file is below. This particular file was also used to build ESMF and therefore includes ESMF-only environment variables as well as those required by GCHP. It also sets certain system limits to avoid potential problems with memory.

if  $- = *i*  ; then
  echo "Loading modules, please wait ..."
fi

#==============================================================================
# Load modules
#==============================================================================

module purge
module load git/2.17.0-fasrc01
module load intel/18.0.5-fasrc01
module load openmpi/4.0.1-fasrc01
module load netcdf-fortran/4.5.2-fasrc01
module load cmake/3.16.1-fasrc01

#==============================================================================
# Environment variables
#==============================================================================

# Make all files world-readable by default
umask 022

# Specify compilers
export CC=gcc
export CXX=g++
export FC=ifort

# MPI
export MPI_ROOT=${MPI_HOME}

# For ESMF
export ESMF_COMPILER=intel
export ESMF_COMM=openmpi
export ESMF_DIR=/n/jacob_lab/Lab/ESMF/ESMF_8_0_1
export ESMF_INSTALL_PREFIX=${ESMF_DIR}/INSTALL_ifort18_openmpi4

# For GCHP
export ESMF_ROOT=${ESMF_INSTALL_PREFIX}

#==============================================================================
# Set limits
#==============================================================================

ulimit -c 0                      # coredumpsize
ulimit -l unlimited              # memorylocked
ulimit -u 50000                  # maxproc
ulimit -v unlimited              # vmemoryuse
ulimit -s unlimited              # stacksize

#==============================================================================
# Print information
#==============================================================================

module list
echo ""
echo "Environment:"
echo ""
echo "CC: ${CC}"
echo "CXX: ${CXX}"
echo "FC: ${FC}"
echo "ESMF_COMM: ${ESMF_COMM}"
echo "ESMF_COMPILER: ${ESMF_COMPILER}"
echo "ESMF_DIR: ${ESMF_DIR}"
echo "ESMF_INSTALL_PREFIX: ${ESMF_INSTALL_PREFIX}"
echo "ESMF_ROOT: ${ESMF_ROOT}"
echo "MPI_HOME: ${MPI_HOME}"
echo "NetCDF C: $(nc-config --prefix)"
echo "NetCDF Fortran: $(nf-config --prefix)"
echo ""
echo "Done sourcing ${BASH_SOURCE[0]}"

System memory limits and stack size should be set to unlimited to avoid memory problems. Such problems would manifest as sudden termination upon file read or a segmentation fault during advection. You can find out what you system limits are by typing the following at the command prompt:

ulimit -a

You will see something like this:

core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 1030083
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 100000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) unlimited
cpu time               (seconds, -t) unlimited
max user processes              (-u) 4096
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Previous | Next | Getting Started with GCHP | GCHP Main Page