Difference between revisions of "Setting Up the GCHP Environment"

From Geos-chem
Jump to: navigation, search
(→‎Expanding MPI Options (Advanced))
m
 
(17 intermediate revisions by one other user not shown)
Line 1: Line 1:
 +
----
 +
<span style="color:crimson;font-size:120%">'''The GCHP documentation has moved to https://gchp.readthedocs.io/.''' The GCHP documentation on http://wiki.seas.harvard.edu/ will stay online for several months, but it is outdated and no longer active!</span>
 +
----
 +
 
__FORCETOC__
 
__FORCETOC__
−
'''''[[Obtaining_a_GCHP_Run_Directory|Previous]] | [[Compiling_GCHP|Next]] |  [[Getting Started With GCHP]] | [[GCHP Main Page]]'''''
+
'''''[[GCHP_Hardware_and_Software_Requirements|Previous]] | [[Downloading_GCHP|Next]] |  [[Getting Started with GCHP]] | [[GCHP Main Page]]'''''
 
#[[GCHP_Hardware_and_Software_Requirements|Hardware and Software Requirements]]
 
#[[GCHP_Hardware_and_Software_Requirements|Hardware and Software Requirements]]
 +
#<span style="color:blue">'''Setting Up the GCHP Environment'''</span>
 
#[[Downloading_GCHP|Downloading Source Code and Data Directories]]
 
#[[Downloading_GCHP|Downloading Source Code and Data Directories]]
 +
#[[Compiling_GCHP|Compiling]]
 
#[[Obtaining_a_GCHP_Run_Directory|Obtaining a Run Directory]]
 
#[[Obtaining_a_GCHP_Run_Directory|Obtaining a Run Directory]]
−
#<span style="color:blue">'''Setting Up the GCHP Environment'''</span>
 
−
#[[Compiling_GCHP|Compiling]]
 
 
#[[Running_GCHP:_Basics|Running GCHP: Basics]]
 
#[[Running_GCHP:_Basics|Running GCHP: Basics]]
 
#[[Running_GCHP:_Configuration|Running GCHP: Configuration]]
 
#[[Running_GCHP:_Configuration|Running GCHP: Configuration]]
Line 15: Line 19:
 
== Create an Environment File ==
 
== Create an Environment File ==
  
−
You must load all necessary libraries and export certain environment variables before compiling GCHP. The GCHP environment is different from GEOS-Chem Classic and is often considered the largest obstacle to getting GCHP up and running for the first time. We have tried to make setting libraries and variables as automatic as possible to minimize problems. However, libraries will always be specific to your local compute cluster which presents challenges for compatibility. We recommend simplifying the environment setup process by customizing a GCHP-specific environment file that works on your system and saving it for future work.
+
You must load all necessary libraries and export certain environment variables before compiling and running GCHP. We recommend simplifying the environment setup process by customizing one or more GCHP-specific environment files that work on your system and storing them in a central location for repeated use.
  
−
Sample environment files are included in the run directory, several for the [https://www.rc.fas.harvard.edu/resources/odyssey-architecture/ Harvard University Odyssey cluster] and one for a more generic Linux system. These are located in the <code>environmentFileSamples</code> subdirectory. You can use these to develop one compatible with your system. Each sample environment file is customized for a specific combination of Fortran compiler, MPI implementation, netCDF libraries, and compute cluster. For clarity we recommend using the naming format <tt>gchp.compiler_mpi_cluster.env</tt>. For example, <code>gchp.ifort17_openmpi3_computecanada.env</code>. Open several of the sample environment files and getting familiar with the environment variables set.
+
An example of an environment file is below. This particular file was also used to build ESMF and therefore includes ESMF-only environment variables as well as those required by GCHP. It also sets certain system limits to avoid potential problems with memory.
  
−
An example of environment variables needed for GCHP are as follows:
+
if [[ $- = *i* ]] ; then
−
 
+
  echo "Loading modules, please wait ..."
−
[[File:gchp Env.png]]
+
fi
 +
 +
#==============================================================================
 +
# Load modules
 +
#==============================================================================
 +
 +
module purge
 +
module load git/2.17.0-fasrc01
 +
module load intel/18.0.5-fasrc01
 +
module load openmpi/4.0.1-fasrc01
 +
module load netcdf-fortran/4.5.2-fasrc01
 +
module load cmake/3.16.1-fasrc01
 +
 +
#==============================================================================
 +
# Environment variables
 +
#==============================================================================
 +
 +
# Make all files world-readable by default
 +
umask 022
 +
 +
# Specify compilers
 +
export CC=gcc
 +
export CXX=g++
 +
export FC=ifort
 +
 +
# MPI
 +
export MPI_ROOT=${MPI_HOME}
 +
 +
# For ESMF
 +
export ESMF_COMPILER=intel
 +
export ESMF_COMM=openmpi
 +
export ESMF_DIR=/n/jacob_lab/Lab/ESMF/ESMF_8_0_1
 +
export ESMF_INSTALL_PREFIX=${ESMF_DIR}/INSTALL_ifort18_openmpi4
 +
 +
# For GCHP
 +
export ESMF_ROOT=${ESMF_INSTALL_PREFIX}
 +
 +
#==============================================================================
 +
# Set limits
 +
#==============================================================================
 +
 +
ulimit -c 0                      # coredumpsize
 +
ulimit -l unlimited              # memorylocked
 +
ulimit -u 50000                  # maxproc
 +
ulimit -v unlimited              # vmemoryuse
 +
ulimit -s unlimited              # stacksize
 +
 +
#==============================================================================
 +
# Print information
 +
#==============================================================================
 +
 +
module list
 +
echo ""
 +
echo "Environment:"
 +
echo ""
 +
echo "CC: ${CC}"
 +
echo "CXX: ${CXX}"
 +
echo "FC: ${FC}"
 +
echo "ESMF_COMM: ${ESMF_COMM}"
 +
echo "ESMF_COMPILER: ${ESMF_COMPILER}"
 +
echo "ESMF_DIR: ${ESMF_DIR}"
 +
echo "ESMF_INSTALL_PREFIX: ${ESMF_INSTALL_PREFIX}"
 +
echo "ESMF_ROOT: ${ESMF_ROOT}"
 +
echo "MPI_HOME: ${MPI_HOME}"
 +
echo "NetCDF C: $(nc-config --prefix)"
 +
echo "NetCDF Fortran: $(nf-config --prefix)"
 +
echo ""
 +
echo "Done sourcing ${BASH_SOURCE[0]}"
  
 
System memory limits and stack size should be set to unlimited to avoid memory problems. Such problems would manifest as sudden termination upon file read or a segmentation fault during advection. You can find out what you system limits are by typing the following at the command prompt:
 
System memory limits and stack size should be set to unlimited to avoid memory problems. Such problems would manifest as sudden termination upon file read or a segmentation fault during advection. You can find out what you system limits are by typing the following at the command prompt:
Line 45: Line 116:
 
  virtual memory          (kbytes, -v) unlimited
 
  virtual memory          (kbytes, -v) unlimited
 
  file locks                      (-x) unlimited
 
  file locks                      (-x) unlimited
−
 
−
All example environment files for GCHP explicitly set several of these to unlimited. If you run into a memory issue be sure to check your limits against the list above to see if anything may be limiting your run. Our current recommendations are as follows:
 
−
 
−
ulimit -c unlimited              # coredumpsize
 
−
ulimit -l unlimited              # memorylocked
 
−
ulimit -u 50000                  # maxproc
 
−
ulimit -v unlimited              # vmemoryuse
 
−
ulimit -s unlimited              # stacksize
 
−
 
−
== Expanding MPI Options (Advanced) ==
 
−
 
−
GCHP is currently not tested with MPI implementations other than OpenMPI3. However, we encourage users to experiment with other MPI implementations. To do this, follow the steps below. You may need to make tweaks and it is possible it may fail. Please report what you are trying and your results by opening a [https://github.com/geoschem/gchp/issues GitHub GCHP issue] whether you fail or succeed. Before you begin, check the [[GCHP_Hardware_and_Software_Requirements#MPI|MPI section of this guide]] to see if there is any news about known issues.
 
−
 
−
#Specify environment variable <tt>ESMF_COMM</tt> to match the MPI implementation. Options are currently in place for MVAPICH2 (ESMF_COMM=mvapich2), OpenMPI (ESMF_COMM=openmpi), and a generic MPI implementation. (ESMF_COMM=mpi). The generic option is sufficient when, for example, running with the SGI MPI implementation on NASA's Pleiades servers. If you are using a new MPI implementation not covered by one of these options, we recommend running first with ESMF_COMM=mpi.
 
−
#Check that ${MPI_HOME} exists and contains the path to your MPI library. It should be set automatically when you load the library.
 
−
#Ensure that you have valid <tt>mpif90</tt> and <tt>mpifort</tt> executables. These almost always perform the same role but both names are invoked in the build sequence. If you have one but not the other, we strongly recommend that you make a symbolic link to the working binary with the name of the missing binary in a dedicated folder, and then add that folder to your path at the command line and in your <tt>.bashrc</tt>. For example, if you have a <tt>mpifort</tt> binary but not an <tt>mpif90</tt> binary, run the following commands:
 
−
mkdir $HOME/mpi_extra
 
−
cd $HOME/mpi_extra
 
−
ln -s $( which mpifort ) mpif90
 
−
export PATH=${PATH}:${HOME}/mpi_extra
 
−
 
−
You should now try to compile GCHP. If the generic option does not work then you will need to implement a new option. This involves updating GCHP source code. An example of how to do this for Intel MPI is as follows:
 
−
#Decide a new name, such as <tt>ESMF_COMM=intel</tt> for the Intel MPI implementation.
 
−
#Determine the relevant include path and linker commands for your MPI implementation. In this example for Intel MPI they are <tt>$(MPI_ROOT)/include</tt> and <tt>-L$(MPI_ROOT)/lib -lmpi -lmpi++</tt> respectively.
 
−
#Update source code files <tt>CodeDir/GCHP/GIGC.mk</tt> and <tt>CodeDir/GCHP/Shared/Config/ESMA_base.mk</tt>. In both files, search for environment variable <tt>ESMF_COMM</tt> in the file. You should find a small set of occurrences in a single "if..else.." block. Add a new clause below the one for mvapich2 as follows.
 
−
 
−
In <tt>GIGC.mk</tt>:
 
−
else ifeq ($(ESMF_COMM),intel)
 
−
    # %%%%% Intel MPI %%%%%
 
−
    MPI_LIB    := -L$(MPI_ROOT)/lib -lmpi -lmpi++
 
−
 
−
In <tt>ESMA_base.mk</tt>:
 
−
else ifeq ($(ESMF_COMM),intel)
 
−
    INC_MPI := $(MPI_ROOT)/include
 
−
    LIB_MPI := -L$(MPI_ROOT)/lib -lmpi -lmpi++
 
  
 
--------------------------------------
 
--------------------------------------
  
−
'''''[[Obtaining_a_GCHP_Run_Directory|Previous]] | [[Compiling_GCHP|Next]] | [[Getting Started With GCHP]] | [[GCHP Main Page]]'''''
+
'''''[[GCHP_Hardware_and_Software_Requirements|Previous]] | [[Downloading_GCHP|Next]] | [[Getting Started with GCHP]] | [[GCHP Main Page]]'''''

Latest revision as of 15:40, 8 December 2020

The GCHP documentation has moved to https://gchp.readthedocs.io/. The GCHP documentation on http://wiki.seas.harvard.edu/ will stay online for several months, but it is outdated and no longer active!


Previous | Next | Getting Started with GCHP | GCHP Main Page

  1. Hardware and Software Requirements
  2. Setting Up the GCHP Environment
  3. Downloading Source Code and Data Directories
  4. Compiling
  5. Obtaining a Run Directory
  6. Running GCHP: Basics
  7. Running GCHP: Configuration
  8. Output Data
  9. Developing GCHP
  10. Run Configuration Files


Create an Environment File

You must load all necessary libraries and export certain environment variables before compiling and running GCHP. We recommend simplifying the environment setup process by customizing one or more GCHP-specific environment files that work on your system and storing them in a central location for repeated use.

An example of an environment file is below. This particular file was also used to build ESMF and therefore includes ESMF-only environment variables as well as those required by GCHP. It also sets certain system limits to avoid potential problems with memory. 

if  $- = *i*  ; then
  echo "Loading modules, please wait ..."
fi

#==============================================================================
# Load modules
#==============================================================================

module purge
module load git/2.17.0-fasrc01
module load intel/18.0.5-fasrc01
module load openmpi/4.0.1-fasrc01
module load netcdf-fortran/4.5.2-fasrc01
module load cmake/3.16.1-fasrc01

#==============================================================================
# Environment variables
#==============================================================================

# Make all files world-readable by default
umask 022

# Specify compilers
export CC=gcc
export CXX=g++
export FC=ifort

# MPI
export MPI_ROOT=${MPI_HOME}

# For ESMF
export ESMF_COMPILER=intel
export ESMF_COMM=openmpi
export ESMF_DIR=/n/jacob_lab/Lab/ESMF/ESMF_8_0_1
export ESMF_INSTALL_PREFIX=${ESMF_DIR}/INSTALL_ifort18_openmpi4

# For GCHP
export ESMF_ROOT=${ESMF_INSTALL_PREFIX}

#==============================================================================
# Set limits
#==============================================================================

ulimit -c 0                      # coredumpsize
ulimit -l unlimited              # memorylocked
ulimit -u 50000                  # maxproc
ulimit -v unlimited              # vmemoryuse
ulimit -s unlimited              # stacksize

#==============================================================================
# Print information
#==============================================================================

module list
echo ""
echo "Environment:"
echo ""
echo "CC: ${CC}"
echo "CXX: ${CXX}"
echo "FC: ${FC}"
echo "ESMF_COMM: ${ESMF_COMM}"
echo "ESMF_COMPILER: ${ESMF_COMPILER}"
echo "ESMF_DIR: ${ESMF_DIR}"
echo "ESMF_INSTALL_PREFIX: ${ESMF_INSTALL_PREFIX}"
echo "ESMF_ROOT: ${ESMF_ROOT}"
echo "MPI_HOME: ${MPI_HOME}"
echo "NetCDF C: $(nc-config --prefix)"
echo "NetCDF Fortran: $(nf-config --prefix)"
echo ""
echo "Done sourcing ${BASH_SOURCE[0]}"

System memory limits and stack size should be set to unlimited to avoid memory problems. Such problems would manifest as sudden termination upon file read or a segmentation fault during advection. You can find out what you system limits are by typing the following at the command prompt: 

ulimit -a

You will see something like this: 

core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 1030083
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 100000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) unlimited
cpu time               (seconds, -t) unlimited
max user processes              (-u) 4096
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Previous | Next | Getting Started with GCHP | GCHP Main Page

Retrieved from "https://wiki.seas.harvard.edu/geos-chem/index.php?title=Setting_Up_the_GCHP_Environment&oldid=48669"