Difference between revisions of "Setting Up the GCHP Environment"

From Geos-chem
Jump to: navigation, search
m
 
(26 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 +
----
 +
<span style="color:crimson;font-size:120%">'''The GCHP documentation has moved to https://gchp.readthedocs.io/.''' The GCHP documentation on http://wiki.seas.harvard.edu/ will stay online for several months, but it is outdated and no longer active!</span>
 +
----
 +
 
__FORCETOC__
 
__FORCETOC__
−
'''''[[Obtaining_a_GCHP_Run_Directory|Previous]] | [[Compiling_GCHP|Next]] |  [[Getting_Started_With_GCHP|User Manual Home]] | [[GEOS-Chem_HP|GCHP Home]]'''''
+
'''''[[GCHP_Hardware_and_Software_Requirements|Previous]] | [[Downloading_GCHP|Next]] |  [[Getting Started with GCHP]] | [[GCHP Main Page]]'''''
 
#[[GCHP_Hardware_and_Software_Requirements|Hardware and Software Requirements]]
 
#[[GCHP_Hardware_and_Software_Requirements|Hardware and Software Requirements]]
 +
#<span style="color:blue">'''Setting Up the GCHP Environment'''</span>
 
#[[Downloading_GCHP|Downloading Source Code and Data Directories]]
 
#[[Downloading_GCHP|Downloading Source Code and Data Directories]]
 +
#[[Compiling_GCHP|Compiling]]
 
#[[Obtaining_a_GCHP_Run_Directory|Obtaining a Run Directory]]
 
#[[Obtaining_a_GCHP_Run_Directory|Obtaining a Run Directory]]
−
#<span style="color:blue">'''Setting Up the GCHP Environment'''</span>
+
#[[Running_GCHP:_Basics|Running GCHP: Basics]]
−
#[[Compiling_GCHP|Compiling]]
+
#[[Running_GCHP:_Configuration|Running GCHP: Configuration]]
−
#[[GCHP_Basic_Example_Run|Running GCHP: Part 1]]
+
#[[GCHP_Output_Data|Output Data]]
−
#[[Running_GCHP|Running GCHP: Part 2]]
+
−
#[[GEOS-Chem_HP_Output_Data|Output Data]]
+
 
#[[Developing_GCHP|Developing GCHP]]
 
#[[Developing_GCHP|Developing GCHP]]
 
#[[GCHP_Run_Configuration_Files|Run Configuration Files]]
 
#[[GCHP_Run_Configuration_Files|Run Configuration Files]]
Line 15: Line 19:
 
== Create an Environment File ==
 
== Create an Environment File ==
  
−
You must load all necessary libraries and export certain environment variables before compiling GCHP. The GCHP environment is different from GEOS-Chem Classic and is often considered the largest obstacle to getting GCHP up and running for the first time. We have tried to make setting libraries and variables as automatic as possible to minimize problems. However, libraries will always be specific to your local compute cluster which presents challenges for compatibility. We recommend simplifying the environment setup process by customizing a GCHP-specific environment file that works on your system and saving it for future work.
+
You must load all necessary libraries and export certain environment variables before compiling and running GCHP. We recommend simplifying the environment setup process by customizing one or more GCHP-specific environment files that work on your system and storing them in a central location for repeated use.
−
 
+
−
Sample environment files are included in the run directory, several for the [https://www.rc.fas.harvard.edu/resources/odyssey-architecture/ Harvard University Odyssey cluster] and one for a more generic Linux system. These are located in the <tt>bashrcSamples</tt> subdirectory for versions prior to 12.1.0, and in the <tt>environmentFileSamples</tt> subdirectory starting in 12.1.0. You can use these to develop one compatible with your system. Each sample environment file is customized for a specific combination of Fortran compiler, MPI implementation, netCDF libraries, and compute cluster. For clarity we recommend using the naming format <tt>gchp.compiler_mpi_cluster.env</tt>. For example, <tt>gchp.ifort17_openMPI3_computecanada.env</tt>.
+
−
 
+
−
We recommend opening several of the sample environment files and getting familiar with the environment variables set. The files include printing the variables to screen when sourced. This is particularly useful for logging purposes if you automatically source the environment file within a run script as all of the run script samples included in the run directory do. The environment variables will be printed to your system log file and will reflect the settings used during your run, potentially useful for debugging and archiving.
+
−
 
+
−
An example of environment variables needed for GCHP are as follows:
+
−
 
+
−
[[File:gchp Env.png]]
+
−
 
+
−
System memory limits should also be set to unlimited if possible.
+
−
 
+
−
[[File:gchp Memory.png]]
+
−
 
+
−
Also included in the sample environment files are a few aliases for commands that are commonly used when developing, compiling, and running GCHP. It may be useful to look at them to see if you would like to adopt them or add your own.
+
−
 
+
−
[[File:GCHP_aliases.png]]
+
  
−
== Expanding MPI Options ==
+
An example of an environment file is below. This particular file was also used to build ESMF and therefore includes ESMF-only environment variables as well as those required by GCHP. It also sets certain system limits to avoid potential problems with memory.
  
−
GCHP is compatible with MPICH, OpenMPI, and MVAPICH2 MPI implementations. However, you may want to use a different implementation, possibly out of necessity. To do this, follow the steps below.  
+
if [[ $- = *i* ]] ; then
 +
  echo "Loading modules, please wait ..."
 +
fi
 +
 +
#==============================================================================
 +
# Load modules
 +
#==============================================================================
 +
 +
module purge
 +
module load git/2.17.0-fasrc01
 +
module load intel/18.0.5-fasrc01
 +
module load openmpi/4.0.1-fasrc01
 +
module load netcdf-fortran/4.5.2-fasrc01
 +
module load cmake/3.16.1-fasrc01
 +
 +
#==============================================================================
 +
# Environment variables
 +
#==============================================================================
 +
 +
# Make all files world-readable by default
 +
umask 022
 +
 +
# Specify compilers
 +
export CC=gcc
 +
export CXX=g++
 +
export FC=ifort
 +
 +
# MPI
 +
export MPI_ROOT=${MPI_HOME}
 +
 +
# For ESMF
 +
export ESMF_COMPILER=intel
 +
export ESMF_COMM=openmpi
 +
export ESMF_DIR=/n/jacob_lab/Lab/ESMF/ESMF_8_0_1
 +
export ESMF_INSTALL_PREFIX=${ESMF_DIR}/INSTALL_ifort18_openmpi4
 +
 +
# For GCHP
 +
export ESMF_ROOT=${ESMF_INSTALL_PREFIX}
 +
 +
#==============================================================================
 +
# Set limits
 +
#==============================================================================
 +
 +
ulimit -c 0                      # coredumpsize
 +
ulimit -l unlimited              # memorylocked
 +
ulimit -u 50000                  # maxproc
 +
ulimit -v unlimited              # vmemoryuse
 +
ulimit -s unlimited              # stacksize
 +
 +
#==============================================================================
 +
# Print information
 +
#==============================================================================
 +
 +
module list
 +
echo ""
 +
echo "Environment:"
 +
echo ""
 +
echo "CC: ${CC}"
 +
echo "CXX: ${CXX}"
 +
echo "FC: ${FC}"
 +
echo "ESMF_COMM: ${ESMF_COMM}"
 +
echo "ESMF_COMPILER: ${ESMF_COMPILER}"
 +
echo "ESMF_DIR: ${ESMF_DIR}"
 +
echo "ESMF_INSTALL_PREFIX: ${ESMF_INSTALL_PREFIX}"
 +
echo "ESMF_ROOT: ${ESMF_ROOT}"
 +
echo "MPI_HOME: ${MPI_HOME}"
 +
echo "NetCDF C: $(nc-config --prefix)"
 +
echo "NetCDF Fortran: $(nf-config --prefix)"
 +
echo ""
 +
echo "Done sourcing ${BASH_SOURCE[0]}"
  
−
#Specify environment variable <tt>ESMF_COMM</tt> to match the MPI implementation. Options are currently in place for MVAPICH2 (ESMF_COMM=mvapich2), OpenMPI (ESMF_COMM=openmpi), and a generic MPI implementation. (ESMF_COMM=mpi). The generic option is sufficient when, for example, running with the SGI MPI implementation on NASA's Pleiades servers. If you are using a new MPI implementation not covered by one of these options, we recommend running first with ESMF_COMM=mpi.
+
System memory limits and stack size should be set to unlimited to avoid memory problems. Such problems would manifest as sudden termination upon file read or a segmentation fault during advection. You can find out what you system limits are by typing the following at the command prompt:
−
#Specify the environment variable MPI_ROOT to point to the MPI root directory, such that $MPI_ROOT/bin/mpirun points to the correct MPI run binary.
+
−
#Ensure that you have valid <tt>mpif90</tt> and <tt>mpifort</tt> executables. These almost always perform the same role but both names are invoked in the build sequence. If you have one but not the other, we strongly recommend that you make a symbolic link to the working binary with the name of the missing binary in a dedicated folder, and then add that folder to your path at the command line and in your <tt>.bashrc</tt>. For example, if you have a <tt>mpifort</tt> binary but not an <tt>mpif90</tt> binary, run the following commands:
+
−
mkdir $HOME/mpi_extra
+
−
cd $HOME/mpi_extra
+
−
ln -s $( which mpifort ) mpif90
+
−
export PATH=${PATH}:${HOME}/mpi_extra
+
  
−
You should now try to compile GCHP. If the generic option does not work then you will need to implement a new option. This involves updating GCHP source code. An example of how to do this for Intel MPI is as follows:
+
ulimit -a
−
#Decide a new name, such as <tt>ESMF_COMM=intel</tt> for the Intel MPI implementation.
+
−
#Determine the relevant include path and linker commands for your MPI implementation. In this example for Intel MPI they are <tt>$(MPI_ROOT)/include</tt> and <tt>-L$(MPI_ROOT)/lib -lmpi -lmpi++</tt> respectively.
+
−
#Update source code files <tt>CodeDir/GCHP/GIGC.mk</tt> and <tt>CodeDir/GCHP/Shared/Config/ESMA_base.mk</tt>. In both files, search for environment variable <tt>ESMF_COMM</tt> in the file. You should find a small set of occurrences in a single "if..else.." block. Add a new clause below the one for mvapich2 as follows.
+
  
−
In <tt>GIGC.mk</tt>:
+
You will see something like this:
−
else ifeq ($(ESMF_COMM),intel)
+
−
    # %%%%% Intel MPI %%%%%
+
−
    MPI_LIB    := -L$(MPI_ROOT)/lib -lmpi -lmpi++
+
  
−
In <tt>ESMA_base.mk</tt>:
+
core file size          (blocks, -c) unlimited
−
  else ifeq ($(ESMF_COMM),intel)
+
  data seg size          (kbytes, -d) unlimited
−
    INC_MPI := $(MPI_ROOT)/include
+
scheduling priority            (-e) 0
−
     LIB_MPI := -L$(MPI_ROOT)/lib -lmpi -lmpi++
+
file size              (blocks, -f) unlimited
 +
pending signals                (-i) 1030083
 +
max locked memory      (kbytes, -l) unlimited
 +
max memory size        (kbytes, -m) unlimited
 +
open files                      (-n) 100000
 +
pipe size            (512 bytes, -p) 8
 +
POSIX message queues     (bytes, -q) 819200
 +
real-time priority              (-r) 0
 +
stack size              (kbytes, -s) unlimited
 +
cpu time              (seconds, -t) unlimited
 +
max user processes              (-u) 4096
 +
virtual memory          (kbytes, -v) unlimited
 +
file locks                      (-x) unlimited
  
 
--------------------------------------
 
--------------------------------------
  
−
'''''[[Obtaining_a_GCHP_Run_Directory|Previous]] | [[Compiling_GCHP|Next]] | [[Getting_Started_With_GCHP|User Manual Home]] | [[GEOS-Chem_HP|GCHP Home]]'''''
+
'''''[[GCHP_Hardware_and_Software_Requirements|Previous]] | [[Downloading_GCHP|Next]] | [[Getting Started with GCHP]] | [[GCHP Main Page]]'''''

Latest revision as of 15:40, 8 December 2020

The GCHP documentation has moved to https://gchp.readthedocs.io/. The GCHP documentation on http://wiki.seas.harvard.edu/ will stay online for several months, but it is outdated and no longer active!


Previous | Next | Getting Started with GCHP | GCHP Main Page

  1. Hardware and Software Requirements
  2. Setting Up the GCHP Environment
  3. Downloading Source Code and Data Directories
  4. Compiling
  5. Obtaining a Run Directory
  6. Running GCHP: Basics
  7. Running GCHP: Configuration
  8. Output Data
  9. Developing GCHP
  10. Run Configuration Files


Create an Environment File

You must load all necessary libraries and export certain environment variables before compiling and running GCHP. We recommend simplifying the environment setup process by customizing one or more GCHP-specific environment files that work on your system and storing them in a central location for repeated use.

An example of an environment file is below. This particular file was also used to build ESMF and therefore includes ESMF-only environment variables as well as those required by GCHP. It also sets certain system limits to avoid potential problems with memory. 

if  $- = *i*  ; then
  echo "Loading modules, please wait ..."
fi

#==============================================================================
# Load modules
#==============================================================================

module purge
module load git/2.17.0-fasrc01
module load intel/18.0.5-fasrc01
module load openmpi/4.0.1-fasrc01
module load netcdf-fortran/4.5.2-fasrc01
module load cmake/3.16.1-fasrc01

#==============================================================================
# Environment variables
#==============================================================================

# Make all files world-readable by default
umask 022

# Specify compilers
export CC=gcc
export CXX=g++
export FC=ifort

# MPI
export MPI_ROOT=${MPI_HOME}

# For ESMF
export ESMF_COMPILER=intel
export ESMF_COMM=openmpi
export ESMF_DIR=/n/jacob_lab/Lab/ESMF/ESMF_8_0_1
export ESMF_INSTALL_PREFIX=${ESMF_DIR}/INSTALL_ifort18_openmpi4

# For GCHP
export ESMF_ROOT=${ESMF_INSTALL_PREFIX}

#==============================================================================
# Set limits
#==============================================================================

ulimit -c 0                      # coredumpsize
ulimit -l unlimited              # memorylocked
ulimit -u 50000                  # maxproc
ulimit -v unlimited              # vmemoryuse
ulimit -s unlimited              # stacksize

#==============================================================================
# Print information
#==============================================================================

module list
echo ""
echo "Environment:"
echo ""
echo "CC: ${CC}"
echo "CXX: ${CXX}"
echo "FC: ${FC}"
echo "ESMF_COMM: ${ESMF_COMM}"
echo "ESMF_COMPILER: ${ESMF_COMPILER}"
echo "ESMF_DIR: ${ESMF_DIR}"
echo "ESMF_INSTALL_PREFIX: ${ESMF_INSTALL_PREFIX}"
echo "ESMF_ROOT: ${ESMF_ROOT}"
echo "MPI_HOME: ${MPI_HOME}"
echo "NetCDF C: $(nc-config --prefix)"
echo "NetCDF Fortran: $(nf-config --prefix)"
echo ""
echo "Done sourcing ${BASH_SOURCE[0]}"

System memory limits and stack size should be set to unlimited to avoid memory problems. Such problems would manifest as sudden termination upon file read or a segmentation fault during advection. You can find out what you system limits are by typing the following at the command prompt: 

ulimit -a

You will see something like this: 

core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 1030083
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 100000
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) unlimited
cpu time               (seconds, -t) unlimited
max user processes              (-u) 4096
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Previous | Next | Getting Started with GCHP | GCHP Main Page

Retrieved from "https://wiki.seas.harvard.edu/geos-chem/index.php?title=Setting_Up_the_GCHP_Environment&oldid=48669"