Difference between revisions of "GCHP Hardware and Software Requirements"

Latest revision as of 15:40, 8 December 2020

The GCHP documentation has moved to https://gchp.readthedocs.io/. The GCHP documentation on http://wiki.seas.harvard.edu/ will stay online for several months, but it is outdated and no longer active!

Next | Getting Started with GCHP | GCHP Main Page

1. Hardware and Software Requirements
2. Setting Up the GCHP Environment
3. Downloading Source Code and Data Directories
4. Compiling
5. Obtaining a Run Directory
6. Running GCHP: Basics
7. Running GCHP: Configuration
8. Output Data
9. Developing GCHP
10. Run Configuration Files

Hardware Requirements

The following is a short list of hardware requirements for GCHP and related information:

1. GCHP requires a minimum of 6 cores to run, and beyond this the number of cores must be a multiple of 6.
2. Most GCHP and GEOS-Chem Classic users use the Intel Fortran Compiler (ifort). Starting with version 12.5.0, GCHP also has compatibility with open-source GNU compilers. Regardless of whether you use an Intel compiler or not, we highly recommend that you use Intel CPUs, preferably Xeon. It is a known issue that ifort does not optimize well on AMD chips (this was actually intended). Don't worry too much about CPU speed if you can use Xeon CPUs.
3. Most clusters are built upon nodes with 16 to 32 CPUs each. One node of 32 CPUs will provide sufficient resources for standard GCHP c48 runs. It will have the added benefit of not needing to use the network interconnect for MPI. In general, GCHP will have better performance with more cores per node due to how input handling is parallelized.
4. GCHP can run with ~7 GB per CPU if using C24, the cubed sphere resolution equivalent to 4° x 5°. Higher resolutions will require more memory per core depending on how many cores you use for your run. We have found that the biggest bottleneck for running high resolution simulations is the amount of memory available per node since it limits the memory available per core. The best solution when running into memory per core limitations is to request more nodes, reserve all memory per node by requesting all cores, and use fewer cores per node than you have requested for your actual GCHP run.
5. InfiniBand is recommended if you can afford it. If not, a 10 Gigabit Ethernet is a good alternative when using high-end interconnects like Cisco or high-end Hewlett-Packard (HP). A 1000 Mbps Gigabit Ethernet should also work if you get an optimized router like Cisco or HP, but it will be less speed-dependent and have more packet latency. This is minimized in the high-end network interconnects.
6. If using an interconnect, it would be very helpful if the system had two: one interconnect for file transfer, log-in, etc, and the other interconnect for MPI-only communication. Using two interconnects in this way prevents problems such as file I/O interfering with MPI packet transfer. This is probably now standard on turn-key systems. Ask your local cluster administrator what is available for you to use.
7. Ensure all MPI communication nodes use the same interconnect hardware. There have been reports of problems running at high resolution with mixed hardware configurations.

Software Requirements

GCHP requires C and Fortran compilers, netCDF C and Fortran libraries, an implementation of a Message Passing Interface (MPI), git version control software, and C-preprocessor software. This software is standard on many primary HPC systems in use by academic and scientific institutions. If any of the required software is not available then you must acquire it to compile and run GCHP.

For systems with pre-installed packages, such as an institutional compute cluster, you must determine how each package was compiled to ensure that you will be able to build and run GCHP successfully. It is absolutely necessary to use MPI and netCDF libraries built with the same compiler you will use to build GEOS-Chem and ESMF. If your pre-installed packages were not built with the same compilers then you will need to build them yourself or have them built for you by your organization's technical support staff.

Operating System

We are currently only supporting GCHP on Linux distributions.

Git

You should acquire source code by cloning or forking the GEOS-Chem and GCHP repositories on GitHub using git. The GCHP Makefile assumes use of git version >1.8. Otherwise you will get an error that the -C option is not available. Check the default git version on your cluster by typing 'git --version' at the command line. If you have version 1.8 and prior then check with your system administrator on modules available with a more recent version of git, or download a newer version for free from the web.

If you are new to git then you should take time to get familiar with it. The UCAR CESM project has an excellent git guide, CIME Git Workflow, that is a great resource regardless of your skill level. The same basic workflow can be used for GEOS-Chem.

CMake

CMake 3.13 or higher is required for building GCHP starting in version 13.0.0. For more information about CMake, see the GEOS-Chem wiki page Compiling with CMake.

MPI

The Message Passing Interface (MPI) is the fundamental requirement for distributed memory computing. We currently support using GCHP with OpenMPI version 3.1 or higher. OpenMPI 3.0 will also work with GCHP, but versions prior to it will not.

GCHP 13.0.0 has been successfully compiled and executed in small multi-node runs with OpenMPI 4.0.3-4.0.5, MVAPICH2 2.3.4, and Intel MPI 2019 Update 7 and Update 8. More extensive version testing is forthcoming.

We have not actively tested GCHP 12 with Intel MPI but it has been achieved within the user community with some modification. See GCHP GitHub Issue #71 for details.

Running very old versions of GCHP has also been successful with SGI MPT, MPICH, and MVAPICH2, a derivation of MPICH for use with the Infiniband interconnect. All tests were configured with Intel Fortran + Intel C, Intel Fortran + GNU C, or GNU Fortran + GNU C. We have had reports of diagnostic write problems when using MPICH and MVAPICH2 on some systems. Switching to OpenMPI version 3.0 or 3.1 appears to resolve the problems. We therefore recommend trying OpenMPI 3 or higher before other implementations of MPI if possible. See the GCHP issues page on GitHub, and filter for tag "MPI", for more information.

You may test which compiler your version of MPI is using with:

$ mpif90 --version (for Fortran Compiler)
$ mpicc  --version (for C compiler)

Fortran Compiler

GCHP works with Intel and GNU fortran compilers. You may use Intel Fortran Compiler (ifort) 18.0.4 or above or GNU fortran 8 or above. Earlier versions are not fully compatible with the NASA/GMAO MAPL library.

ESMF

The Earth System Modeling Framework (ESMF) library provides a software infrastructure that allows different components of Earth System Models to communicate with each other using MPI parallelization. For more information about ESMF, please see: https://www.earthsystemcog.org/projects/esmf/.

Unlike previous versions, GCHP 13.0.0 requires download and build of ESMF outside of GCHP. ESMF is no longer be included in the GCHP repository. The primary benefits of using ESMF as an external library are:

1. Only one download of ESMF source code is needed
2. Only one ESMF build will be required for each combination of libraries you use
3. GCHP download time will be much faster due to a more lightweight repository
4. GCHP initial build time will be more than 50% reduced

To download ESMF, please see instructions on the ESMF webpage for "How to Git Clone ESMF" and follow instructions to "clone the entire repository."

Once downloaded, browse file ESMF/README for information about build requirements. ESMF requires that you define environment variable ESMF_DIR to be the path to your ESMF clone. To create multiple builds in the library, also set environment variable ESMF_INSTALL_PREFIX indicating your environment, such as INSTALL_ifort18_openmpi4. Most other environment variables that are required for GEOS-Chem are also required for ESMF. These variables are detailed in the next chapter, Setting Up the GCHP Environment.

You should load libraries you plan on running with GCHP prior to building ESMF. This is best done by saving your library load commands to an environment file that you can save and reuse. Once you are ready to build execute the following commands:

$ source path_to_your_env_file
$ cd $ESMF_DIR
$ make &> compile.log

Once compilation completes, check the end of file compile.log to see if compilation was successful. If yes, continue as follows:

$ make install &> install.log

Check the end of file install.log. A message that installation was complete should be there if ESMF installation was a success. There should now be a folder in the top-level ESMF directory corresponding to what you defined as environment variable ESMF_INSTALL_PREFIX. Archive your compile and install logs to that directory.

$ mv compile.log $ESMF_INSTALL_PREFIX
$ mv install.log $ESMF_INSTALL_PREFIX

Calling make builds ESMF and calling make install places the build into your install directory. In that folder the build files are placed within subdirectories such as bin and lib, among others. The install directory is not deleted when you clean ESMF source code with make distclean in the top-level ESMF directory. Therefore you can clean and rebuild ESMF with different combinations of libraries in advance of needing them to build and run GCHP. Just remember to clean the source code and source the environment file you intend to use prior to creating a new build. You also must specify a different ${ESMF_INSTALL_PREFIX} for each unique build so as not to overwrite others.

$ cd $ESMF_DIR
$ make distclean
$ source path_to_your_env_file_with_unique_ESMF_INSTALL_PREFIX
$ make &> compile.log
$ install $> install.log
$ mv compile.log $ESMF_INSTALL_PREFIX
$ mv install.log $ESMF_INSTALL_PREFIX

Please contact the GEOS-Chem Support Team if you run into problems and please include your compile.log and install.log (if you have one). Depending on the issue, if you run into problems downloading or building ESMF we may refer you to ESMF Support at https://www.earthsystemcog.org/projects/esmf/contactus/.

C/Fortran Pre-processor

To compile certain ESMF components in GCHP, a compatible C pre-processor (cpp) needs to be available. If you are having compilation errors in the GCHP/Shared directory, you may check your C pre-processor version with the following command:

$ cpp --version

cpp versions in the 4.x series (4.8.5) have been used to successfully build GCHP, while at least one version in the 5.x series (5.4.0) have failed to correctly process the Fortran sources in the Shared directories for compilation.

NetCDF Libraries

A netCDF library installation is required to run GCHP. If you are using GEOS-Chem on a shared computer system, chances are that your IT staff will have already installed one or more netCDF library versions that you can use. Please note that parallel reading and writing requires netCDF-4 and requires that it be compiled with parallel-enabled HDF5 libraries. NetCDF-3 does not have parallel capabilities.

Starting with netCDF-4.2, the Fortran netCDF library split off as an independent distribution to be built after building the C library. Prior to that version the Fortran netCDF library was bundled with the C library in a single distribution of netCDF. We have successfully built GCHP using versions of netCDF both before and after the split.

For more information about netCDF libraries, please see the wiki Guide to netCDF in GEOS-Chem.

Downloading and Installing Software using Spack Package Manager

The Spack Package Manager may be used to download and build CMake, MPI, and NetCDF libraries needed for GCHP. You will need to have a C/C++/Fortran compiler such as GNU Compiler Collection available locally before you start. You can use this local compiler to later install a different compiler version through Spack. The following steps successfully create a GCHP environment on Harvard's Cannon cluster using GCC 9.3.0 and OpenMPI 4.0.4 through Spack. Info on using Intel compilers is coming soon.

To begin using Spack, clone the latest version by typing git clone https://github.com/spack/spack.git. Execute the following commands to initialize Spack's environment (replacing /path/to/spack with the path of your spack directory). Add these commands to an environment initialization script for easy reuse.

       $ export SPACK_ROOT=/path/to/spack
       $ . $SPACK_ROOT/share/spack/setup-env.sh

If you do not already have a copy of your preferred text editor, you can use Spack to install and load one before proceeding (e.g. spack install emacs; spack load emacs).

Ensure Spack recognizes a compiler by typing spack compilers. You will use this compiler to build GCC 9.3.0. Execute the following at the command prompt:

       $ spack install gcc@9.3.0

The @VERSION notation is common to all packages in Spack and can be customized to choose different versions. The installation of gcc may take a long time. Once it is complete, you'll need to add it to Spack's list of compilers using the following command:

       $ spack compiler find $(spack location -i gcc@9.3.0)

Make sure that spack recognizes your new compiler by typing spack compilers, which should display a list of compilers including GCC 9.3.0.

You should now install Git and CMake using Spack:

       $ spack install git@2.17.0%gcc@9.3.0
       $ spack install cmake@3.16.1%gcc@9.3.0

If you do not intend to use a job scheduler like Slurm to run GCHP, use the following commands to install OpenMPI and NetCDF-Fortran. Otherwise, scroll down to see necessary modifications you must make to include Slurm support.

       $ spack install openmpi@4.0.4%gcc@9.3.0
       $ spack install netcdf-fortran%gcc@9.3.0 ^netcdf-c^hdf5^openmpi@4.0.4

Configuring libraries with Slurm support

If you intend to submit GCHP runs through a job scheduler like Slurm, you will need to explicitly build your MPI library with support for your pre-installed job scheduler (using Slurm as an example). If you know the install location of Slurm, edit your spack packages settings at $HOME/.spack/packages.yaml (you may need to create this file) with the following:

packages:
 slurm:
   paths:
     slurm: /path/to/slurm
   buildable: False

This will ensure that when your MPI library is built with Slurm support requested, Spack will correctly use your preinstalled Slurm rather than trying to install a new version.

If you do not know the install location of Slurm, modify $HOME/.spack/packages.yaml with the lines above, but exclude the path designation:

packages:
 slurm:
   buildable: False

In this case where you do not know Slurm's location, you will also need to modify the arguments passed to the OpenMPI build configuration through Spack. Open $SPACK_ROOT/var/spack/repos/builtin/packages/openmpi/package.py and locate the following lines:

       if spec.satisfies('schedulers=slurm'):
           config_args.append('--with-pmi={0}'.format(spec['slurm'].prefix))

Replace these lines with the following:

       if spec.satisfies('schedulers=slurm'):
           #config_args.append('--with-pmi={0}'.format(spec['slurm'].prefix))
           config_args.append('--with-pmi')

This modification should allow the build system to automatically locate Slurm and use a PMI setup that matches your cluster's Slurm configuration.

You may also run into issues building OpenMPI if your cluster has preexisting versions of PMIx that are newer than OpenMPI's internal version. OpenMPI will search for and use the newest version of PMIx installed on your system, which will likely cause a crash during build because OpenMPI requires you to build with the same libevent library as was used to build PMIx. This information may not be readily available to you, in which case you can tweak the build arguments for OpenMPI to always use OpenMPI's internal version of PMIx. Open $SPACK_ROOT/var/spack/repos/builtin/packages/openmpi/package.py and navigate to the configure_args function. In the body of this function, place the following line:

       config_args.append('--with-pmix=internal')

Building libraries with Slurm support

Once any necessary tweaks have been made, you need to tell Spack to build OpenMPI with Slurm support and NetCDF-Fortran with the correct OpenMPI version:

   $ spack install openmpi@4.0.4%gcc@9.3.0 +pmi schedulers=slurm
   $ spack install netcdf-fortran%gcc@9.3.0  ^netcdf-c^hdf5^openmpi@4.0.4+pmi schedulers=slurm

Loading Spack libraries for use with GCHP

After installing the necessary libraries, place the following in a script that you will run before building/running GCHP (such as .bashrc or a separate environment script):

export SPACK_ROOT=$HOME/spack #your path to Spack
source $SPACK_ROOT/share/spack/setup-env.sh
if  $- = *i*  ; then
 echo "Loading Spackages, please wait ..."
fi
#==============================================================================
# %%%%% Load Spackages %%%%%
#==============================================================================
# List each Spack package that you want to load
pkgs=(gcc@9.3.0            \
      git@2.17.0           \
      netcdf-fortran@4.5.2 \
      cmake@3.16.1         \
      openmpi@4.0.4        )

# Load each Spack package
for f in ${pkgs[@]}; do
    echo "Loading $f"
    spack load $f
done
export MPI_ROOT=$(spack location -i openmpi)
export ESMF_COMM=openmpi

You can also add other packages you've installed with Spack like emacs to the pkgs list above.

Installing and Using Intel MPI with Spack

Intel MPI 2019 is available through Spack. Unlike MPI, there are no spec options for using Intel MPI with Slurm. Instead, you will need to set extra environment variables when using Intel MPI.

To build libraries using Intel MPI with GNU compilers, type:

       $ spack install intel-mpi%gcc@9.3.0
       $ spack install netcdf-fortran%gcc@9.3.0 ^intel-mpi

An Intel MPI environment file should include the following:

export SPACK_ROOT=$HOME/spack #your path to Spack
source $SPACK_ROOT/share/spack/setup-env.sh
if  $- = *i*  ; then
 echo "Loading Spackages, please wait ..."
fi
#==============================================================================
# %%%%% Load Spackages %%%%%
#==============================================================================
# List each Spack package that you want to load
pkgs=(gcc@9.3.0            \
      git@2.17.0           \
      netcdf-fortran@4.5.2 \
      cmake@3.16.1         \
      intel-mpi        )

# Load each Spack package
for f in ${pkgs[@]}; do
    echo "Loading $f"
    spack load $f
done

export I_MPI_CC=gcc
export I_MPI_CXX=g++
export I_MPI_FC=gfortran
export I_MPI_F77=gfortran
export I_MPI_F90=gfortran

export I_MPI_PMI_LIBRARY=/path/to/slurm/libpmi2.so #when using srun
#unset I_MPI_PMI_LIBRARY #when using mpirun

export MPI_ROOT=$(spack location -i intelmpi)
export ESMF_COMM=intelmpi

Next | Getting Started with GCHP | GCHP Main Page

--Lizzie Lundgren (talk) 18:06, 6 September 2019 (UTC)