Use Spack to install netCDF on your system

From Geos-chem
Jump to: navigation, search

Previous | Next | Guide to netCDF in GEOS-Chem

  1. Introduction to netCDF
  2. Check if netCDF is already installed on your system
  3. Use Spack to install netCDF on your system
  4. The COARDS netCDF conventions for earth science data
  5. Working with netCDF data files
  6. Creating netCDF data files for GEOS-Chem
  7. Other libraries used by GEOS-Chem


On this page we describe how you can use the Spack package manager tool to install netCDF on your system (if it hasn't already been installed).

What is Spack?

From the Spack site:

Spack is a package management tool designed to support multiple versions and configurations of software on a wide variety of platforms and environments. It was designed for large supercomputing centers, where many users and application teams share common installations of software on clusters with exotic architectures....Spack is non-destructive: installing a new version does not break existing installations, so many configurations can coexist on the same system.
Most importantly, Spack is simple. It offers a simple spec syntax so that users can specify versions and configuration options concisely. Spack is also simple for package authors: package files are written in pure Python, and specs allow package authors to maintain a single file for many different builds of the same package.

Complete Spack documentation may be found at spack.readthedocs.io

If you do not already have a pre-built netCDF library on your system, we recommend using Spack to install the required libraries. Spack should be able to install netCDF and all required libraries for a variety of compiler/platform combinations (including Linux and MacOS).

Instructions for installing netCDF with Spack

Follow these steps to install netCDF on your system. Spack will also install all of the other libraries (such as HDF5 and MPI) that netCDF depends on.

Download Spack

Clone the Spack repository, which is hosted at Github, to your disk space:

git clone https://github.com/spack/spack.git

Set up the environment for Spack

After cloning Spack, you need to run a script that initializes the environment for Spack.

If you use bash, then type:

# For bash/zsh users
$ export SPACK_ROOT=/path/to/spack
$ . $SPACK_ROOT/share/spack/setup-env.sh 

If you use csh or tcsh, then type:

# For tcsh or csh users (note you must set SPACK_ROOT)
$ setenv SPACK_ROOT /path/to/spack
$ source $SPACK_ROOT/share/spack/setup-env.csh

In both examples above, the /path/to/spack is the top-level Spack directory folder on your disk.

After running the setup-env script, the path to the Spack executable will be added to your Unix PATH variable. Therefore, you can just run Spack commands by typing spack COMMAND-NAME.

Make sure that Spack can find your compiler

Before you execute Spack, check to see if Spack recognizes your compiler. Type:

 spack compilers

You should get output similar to this:

==> Available compilers
-- gcc ubuntu18.04-x86_64 ---------------------------------------
gcc@7.3.0

The compilers should be added to your ~/.spack/compilers.yaml file (sometimes this is located at ~/.spack/linux/compilers.yaml, depending on the operating system). To check to see if Spack has found your compiler, type:

spack config get compilers

whichs should give you output similar to this:

compilers:
- compiler:
    environment: {}
    extra_rpaths: []
    flags: {}
    modules: []
    operating_system: ubuntu18.04
    paths:
      cc: /usr/bin/gcc-7
      cxx: /usr/bin/g++-7
      f77: /usr/bin/gfortran-7
      fc: /usr/bin/gfortran-7
    spec: gcc@7.3.0
    target: x86_64

If the cc, cxx, f77, and fc variables are blank, then you can manually specify the paths to these compilers by typing:

spack config edit compilers

Once you are sure that Spack "sees" your compiler, then you may proceed to installing libraries.

You can specify multiple compilers by adding additional -compiler blocks to the compilers.yaml file.

For more information, please see the Compiler configuration section of the Spack manual.

Install libraries with Spack: basic usage

Once you have make sure that Spack has found the compiler, you can proceed to installing the libraries. Here are the commands that you need:

# Configuration Spack installation commands
1 Install netCDF for use with both GEOS-Chem "Classic" and GCHP (also includes the MPI library)
THIS IS OUR RECOMMENDED CONFIGURATION
spack install netcdf-fortran
2 Install netCDF for use with GEOS-Chem "Classic" only
(i.e. will NOT install the MPI library)
spack install netcdf-fortran ^netcdf~mpi ^hdf5~mpi

The commands will tell Spack to download and install the netCDF Fortran-language library along with all of its dependent libraries (such as the netCDF C-language library, the HDF-5 library, an MPI library, etc.). By default, Spack picks the most recent library versions that are available, but you can modify this behavior this with the commands described below. The installation can take about 30-60 minutes, depending on the options that you specify.

If you think you will be using GEOS-Chem in both its "Classic" mode and its high-performance mode (aka GCHP), we recommend that you install netCDF with MPI (Configuration #1). If you are only going to use GEOS-Chem in "Classic" mode, you can omit installing MPI (Configuration #2).

If you do not specify a version number, Spack will install netCDF version 4.7.0 and netCDF-Fortran 4.4.0. These are the latest versions as of June 2019 (but are subject to change). For information about how to install different netCDF versions, please see the next section.

NOTE: For more information on why the netCDF C-language and Fortran-language libraries are installed separately, please follow this link.

Customize the installation process

Using Spack to install our recommended library configuration listed above) should be sufficient for most GEOS-Chem applications. But Spack also lets you customize certain aspects of the installation process. The table below gives some common examples:

Action Spack commands
Tell Spack to build libraries without depending on other libraries
(e.g. build netCDF and its HDF5 dependency without MPI)
spack install netcdf-fortran ^netcdf~mpi ^hdf5~mpi
Tell Spack to install specific library versions instead of the most recent versions: spack install netcdf-fortran@4.4.0 netcdf@4.6
Tell Spack to install libraries using a specific compiler version:
  • You might need to explicitly specify the compiler to use if Spack found multiple compilers on your system (e.g. Intel, GNU)
spack install netcdf-fortran %gcc@8.2.0

For more information about customization, please see the Spack beginner's tutorial:

IMPORTANT NOTE: You can use Spack to install multiple versions of each library (e.g. netCDF for Intel Fortran, netCDF for GNU Fortran, or different netCDF versions for a given compiler). Spack will keep each library installation separate from each other, and you can just simply pick which one you would like to use with GEOS-Chem (see next section).

--Bob Yantosca (talk) 16:01, 11 June 2019 (UTC)

Define GEOS-Chem environment variables that point to the Spack library paths

We recommend that you use the spack location command as shown below to automatically insert the paths to the netCDF and netCDF-Fortran libraries into one of your Unix environment startup scripts (such as .bashrc or .bash_aliases):

# Environment variables for the netCDF C-language interface
export NETCDF_HOME=$(spack location -i netcdf)
export GC_BIN=$NETCDF_HOME/bin
export GC_INCLUDE=$NETCDF_HOME/include
export GC_LIB=$NETCDF_HOME/lib

# Environment variables for the netCDF Fortran-languge interface
export NETCDF_FORTRAN_HOME=$(spack location -i netcdf-fortran)
export GC_F_BIN=$NETCDF_FORTRAN_HOME/bin
export GC_F_INCLUDE=$NETCDF_FORTRAN_HOME/include
export GC_F_LIB=$NETCDF_FORTRAN_HOME/lib

If you have more than one version of the same library installed, then you can use specifiers in the spack location command to explicitly request a given version:

# Get the path for netCDF 4.7.0 compiled with GNU Fortran 7.3.0
export NETCDF_HOME=$(spack location -i netcdf@4.7.0%gcc@7.3.0)
... etc ... 

# Get the path for netCDF-Fortran 4.4.0 compiled with GNU Fortran 7.3.0
export NETCDF_FORTRAN_HOME=$(spack location -i netcdf-fortran@4.4.0%gcc@7.3.0)
... etc ... 

If you need to manually find the paths where Spack has installed the netCDF and netCDF-Fortran libraries, you can use these commands:

spack find --paths netcdf netcdf-fortran

which will display output similar to this:

==> 2 installed packages
-- linux-ubuntu18.04-x86_64 / gcc@7.3.0 -------------------------
    netcdf@4.7.0          /home/ubuntu/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.3.0/netcdf-4.7.0-cryif4blmrhr2i43pif2scmlwv3yu3nq
    netcdf-fortran@4.4.4  /home/ubuntu/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.3.0/netcdf-fortran-4.4.4-5xmcthsjv7qqmlyby3szrbs3pcphh3tp

Please see our Setting Unix environment variables for GEOS-Chem wiki page for more information about other environment variables that you may need to define.

For more information

For complete instructions on using Spack, please see the Spack manual

If you need to manage a lot of separate software libraries, consider creating Spack environments. You can easily switch back and forth between different environments. Please see this tutorial for more information:

You can use Spack to create module files for use with the Lmod module manager, which is used on many HPC cluster systems. For more information, please see this tutorial:

Here is a useful tutorial about using Spack to install libraries for High-Performance Computing applications:

For more information about using Spack with Docker and Singularity software containers, please see this tutorial:

Report errors or technical issues to Spack Support

While the GEOS-Chem Support Team recommends Spack as a useful third-party tool, it is not responsible for solving Spack-related issues.

We encourage you to report all errors and technical issues directly to the Spack Support Team by opening a ticket on the Spack issue tracker:

Please describe fully the commands that you used, as well as the type of system (OS, compilers, etc.) that you are using. A member of the Spack Support Team should be able to answer your question more fully.

Spack is a cutting-edge tool for installing HPC packages. That being said, there may still be compatibility issues with certain library/OS/compiler combinations. The Spack Support Team is continually trying to improve Spack and to resolve any such issues.

--Bob Yantosca (talk) 16:15, 11 June 2019 (UTC)



Previous | Next | Guide to netCDF in GEOS-Chem