Difference between revisions of "Use Spack to install netCDF on your system"

From Geos-chem
Jump to: navigation, search
(Install libraries with Spack: basic usage)
(Instructions for installing netCDF with Spack)
Line 37: Line 37:
 
If you do not already have a pre-built netCDF library on your system, we recommend using Spack to install the required libraries.  Spack should be able to install netCDF and all required libraries for a variety of compiler/platform combinations (including Linux and MacOS).
 
If you do not already have a pre-built netCDF library on your system, we recommend using Spack to install the required libraries.  Spack should be able to install netCDF and all required libraries for a variety of compiler/platform combinations (including Linux and MacOS).
  
== Instructions for installing netCDF with Spack ==
+
== Using Spack to install libraries for GEOS-Chem Classic ==
  
Follow these steps to install netCDF (and its dependent libraries) on your system. (We will assume that you are using the bash Unix shell.)
+
This documentation has been migrated to our [https://github.com/geoschem/geos-chem/issues/637 Github issue geoschem/geos-chem #637].
  
=== Optional: Install a module manager ===
+
== Using Spack to install libraries for GCHP ==
  
You can use a module manager such as [https://lmod.readthedocs.io/en/latest/ '''Lmod'''] or [https://modules.readthedocs.io/en/latest/ '''environment-modules'''] to load Spack-built libraries into your computational environment.  This step is not necessary, but it does make loading Spack-bullt libraries much easier.
+
This documentation has been migrated to our [https://gchp.readthedocs.io GCHP ReadTheDocs site].  Please see the chapter entitled [https://gchp.readthedocs.io/en/latest/supplement/spack.html ''Installing GCHP and Dependencies with Spack''].
 
+
For example, If you are using the Ubuntu OS, you can install '''environment-modules''' with the commands listed below. 
+
 
+
sudo apt install environment-modules
+
. /etc/profile.d/modules.sh
+
 
+
You must have root permission to execute the <tt>sudo</tt> command.  If you do not have root permission, you can ask your IT staff or sysadmin to install a module manager for you.
+
 
+
''NOTE: If you are on an Amazon Web Services cloud computing instance, then you own the entire node that comes with the instance, so you are automatically granted root permission.''
+
 
+
If you are using CentOS, then you can follow a similar command: replace <tt>apt</tt> with <tt>yum</tt>, which is the CentOS package manager.
+
 
+
=== Download Spack ===
+
 
+
Clone the Spack repository, which is hosted at Github, to your disk space:
+
 
+
git clone https://github.com/spack/spack.git
+
 
+
=== Set up the environment for Spack ===
+
 
+
After cloning Spack, you need to run a script that initializes the environment for Spack:
+
 
+
export SPACK_ROOT=/path/to/spack
+
. $SPACK_ROOT/share/spack/setup-env.sh
+
 
+
After running the setup-env script, the path to the Spack executable will be added to your Unix PATH variable.  Therefore, you can just run Spack commands by typing <tt>spack COMMAND-NAME</tt>.
+
 
+
=== Make sure that Spack can find your compiler ===
+
 
+
Before you execute Spack, check to see if Spack recognizes your compiler.  Type:
+
 
+
  spack compilers
+
 
+
You should get output similar to this:
+
 
+
==> Available compilers
+
-- gcc ubuntu18.04-x86_64 ---------------------------------------
+
gcc@7.3.0
+
 
+
The compilers should be added to your <tt>~/.spack/linux/compilers.yaml</tt> file.  To check if Spack has found your compiler, type:
+
 
+
spack config get compilers
+
 
+
whichs should give you output similar to this:
+
 
+
compilers:
+
- compiler:
+
    environment: {}
+
    extra_rpaths: []
+
    flags: {}
+
    modules: []
+
    operating_system: ubuntu18.04
+
    paths:
+
      cc: /usr/bin/gcc-7
+
      cxx: /usr/bin/g++-7
+
      f77: /usr/bin/gfortran-7
+
      fc: /usr/bin/gfortran-7
+
    spec: gcc@7.3.0
+
    target: x86_64
+
 
+
If the cc, cxx, f77, and fc variables are blank, then you can manually specify the paths to these compilers by typing:
+
 
+
spack config edit compilers
+
 
+
which will bring the <tt>~/.spack/linux/compilers.yaml</tt> file in an editor window, usually <tt>vim</tt>.  Or you can manually open the <tt>~/.spack/linux/compilers.yaml</tt> file in your favorite editor (e.g. emacs) and save it.  You can specify multiple compilers by adding additional <tt>-compiler</tt> blocks to the <tt>compilers.yaml</tt> file.
+
 
+
For more information, please see the [https://spack.readthedocs.io/en/latest/getting_started.html#compiler-configuration ''Compiler configuration'' section of the Spack manual].
+
 
+
Once you are sure that Spack "sees" your compiler, then you may proceed to installing libraries.
+
 
+
=== Install libraries with Spack: basic usage ===
+
 
+
Once you have make sure that Spack has found the compiler, you can proceed to installing the librariesHere are the commands that you need:
+
 
+
{| border=1 cellspacing=0 cellpadding=5
+
|-bgcolor="#CCCCCC" valign="top"
+
!width="50px"|#
+
!width="525px"|Configuration
+
!width="400px"|Spack installation commands
+
 
+
|-valign="top"
+
|1
+
|Install netCDF (and dependent libraries, also including MPI).<br>'''''THIS IS OUR RECOMMENDED CONFIGURATION.'''''
+
|<tt>spack install netcdf-fortran</tt>
+
 
+
|-valign="top"
+
|2
+
|Install netCDF for use with GEOS-Chem "Classic" only<br>'''''This will NOT install the MPI library.'''''
+
+
''NOTE: This configuration can cause problems when [[#Optional: Install other applications via Spack|installing other packages with Spack]].  Use with caution!''
+
|<tt>spack install netcdf-fortran<span style="color:darkorange">~mpi</span></tt>
+
 
+
|}
+
 
+
The commands will tell Spack to download and install the '''netCDF Fortran-language library''' along with all of its dependent libraries (such as the '''netCDF C-language library''', the '''HDF-5 library''', an '''MPI library''', etc.).  By default, Spack picks the most recent library versions that are available, but you can modify this behavior this with the commands described below.  The installation can take about 30-60 minutes, depending on the options that you specify.
+
 
+
If you think you will be using GEOS-Chem in both its "Classic" mode and its high-performance mode (aka GCHP), we recommend that you install netCDF with MPI (Configuration #1).  If you are only going to use GEOS-Chem in "Classic" mode, you can omit installing MPI (Configuration #2).
+
 
+
If you do not specify a version number, Spack will install <tt>netcdf-c</tt> version 4.7.3 and <tt>netcdf-fortran</tt> version 4.5.2.  These are the latest versions as of January 2020 (but are subject to change).  For information about how to install different netCDF versions, [[#Customize the installation process|please see the next section]].
+
 
+
''NOTE: For more information on why the netCDF C-language and Fortran-language libraries are installed separately, [[Introduction to netCDF#netCDF_4.2_and_later_versions_require_a_separate_netCDF-Fortran_installation|please follow this link]].
+
 
+
=== Customize the installation process ===
+
 
+
Using Spack to install [[#Install libraries with Spack|our recommended library configuration listed above]] should be sufficient for most GEOS-Chem applications.  But Spack also lets you customize certain aspects of the installation process.  The table below gives some common examples:
+
 
+
{| border=1 cellspacing=0 cellpadding=5
+
|-bgcolor="#CCCCCC" valign="top"
+
!width="600px"|Action
+
!width="400px"|Spack commands
+
 
+
|-valign="top"
+
|<span style="color:green">Turn ON an option</span> when installing a library:
+
*e.g. build emacs <span style="color:green">with the X11 library</span>
+
*NOTE: <tt>spack info library-name</tt> lists all installation options for a given library.
+
*NOTE: Library names must use all lowercase letters.
+
|<tt>spack install emacs<span style="color:green">+X</span></tt>
+
 
+
|-valign="top"
+
|<span style="color:darkorange">Turn OFF an option</span> when installing a library:
+
*e.g. build <tt>netcdf-fortran</tt> and dependent libraries <span style="color:darkorange">without MPI</span>
+
 
+
|<tt>spack install netcdf-fortran<span style="color:darkorange">~mpi</span></tt> OR<br><tt>spack install netcdf-fortran<span style="color:darkorange">-mpi</span></tt>
+
 
+
|-valign="top"
+
|Tell Spack to install libraries using a <span style="color:blue">specific compiler version</span>:
+
*e.g. build <tt>netcdf-fortran</tt> etc. <span style="color:blue">with the GNU Compiler Collection 8.2.0</span>
+
|<tt>spack install netcdf-fortran <span style="color:blue">%gcc@8.2.0</span></tt>
+
 
+
|-valign="top"
+
|Tell Spack to install <span style="color:red">specific library versions</span> instead of the most recent versions:
+
|<tt>spack install netcdf-fortran<span style="color:red">@4.4.0</span> netcdf<span style="color:red">@4.6</span></tt>
+
 
+
|}
+
 
+
For more information about customization, please see [https://spack.readthedocs.io/en/latest/tutorial_basics.html Spack Tutorial 101] on the Spack documentation page.
+
 
+
'''IMPORTANT NOTE:''' You can use Spack to install multiple versions of each library (e.g. for Intel Fortran, for GNU Fortran, or different netCDF versions for the same compiler).  Spack will keep each library installation separate from each other, and you can just simply pick which one you would like to use with GEOS-Chem (see next section).
+
 
+
=== Define GEOS-Chem environment variables that point to the Spack library paths ===
+
 
+
We recommend that you use the <tt>spack location</tt> command as shown below to automatically insert the paths to the <tt>netcdf-c</tt> and <tt>netcdf-fortran</tt> libraries into one of your Unix environment startup scripts (such as <tt>.bashrc</tt> or  <tt>.bash_aliases</tt>):
+
 
+
# Environment variables for the netCDF C-language interface
+
export NETCDF_HOME=<span style="color:red">$(spack location -i netcdf-c)</span>
+
export GC_BIN=$NETCDF_HOME/bin
+
export GC_INCLUDE=$NETCDF_HOME/include
+
export GC_LIB=$NETCDF_HOME/lib
+
+
# Environment variables for the netCDF Fortran-languge interface
+
export NETCDF_FORTRAN_HOME=<span style="color:red">$(spack location -i netcdf-fortran)</span>
+
export GC_F_BIN=$NETCDF_FORTRAN_HOME/bin
+
export GC_F_INCLUDE=$NETCDF_FORTRAN_HOME/include
+
export GC_F_LIB=$NETCDF_FORTRAN_HOME/lib
+
 
+
NOTE: Instead of placing each <tt>spack location</tt> command inside <tt>$()</tt>, you can use backticks instead:
+
 
+
export NETCDF_HOME=<span style="color:red">`spack location -i netcdf-c`</span>
+
...
+
export NETCDF_FORTRAN_HOME=<span style="color:red">`spack location -i netcdf-fortran`</span>
+
 
+
Either syntax will execute each <tt>spack location</tt> command, and store its output in the corresponding environment variable.
+
 
+
If you have more than one version of the same library installed, then you can use specifiers in the <tt>spack location</tt> command to explicitly request a given version:
+
 
+
# Get the path for <span style="color:red">netCDF 4.7.0</span> compiled with <span style="color:blue">GNU Fortran 7.3.0</span>
+
export NETCDF_HOME=$(spack location -i <span style="color:red">netcdf@4.7.3</span><span style="color:blue">%gcc@7.3.0</span>)
+
... etc ...
+
+
# Get the path for <span style="color:darkorange">netCDF-Fortran 4.4.0</span> compiled with <span style="color:blue">GNU Fortran 7.3.0</span>
+
export NETCDF_FORTRAN_HOME=$(spack location -i <span style="color:darkorange">netcdf-fortran@4.5.2</span><span style="color:blue">%gcc@7.3.0</span>)
+
... etc ...
+
 
+
If you need to manually find the paths where Spack has installed the netCDF and netCDF-Fortran libraries, you can use these commands:
+
 
+
spack find --paths netcdf-c netcdf-fortran
+
 
+
which will display output similar to this:
+
 
+
==> 2 installed packages
+
-- linux-ubuntu18.04-x86_64 / gcc@7.3.0 -------------------------
+
    netcdf-c@4.7.3        /home/ubuntu/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.3.0/netcdf-4.7.3-cryif4blmrhr2i43pif2scmlwv3yu3nq
+
    netcdf-fortran@4.5.2  /home/ubuntu/spack/opt/spack/linux-ubuntu18.04-x86_64/gcc-7.3.0/netcdf-fortran-4.5.2-xmcthsjv7qqmlyby3szrbs3pcphh3tp
+
 
+
Please see ''[[Configuring your computational environment]]'' for more information about other environment variables that you may need to define.
+
 
+
==== netcdf or netcdf-c? ====
+
 
+
Some older versions of Spack used <tt>netcdf</tt> to refer to the netCDF C-library installation.  Newer versions of Spack now use <tt>netcdf-c</tt>.  This can sometimes cause confusion.
+
 
+
=== Optional: Load Spack libraries with a module manager ===
+
 
+
If you have [[#Optional: Install a module manager|installed a module manager]] on your system, then you can load Spack libraries by typing
+
 
+
spack load netcdf-fortran
+
spack load netcdf-c
+
 
+
etc.  This will update your Unix <tt>$PATH</tt> variable with the file paths of any executable files (such as <tt>ncdump</tt>, <tt>nc-config</tt>, <tt>nf-config</tt>, etc, so that your Unix shell will know where to find it.
+
 
+
=== Optional: Install other applications via Spack ===
+
 
+
Although we have focused on installing <tt>netcdf-c</tt>, <tt>netcdf-fortran</tt>, and their dependent libraries, you can also use Spack to install other useful tools for working with GEOS-Chem.  Some of these tools are:
+
 
+
{| border=1 cellspacing=0 cellpadding=5
+
|-bgcolor="#CCCCCC" valign="top"
+
!width="100px"|Library
+
!width="800px"|Description
+
 
+
|-valign="top"
+
|<tt>cdo</tt> || The Climate Data Operators, useful for [[Working with netCDF data files|working data files in netCDF format]].
+
 
+
|-valign="top"
+
|<tt>cgdb</tt> || A graphical interface wrapper for GNU debugger (shows code in a separate window)
+
 
+
|-valign="top"
+
|<tt>cmake</tt> || The CMake build system.  Creates Makefiles that can be used to build [[GEOS-Chem 12#12.6.0|GEOS-Chem 12.6.0]] and later versions.
+
 
+
|-valign="top"
+
|<tt>flex</tt> || The Fast Lexical Analyzer Generator.  This package is a dependency of the [[FlexChem|KPP chemical solver]].  If you create new chemical mechanisms for GEOS-Chem with the KPP code, then you will need to install this.
+
 
+
|-valign="top"
+
|<tt>gdb</tt> || The GNU Debugger.  Useful for finding where a GEOS-Chem simulation halted with an error.
+
 
+
|-valign="top"
+
|<tt>gmake</tt> || The GNU Make build system.  This is the current language in which the GEOS-Chem Makefiles are written.  Can also be used to build GEOS-Chem from CMake-generated Makefiles.
+
 
+
|-valign="top"
+
|<tt>openjdk</tt> || The Open-Source version of the Java Development Kit (JDK). This may be required for other packages.  Also, if you use the Panoply file viewer, you will need to build this if your system does not have a version of JDK installed.
+
 
+
|-valign="top"
+
|<tt>ncview</tt> || Quick & dirty netCDF file viewer.  Useful for checking netCDF files to see if the data contained in the file looks as expected.
+
 
+
|-valign="top"
+
|<tt>nco</tt> || The netCDF operators, useful for [[Working with netCDF data files|working with data files in netCDF format]].
+
 
+
|-valign="top"
+
|<tt>tau</tt> || Tuning and Analysis Utilities, a performance profiler that can find computational bottlenecks in GEOS-Chem subroutines.
+
 
+
|}
+
 
+
Please see our [https://www.youtube.com/watch?v=6tGCD3zQQW0 '''Spack Tutorial #2'''] video on our GEOS-Chem YouTube channel ([http://youtube.geos-chem.org '''youtube.geos-chem.org''']) for more information about installing these additional Spack packages.
+
 
+
=== For more information ===
+
 
+
For complete instructions on using Spack, please see the Spack manual
+
* https://spack.readthedocs.io/en/latest
+
 
+
If you need to manage a lot of separate software libraries, consider creating Spack environments.  You can easily switch back and forth between different environments.  Please see this tutorial for more information:
+
*https://spack.readthedocs.io/en/latest/tutorial_environments.html
+
 
+
You can use Spack to create module files for use with the Lmod module manager, which is used on many HPC cluster systems.  For more information, please see this tutorial:
+
*https://spack.readthedocs.io/en/latest/tutorial_modules.html
+
 
+
Here is a useful tutorial about using Spack to install libraries for High-Performance Computing applications:
+
* https://www.youtube.com/watch?v=BxNOxHu6FAI
+
 
+
For more information about using Spack with Docker and Singularity software containers, please see this tutorial:
+
*https://spack.readthedocs.io/en/latest/workflows.html?highlight=singularity#using-spack-to-create-docker-images
+
 
+
--[[User:Bmy|Bob Yantosca]] ([[User talk:Bmy|talk]]) 22:27, 16 January 2020 (UTC)
+
  
 
== Report errors or technical issues to Spack Support ==
 
== Report errors or technical issues to Spack Support ==

Revision as of 15:03, 2 March 2021

Previous | Next | Guide to netCDF in GEOS-Chem

  1. Introduction to netCDF
  2. Check if netCDF is already installed on your system
  3. Use Spack to install netCDF on your system
  4. The COARDS netCDF conventions for earth science data
  5. Working with netCDF data files
  6. Creating netCDF data files for GEOS-Chem
  7. Other libraries used by GEOS-Chem


On this page we describe how you can use the Spack package manager tool to install netCDF on your system (if it hasn't already been installed).

Featured tutorial videos

The following tutorials about using Spack to build the compiler and libraries are available at our GEOS-Chem Youtube channel: youtube.geos-chem.org:

Spack 1 thumbnail.png
Click HERE to view!!
Spack 2 thumbnail.png
Click HERE to view!!

--Bob Yantosca (talk) 14:56, 16 January 2020 (UTC)

What is Spack?

From the Spack site:

Spack is a package management tool designed to support multiple versions and configurations of software on a wide variety of platforms and environments. It was designed for large supercomputing centers, where many users and application teams share common installations of software on clusters with exotic architectures....Spack is non-destructive: installing a new version does not break existing installations, so many configurations can coexist on the same system.
Most importantly, Spack is simple. It offers a simple spec syntax so that users can specify versions and configuration options concisely. Spack is also simple for package authors: package files are written in pure Python, and specs allow package authors to maintain a single file for many different builds of the same package.

Complete Spack documentation may be found at spack.readthedocs.io

If you do not already have a pre-built netCDF library on your system, we recommend using Spack to install the required libraries. Spack should be able to install netCDF and all required libraries for a variety of compiler/platform combinations (including Linux and MacOS).

Using Spack to install libraries for GEOS-Chem Classic

This documentation has been migrated to our Github issue geoschem/geos-chem #637.

Using Spack to install libraries for GCHP

This documentation has been migrated to our GCHP ReadTheDocs site. Please see the chapter entitled Installing GCHP and Dependencies with Spack.

Report errors or technical issues to Spack Support

While the GEOS-Chem Support Team recommends Spack as a useful third-party tool, it is not responsible for solving Spack-related issues.

We encourage you to report all errors and technical issues directly to the Spack Support Team by opening a ticket on the Spack issue tracker:

Please describe fully the commands that you used, as well as the type of system (OS, compilers, etc.) that you are using. A member of the Spack Support Team should be able to answer your question more fully.

Spack is a cutting-edge tool for installing HPC packages. That being said, there may still be compatibility issues with certain library/OS/compiler combinations. The Spack Support Team is continually trying to improve Spack and to resolve any such issues.

--Bob Yantosca (talk) 16:15, 11 June 2019 (UTC)



Previous | Next | Guide to netCDF in GEOS-Chem