Check if netCDF is already installed on your system

From Geos-chem
Revision as of 22:04, 16 January 2020 by Bmy (Talk | contribs) (If you are using GEOS-Chem on cloud computing platforms)

(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to: navigation, search

Previous | Next | Guide to netCDF in GEOS-Chem

  1. Introduction to netCDF
  2. Check if netCDF is already installed on your system
  3. Use Spack to install netCDF on your system
  4. The COARDS netCDF conventions for earth science data
  5. Working with netCDF data files
  6. Creating netCDF data files for GEOS-Chem
  7. Other libraries used by GEOS-Chem


On this page we describe how you can look to see if netCDF is already installed on your system. If it is, then you don't have to install it yourself!

If you are using GEOS-Chem on a computer cluster

If you are going to use GEOS-Chem on a shared computer system, chances are that your IT staff will have already installed one or more netCDF library versions that you can use. Depending on your system's setup, there are several ways that you can tell your computational environment where to find the netCDF library files, as described below.

First, check if there are available netCDF modules

Many high-performance computing (HPC) clusters use a module manager such as Lmod or environment-modules to load software packages and libraries. A module manager allows you to load different compilers and libraries with simple module load commands. For example, on the Harvard Odyssey cluster, compiler and netCDF libraries are initialized with commands such as these:

module purge
module load gcc/8.2.0-fasrc01
module load openmpi/3.1.1-fasrc01
module load netcdf/4.1.3-fasrc02

The first line removes all pre-loaded modules. The second line loads the GNU C and Fortran compilers (version 8.2.0). The third and fourth lines openmpi 3.1.1 (which netCDF depends on), and finally netCDF 4.1.3 itself. You can add these module load statements into your system startup files (e.g. .bashrc, .bash_aliases), etc.

As a convenience, your module manager may also export the relevant folder paths to your Unix environment. For example, issuing the above module statements on the Harvard Odyssey cluster will export the following environment variables:

$GCC_HOME        # Home folder for gcc 8.2.0
$GCC_INCLUDE     # Folder where include files of gcc 8.2.0 are stored
$GCC_LIB         # Folder where library files of gcc 8.2.0 are stored
$MPI_HOME        # Home folder for openmpi 3.1.1
$MPI_INCLUDE     # Folder where include files (e.g. mpi.h) of openmpi 3.1.1 are stored
$MPI_LIB         # Folder where library files (e.g. libmpi*.a) openmpi 3.1.1 are stored
$NETCDF_HOME     # Home folder for netcdf-4.1.3
$NETCDF_INCLUDE  # Folder where include files (e.g. netcdf.h, netcdf.inc) are stored
$NETCDF_LIB      # Folder where library files (e.g. libnetcdf.a, libnetcdff.a) for netCDF 4.1.3 are stored

You can then use these environment variables to tell GEOS-Chem where it can find the netCDF libraries on your system. See our Setting Unix environment variables for GEOS-Chem wiki page for more information. NOTE: The names of these environment variables may be different on your system (ask you IT staff for more information).

Module managers make it very easy to switch between different compilers and libraries. To load the netCDF libraries that were built with the Intel Fortran Compiler, all one has to do is to use a different set of module load statements, such as:

module purge
module load intel/17.0.4-fasrc01
module load openmpi/2.1.0-fasrc02
module load netcdf/4.3.2-fasrc05
module load netcdf-fortran/4.4.0-fasrc03

NOTE: For an explanation of why netCDF-Fortran is loaded as a separate module, please see this section.

If netCDF-Fortran is installed as a separate module, then your module manager may also define additional environment variables for you. For example, on the Harvard Odyssey cluster, the following environment variables are defined when a netCDF-Fortran module is loaded:

$NETCDF_FORTRAN_HOME     # Home folder for netcdf-4.4.0
$NETCDF_FORTRAN_INCLUDE  # Folder where include files (e.g. netcdf.h, netcdf.inc) for netCDF-Fortran 4.4.0 are stored
$NETCDF_FORTRAN_LIB      # Folder where library files (e.g. libnetcdf.a, libnetcdff.a) for netCDF-Fortran 4.4.0 are stored

One downside of using a module manager is that you are locked into using only those compiler and software versions that have already been installed on your system. For example, an update to computer model that you are using might also updating to a new compiler version that is not yet available on your system. In this case, you will need to request that your IT staff install the new compiler version for you (and wait for them to do it). But in general, module managers succeed in ensuring that only well-tested compiler/software combinations are available to users.

Also note: not all module managers will create relevant environment variables when loading a package, so you may need to figure out how to define those manually. Ask your IT staff or sysadmin for more information.

Next, check if there is a Spack-built netCDF installation

If your system doesn't have a module manager installed, check to see if the netCDF libraries were built the Spack package manager. You can type

spack find

to see if there are any Spack-built netCDF and/or netCDF-Fortran libraries on your system. If your system also has a module manager installed, then you can load libraries with the spack load command, e.g.

spack load netcdf-c
spack load netcdf-fortran
... etc ...

If not, then check to see a Spack environment has been installed. A Spack environment will load several libraries at once (similar to how Conda loads several python packages at once). You can usually use:

spack activate ENVIRONMENT-NAME

and

despacktivate

to enter and exit the environment.

For more information about Spack, the next chapter of this Guide.

Next, check if there is a manual netCDF installation

If your computer system does not use a module manager then the netCDF libraries may have already been installed by your IT staff in one of the usual Unix folder locations (such as /usr/lib or /usr/local/lib). If this is the case, ask your IT staff where these libraries reside.

Once you know the location of the compiler and netCDF libraries, you can set the proper environment variables for GEOS-Chem.

Finally, install netCDF if you cannot find it on your system

Proceed to the next chapter for instructions on how you (or your IT staff) can use Spack to install netCDF and related libraries on your system.

--Bob Yantosca (talk) 18:29, 12 June 2019 (UTC)

If you are using GEOS-Chem on cloud computing platforms

If you are using GEOS-Chem on the Amazon Web Services cloud computing platform, then the netCDF libraries will already be installed for you, either as part of the Amazon Machine Image (AMI) or software container (e.g. Docker or Singularity) that you used to initialize your computational environment. The proper Unix environment variables will also be defined.

Some AMI's will also have Spack-built libraries that can be loaded with the environment-modules module manager, or via a Spack environment.

For more information, please see our comprehensive cloud computing tutorial: cloud.geos-chem.org

--Bob Yantosca (talk) 22:04, 16 January 2020 (UTC)



Previous | Next | Guide to netCDF in GEOS-Chem