Difference between revisions of "Setting Unix environment variables for GEOS-Chem"

From Geos-chem
Jump to: navigation, search
(With bash)
 
(9 intermediate revisions by 2 users not shown)
Line 1: Line 1:
This page contains information about how to set several environment variables in your Unix environment that are needed to [[GEOS-Chem Makefile Structure#Compiling GEOS-Chem|compile GEOS-Chem]].
+
This content has been migrated to the [https://geos-chem.readthedocs.io/en/latest/gcc-guide/01-startup/login-env.html# '''Customize your login environment''' chapter of <tt>geos-chem.readthedocs.io</tt>].
 
+
== Environment variables that specify compiler names ==
+
 
+
GEOS-Chem currently supports the [[Intel Fortran Compiler|Intel Fortran compiler (<tt>ifort</tt>)]], the [[GNU Fortran compiler|GNU Fortran compiler (<tt>gfortran</tt>)]], and the [[PGI Fortran compiler|PGI Fortran compiler (<tt>pgfortran</tt> or <tt>pgf90</tt>)]].  You must set the following variables in your Unix environment to tell GEOS-Chem which compiler type you are using.
+
 
+
{| border=1 cellspacing=0 cellpadding=5
+
|-valign="top" bgcolor="#CCCCCC"
+
!width="100px"|Variable
+
!width="600px"|Description
+
 
+
|-valign="top"
+
|<tt>FC</tt> || Name of the Fortran compiler
+
 
+
|-valign="top"
+
|<tt>CC</tt> || Name of the C compiler.<br><span style="color:darkorange">'''''Not needed for GEOS-Chem "Classic", but is needed for GCHP.'''''</span>
+
 
+
|-valign="top"
+
|<tt>CXX</tt> || Name of the C++ compiler.<br><span style="color:darkorange">'''''Not needed for GEOS-Chem "Classic", but is needed for GCHP.'''''</span>
+
 
+
|}
+
 
+
On many systems (such as the Harvard Odyssey cluster), <tt>FC</tt>, <tt>CC</tt>, and <tt>CXX</tt> will be set automatically for you when you load a software module into your Unix environment with the <tt>module load</tt> command. The easiest way to check if these variables have been automatically set for you is to print them to the screen.  Type at the Unix prompt:
+
 
+
echo $FC
+
echo $CC
+
echo $CXX
+
 
+
On the other hand, if <tt>FC</tt>, <tt>CC</tt>, and <tt>CXX</tt> are all undefined, then you will have to manually set them in your startup script, as described in the table below:
+
 
+
{| border=1 cellspacing=0 cellpadding=5
+
|-valign="top" bgcolor="#CCCCCC"
+
!width="125px"|If your<br>shell is
+
!width="125px"|and your<br>compiler is
+
!width="125px"|add to<br>this file
+
!width="300px"|the following lines of code
+
 
+
|-valign="top"
+
|<tt>/bin/bash</tt>
+
|<tt>ifort</tt>
+
|<tt>.bashrc</tt>
+
|<tt>export FC=ifort<br>export CC=icc<br>export CXX=icpc</tt>
+
 
+
|-valign="top" bgcolor="#CCFFFF"
+
|<tt>/bin/bash</tt>
+
|<tt>gfortran</tt>
+
|<tt>.bashrc</tt>
+
|<tt>export FC=gfortran<br>export CC=gcc<br>export CXX=g++</tt>
+
+
|-valign="top"
+
|<tt>/bin/csh</tt> or<br><tt>/bin/tcsh</tt>
+
|<tt>ifort</tt>
+
|<tt>.cshrc</tt>
+
|<tt>setenv FC ifort<br>setenv CC icc<br>setenv CXX icpc</tt>
+
+
|-valign="top" bgcolor="#CCFFFF"
+
|<tt>/bin/csh</tt> or<br><tt>/bin/tcsh</tt>
+
|<tt>gfortran</tt>
+
|<tt>.cshrc</tt>
+
|<tt>setenv FC gfortran<br>setenv CC gcc<br>setenv CXX g++</tt>
+
 
+
|}
+
 
+
Then make sure to type:
+
 
+
source ~/.bashrc  # if you are using bash
+
source ~/.cshrc    # if you are using csh or tcsh
+
 
+
to apply the changes.
+
 
+
--[[User:Bmy|Bob Yantosca]] ([[User talk:Bmy|talk]]) 18:47, 9 January 2019 (UTC)
+
 
+
== Environment variables that specify library paths ==
+
 
+
GEOS-Chem uses the netCDF library for file I/O.  You should first [[Check_if_netCDF_is_already_installed_on_your_system|check to see if there is a pre-built netCDF library installation on your system already]].  If not, then you (or your IT staff) can install GEOS-Chem with [[Use_Spack_to_install_netCDF_on_your_system|the Spack package manager]].
+
 
+
After you have verified that a netCDF library installation exists on your system, the next step is to tell GEOS-Chem where to find the relevant library, include, and executable files.  Otherwise you will get errors during the compilation process. The easiest way to do this is to set environment variables in your setup files:
+
 
+
#<tt>.bashrc :</tt> or <tt>.bash_aliases</tt>: if you use Bourne-Again shell (bash)
+
#<tt>.cshrc  :</tt> if you use C-shell (csh) or T-shell (tcsh)
+
 
+
There are three environment variables that you need to set:
+
 
+
{| border=1 cellspacing=0 cellpadding=5
+
|- bgcolor="#CCCCCC"
+
!width="125px"|Variable
+
!width="875px"|Description
+
 
+
|-valign="top"
+
|<tt>GC_BIN</tt>
+
|Points to the <tt>bin/</tt> subfolder of the root netCDF path. This is where the <tt>nc-config</tt> and <tt>nf-config</tt> files are located.
+
 
+
|-valign="top"
+
|<tt>GC_INCLUDE</tt>
+
|Points to the <tt>include/</tt> subfolder of the root netCDF path.  This is where netCDF include files (<tt>*.h</tt>, <tt>*.inc</tt>) and compiled module files (<tt>*.mod</tt>) for the netCDF (and HDF5) libraries are located.
+
 
+
|--valign="top"
+
|<tt>GC_LIB</tt>
+
|Points to the <tt>lib/</tt> subfolder of the root netCDF path.  (On some systems this may be named <tt>lib64/</tt> instead.) This is where the netCDF library files (<tt>*.a</tt>) are located.
+
|}
+
 
+
 
+
If (and only if) [[Introduction_to_netCDF#netCDF_4.2_and_later_versions_require_a_separate_netCDF-Fortran_installation|netCDF-Fortran is installed as a separate library on your system]], you will also need to set these variables:
+
 
+
{| border=1 cellspacing=0 cellpadding=5
+
|- bgcolor="#CCCCCC"
+
!width="125px"|Variable
+
!width="875px"|Description
+
 
+
|-valign="top"
+
|<tt>GC_F_BIN</tt>
+
|Points to the <tt>bin/</tt> subfolder of the root netCDF Fortran path.  This is where the <tt>nf-config</tt> file is located.
+
 
+
|-valign="top"
+
|<tt>GC_F_INCLUDE</tt>
+
|Points to the <tt>include/</tt> subfolder of the root netCDF Fortran path.  This is where netCDF include files (<tt>*.h</tt>) and compiled module files (<tt>*.mod</tt>) are located.
+
 
+
|-valign="top"
+
|<tt>GC_F_LIB</tt>
+
|Points to the <tt>lib/</tt> subfolder of the root netCDF Fortran path.  (On some systems this may be named <tt>lib64/</tt> instead.) This is where the netCDF library files (<tt>*.a</tt>) are located.
+
|}
+
 
+
The best way to define these variables is to add them to one of your sysetm startup files.
+
 
+
If you are using a HPC cluster with the [[Installing libraries for GEOS-Chem#Using modules|Lmod module manager]], then environment variables that point to the root library folders (e.g. <tt>NETCDF_HOME</tt>, <tt>NETCDF_FORTRAN_HOME</tt>) will already be loaded into your Unix environment.  We recommend using these variables to define the other environment variables as described in following secttions.
+
 
+
If you have any questions about where the library paths for netCDF are located on your system, ask your IT staff.
+
 
+
=== With bash ===
+
 
+
If you use bash, then add the following lines to one of your Unix environment startup files (e.g. <tt>.bashrc</tt>, <tt>.bash_aliases</tt>, etc.):
+
 
+
# Tell GEOS-Chem where to find netCDF library files
+
export NETCDF_HOME=path to netCDF library files
+
export GC_BIN=$NETCDF_HOME/bin
+
export_GC_INCLUDE=$NETCDF_INCLUDE
+
export GC_LIB=$NETCDF_LIB
+
+
# NOTE: If netCDF-Fortran was loaded as a separate module, then
+
# also define these variables. (Otherwise comment these out.)
+
export NETCDF_FORTRAN_HOME=path to netCDF-Fortran library files
+
export GC_F_BIN=$NETCDF_FORTRAN_HOME/bin
+
export GC_F_INCLUDE=$NETCDF_FORTRAN_INCLUDE
+
export GC_F_LIB=$NETCDF_FORTRAN_LIB
+
 
+
Then to accept the changes, type at the Unix prompt: <tt>source ~/.bashrc</tt> (or <tt>source ~/.bash_aliases</tt>, etc.)
+
 
+
==== With csh or tcsh ====
+
 
+
If you use C-shell (csh) or T-shell (tcsh), then add the following lines to your <tt>.cshrc</tt> file:
+
 
+
# Tell GEOS-Chem where to find netCDF library files
+
setenv NETCDF_HOME        path to netCDF library files
+
setenv GC_BIN              $(NETCDF_HOME)/bin
+
setenv GC_INCLUDE          $(NETCDF_INCLUDE)
+
setenv GC_LIB              $(NETCDF_LIB)
+
+
# NOTE: If netCDF-Fortran was loaded as a separate module, then
+
# also define these variables.  (Otherwise comment these out.)
+
setenv NETCDF_FORTRAN_HOME path to my netCDF-Fortran library files
+
setenv GC_F_BIN            $(NETCDF_FORTRAN_HOME)/bin
+
setenv GC_F_INCLUDE        $(NETCDF_FORTRAN_INCLUDE)
+
setenv GC_F_LIB            $(NETCDF_FORTRAN_LIB)
+
 
+
Then to accept the changes, type <tt>source ~/.cshrc</tt> at the Unix prompt.
+
 
+
--[[User:Bmy|Bob Yantosca]] ([[User talk:Bmy|talk]]) 16:10, 10 January 2019 (UTC)
+
 
+
==== For libraries installed with the Spack package manager ====
+
 
+
If you [[Use_Spack_to_install_netCDF_on_your_system|installed the netCDF libraries with the Spack package manager]], then please see [[Use_Spack_to_install_netCDF_on_your_system#Define_GEOS-Chem_environment_variables_that_point_to_the_Spack_library_paths|this wiki post]] which describes how you can define the environment variables to point to the proper library paths.
+
 
+
--[[User:Bmy|Bob Yantosca]] ([[User talk:Bmy|talk]]) 15:10, 17 June 2019 (UTC)
+
 
+
== Environment variables for OpenMP parallelization ==
+
 
+
GEOS-Chem "Classic" uses [[Parallelizing GEOS-Chem|OpenMP parallelization]], which is an implementation of shared-memory (aka serial) parallelization.  Two Unix environment variables control the OpenMP parallelization settings:
+
 
+
=== OMP_NUM_THREADS ===
+
 
+
The <tt>OMP_NUM_THREADS</tt> environment variable sets the number of computational cores (aka threads) that you would like GEOS-Chem to use.  We recommend that you set <tt>OMP_NUM_THREADS</tt> in your <tt>.bashrc</tt> (or <tt>.cshrc</tt> file) as well as in each GEOS-Chem run script that you use.
+
 
+
The following commands will request that GEOS-Chem use 8 cores by default:
+
 
+
{| border=1 cellspacing=0 cellpadding=5
+
|- bgcolor="#CCCCCC"
+
!width="300px"|If you use bash.<br>use this command
+
!width="300px"|If you use csh or tcsh,<br>use this command
+
 
+
|-valign="top"
+
|<tt>export OMP_NUM_THREADS=8</tt>
+
|<tt>setenv OMP_NUM_THREADS 8</tt>
+
|}
+
 
+
You can of course change the number of cores from 8 to however many you want your GEOS-Chem simulation to use.  The caveat being that OpenMP-parallelized programs cannot execute on more than 1 computational node of a multi-node system.  Most modern computational cores typically contain between 16 and 64 cores.  Therefore, your GEOS-Chem "Classic" simulations will not be able to take advantage of more cores than these.  (We recommend that you consider using [[GEOS-Chem HP|GCHP]] for more computationally-intensive simulations.)
+
 
+
If your system uses the SLURM batch scheduler, then you can write your GEOS-Chem job script using the <tt>SLURM_CPUS_PER_TASK</tt> environment variable so that it will use the same number of cores as the number of cores you requested via SLURM.
+
 
+
#!/bin/bash
+
+
<span style="color:red">#SBATCH -c 24</span>
+
#SBATCH -N 1
+
#SBATCH -t 0-12:00
+
#SBATCH -p MY_QUEUE_NAME
+
#SBATCH --mem=15000
+
+
# Apply your environment settings to the computational queue
+
source ~/.bashrc
+
 
+
# Set the proper # of threads for OpenMP
+
# SLURM_CPUS_PER_TASK ensures this matches the number you set with -c above
+
#
+
# So in this example, we requested that SLURM make 24 cores available,
+
# and GEOS-Chem will use all of these 24 cores.
+
export OMP_NUM_THREADS=<span style="color:red">$SLURM_CPUS_PER_TASK</span>
+
+
... etc ...
+
 
+
'''''IMPORTANT!''''' If you forget to define <tt>OMP_NUM_THREADS</tt> in your Unix environment and/or run scripts, then GEOS-Chem will only execute using one core.  This can cause GEOS-Chem to execute much more slowly than intended.
+
 
+
=== OMP_STACKSIZE ===
+
 
+
In order to use GEOS-Chem "Classic" with [[Parallelizing GEOS-Chem|OpenMP parallelization]], you must request the maximum amount of stack memory in your Unix environment.  (The stack memory is where local automatic variables and temporary <code>!$OMP PRIVATE</code> variables will be created.)  Add the following lines to your system startup file and to your GEOS-Chem run scripts:
+
 
+
{| border=1 cellspacing=0 cellpadding=5
+
|- bgcolor="#CCCCCC"
+
!width="300px"|If you use bash.<br>add this to your .bashrc file
+
!width="300px"|If you use csh or tcsh,<br>add this to your .cshrc file
+
 
+
|-valign="top"
+
|<tt>ulimit -s unlimited<br>export OMP_STACKSIZE=500m</tt>
+
|<tt>limit stacksize unlimited<br>setenv OMP_STACKSIZE 500m</tt>
+
|}
+
 
+
The <tt>ulimit -s unlimited</tt> (for bash) or <tt>limit stacksize unlimited</tt> commands tell the Unix shell to use the maximum amount of stack memory available.
+
 
+
The environment variable <tt>OMP_STACKSIZE</tt> must also be set to a very large number.  In this example, we are nominally requesting 500 MB of memory.  But in practice, this will tell the GNU Fortran compiler to use the maximum amount of stack memory available on your system. The value <tt>500m</tt> is a good round number that is larger than the amount of stack memory on most computer clusters, but you can change this if you wish.
+
 
+
--[[User:Bmy|Bob Yantosca]] ([[User talk:Bmy|talk]]) 15:45, 17 June 2019 (UTC)
+
 
+
== Errors caused by incorrect environment variable settings ==
+
 
+
If you encounter any of the GEOS-Chem errors listed below, please doublecheck your environment variable settings as described in the preceding sections.
+
 
+
#[[Compile-time_warnings_and_errors#f77:_command_not_found|f77: command not found]]
+
 
+
--[[User:Bmy|Bob Yantosca]] ([[User talk:Bmy|talk]]) 15:12, 17 June 2019 (UTC)
+

Latest revision as of 18:40, 10 August 2022

This content has been migrated to the Customize your login environment chapter of geos-chem.readthedocs.io.