Difference between revisions of "Compiling GEOS-Chem"

Revision as of 17:48, 15 April 2015

On this page we list information about the makefiles used to build the GEOS-Chem executable. Please also see Chapter 3.3: GEOS-Chem Makefiles of the GEOS-Chem Online User's Guide.

Overview

Starting with GEOS-Chem v8-02-03, we have modified the directory structure of GEOS-Chem. Rather than keeping all source code files in a single directory, we now have partitioned source code files into several subdirectories with most subdirectories having their own Makefile. This was done for the following reasons:

1. To facilitate the installation of 3rd-party software packages such as:

• HEMCO
• RRTMG radiative transfer model
• KPP chemical solver
• Aerosol microphysics codes (e.g. APM)
• ISORROPIA II
• Terrestrial models (e.g. CASA, GTMM)

into GEOS-Chem. Our guiding principle is that all 3rd-party software packages should be cleanly separable from the main-line GEOS-Chem code. This will allow the 3rd-party software packages to be updated without having an impact on the rest of the GEOS-Chem source code files.

2. To simplify the maintenance of the GEOS–Chem code files. Without subdirectories, there would have been hundreds of source code files in a single directory, and it would have been very difficult to keep track of them all.

Directory Structure

Here we list the directory structure for recent model versions.

GEOS-Chem v10-01

The table below lists the directory structure in GEOS-Chem v10-01 along with descriptions of each subdirectory and its Makefile (if one is present).

--Lizzie Lundgren 13:46, 15 April 2015 (EDT)

GEOS-Chem v9-02

The table below lists the directory structure in GEOS-Chem v9-02 along with a description of each subdirectory and its Makefile (if one is present).

NOTE: Starting with GEOS-Chem v9-02, the GeosTomas subdirectory has been removed and code for the TOMAS aerosol microphysics package has been inlined into the GeosCore directory using C-preprocessor statements. See the TOMAS Aerosol Microphysics wiki page for more information.

--Lizzie Lundgren 13:47, 15 April 2015 (EDT)

Compilation sequence

GEOS-Chem's makefiles compile source code files in the following order:

1. Files in NcdfUtil/ (if GTMM is enabled for the mercury simulation)
2. Files in Headers/
3. Files in KPP/
4. Files in GeosUtil/
5. Files in HEMCO/Core/
6. Files in HEMCO/Extensions/
7. Files in HEMCO/Interfaces/
8. Files in ISOROPIA/
9. Files in GeosRad/ (if RRTMG is enabled)
10. Files in GeosCore/

Each directory has its own makefile. The makefile contains a list of all of the source code files in the subdirectory, plus dependent routines (a.k.a. the "dependencies" list). Source code files that do not refer to any modules have a dependencies listsuch as:

    decomp.o                    : decomp.F

Whereas a source code file that refers to several modules (in this case, GeosCore/aerosol_mod.F) has a dependencies list like this:

    aerosol_mod.o               : aerosol_mod.F                                  \
                                  chemgrid_mod.o          comode_mod.o           \
                                  dao_mod.o               diag_mod.o             \
                                  tracerid_mod.o          ucx_mod.o

Examination of GeosCore/aerosol_mod.F reveals that this module refers to many other modules not listed above. However, many of those modules are found in different directories (such as Headers/ and GeosUtil/), which are compiled prior to GeosCore/. Therefore you do not need to list those modules when creating the dependencies listing. In the example above, we have only specified the 6 modules that are located in the same directory as aerosol_mod.F.

The order of modules in the dependencies list does not matter. The GNU Make utility will define the order of compilation from the list of files (and their dependencies) that you have specified in each Makefile. You will not have to worry about this unless you add new source code files. If you have questions about how to add dependency lists to makefiles, please let us know.

--Melissa Sulprizio 12:54, 14 April 2015 (EDT)

Compiling GEOS-Chem

Makefile Options

To compile GEOS-Chem, you need to be in the top-level code directory. The GEOS-Chem makefiles allow you to choose from several different options. The best way to learn about these options is to type

make help

at the Unix prompt. You will then get a screen similar to this:

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%      GEOS-Chem Help Screen      %%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Usage: make -jN TARGET REQUIRED-FLAGS [ OPTIONAL-FLAGS ]

-jN             Compiles N files at a time (to reduce compilation time)

--------------------------------------------------------
TARGET may be one of the following:
--------------------------------------------------------
all             Default target (synonym for "lib exe")
lib             Builds GEOS-Chem source code
libcore         Builds GEOS-Chem objs & libs only in GeosCore/
libheaders      Builds GEOS-Chem objs & libs only in Headers/
libiso          Builds GEOS-Chem objs & libs only in ISOROPIA/
libkpp          Builds GEOS-Chem objs & libs only in KPP/
libnc           Builds GEOS-Chem objs & libs only in NcdfUtil/
librad          Builds GEOS-Chem objs & libs only in GeosRad/
libutil         Builds GEOS-Chem objs & libs only in GeosUtil/
ncdfcheck       Determines if the netCDF library installation works
exe             Creates GEOS-Chem executable
clean           Removes *.o, *.mod files in source code subdirs only
realclean       Removes all *.o, *mod, *.lib *.a, *.tex, *ps, *pdf files everywhere
distclean       Synonym for "make realclean"
doc             Builds GEOS-Chem documentation (*.ps, *.pdf) in doc/
docclean        Removes *.tex, *.pdf, *,ps from doc/
tauclean        Removes *.pdb, *.inst, *.pp, and *.continue.* files produced by TAU
help            Displays this help screen

Special targets for mercury simulation:
allhg           Default target for Hg simulation (synonym for "libhg exehg")
libhg           Builds GEOS-Chem code for Hg simulation
libgtmm         Builds GEOS-Chem + GTMM code for Hg simulation
exehg           Creates GEOS-Chem executable for Hg simulation

Special targets  for APM aerosol microphysics:
apm             Builds GEOS-Chem + APM (synonym for libapm exeapm)
libapm          Builds GEOS-Chem + APM objs & libs in GeosApm/ subdir
exeapm          Creates GEOS-Chem + APM executable in GeosApm/
cleanapm        Removes *.o *.mod files only in GeosApm/

--------------------------------------------------------
REQUIRED-FLAGS include:
--------------------------------------------------------
MET=____        Specifies the met field type
                --> Options: geosfp geos5 geos4 merra gcap
GRID=___        Specifies the horizontal grid
                --> Options: 4x5 2x25 05x0666 025x03125

--------------------------------------------------------
OPTIONAL-FLAGS may be one or more of the following:
--------------------------------------------------------
COMPILER=___    Specifies the compiler
                --> Options: ifort pgi (default is ifort)
NEST=___        Specifies the nested-grid domain
                 --> Options: CH NA EU SE
NO_REDUCED=yes  Compiles GEOS-Chem with the full vertical grid
                --> Default is to use the reduced vertical grid (i.e. NO_REDUCED=no)

Parallelization and optimization flags:
OMP=no          Turns off OpenMP parallelization (default is yes)
NONUMA=yes      Turns on -mp=nonuma option (pgi only, default is no)
IPO=yes         Turns on optmization options -ipo -static (ifort only, default is no)
OPT=___         Specifies the optimization level (default is -O2)

KPP chemistry solver flags:
CHEM=___        Specifies which chemistry mechanism is used
                --> Options: standard SOA UCX (default is standard)
KPPSOLVER=___   Specifies the integrator used with KPP
                --> Options: lsodes radau5 rosenbrock runge_kutta (default is rosenbrock)

Debug flags:
TRACEBACK=yes   Turns on -traceback option (ifort only, default is no)
DEBUG=yes       Turns on debugging options -g -O0 for running GEOS-Chem in a debugger (default is no)
BOUNDS=yes      Turns on subscript-array checking (default is no)
FPE=yes         Turns on checking for floating-point exceptions (default is no)

UCX trop-strat chemistry flags:
UCX=yes         Turns on the UCX chemistry mechanism and sets CHEM=UCX NO_REDUCED=yes (default is no)

RRTMG radiative transfer flags:
RRTMG=yes       Turns on online radiative transfer using the RRTMG model (default is no)

TOMAS aerosol microphysics flags:
TOMAS=yes       Turns on 30-bin TOMAS aerosol microphysics (default is no)
TOMAS12=yes     Turns on 12-bin TOMAS aerosol microphysics (default is no)
TOMAS15=yes     Turns on 15-bin TOMAS aerosol microphysics (default is no)
TOMAS40=yes     Turns on 40-bin TOMAS aerosol microphysics (default is no)

--Melissa Sulprizio 12:43, 8 April 2015 (EDT)

Specifying compiler flags

Please keep the following items in mind when specifying compiler flags.

• When starting with a new version of GEOS-Chem or a new simulation type you should always issue the following command:

     make realclean

This is especially important if you are changing the met field type or horizontal grid settings. The command will remove all previously-created compiler output files (e.g. *.a, *.o, *.mod) and executables. After doing make realclean, you can recompile GEOS–Chem again with your new options.

• You can speed up compilation by specifying make -j4 to compile 4 files at a time. If you have more CPUs available you can change this number.
• You must specify a value for MET. Please see out Overview of GMAO met data products wiki page for more information on the meteorology fields.
  • Allowable values are: geosfp, geos5, geos4, merra, gcap
• You must specify a value for GRID. Please see Appendix 2 of the GEOS-Chem User's Guide for more information on the horizontal grids.
  • Allowable values are: 4x5, 2x25, 05x0666, 025x03125
• If you select GRID=05x0666 or GRID=025x03125, then you must select a nested-grid option:
  • NEST=ch for the China nested grid
  • NEST=na for the North America nested grid
  • NEST=eu for the Europe nested grid
  • NEST=se for the SE Asia nested grid (GEOS-FP only)
• Makefile target names are case-sensitive. For example, you should type make all, not make ALL.
• Makefile flags will accept case-insensitive output. You may omit dashes from met field names. Each of the following is acceptable:
  • MET=geos5, MET=GEOS5, MET=Geos5, MET=geos-5, etc.
• Makefile flags that require a simple yes/no answer will accept case-insensitive input. For example:
  • DEBUG=yes, DEBUG=Yes, DEBUG=y, DEBUG=No, DEBUG=NO, etc.
• Some debug compiler flags may slow down your simulation, so we recommend using them for short test simulations only. For more information, see this wiki post.

--Melissa Sulprizio 13:00, 8 April 2015 (EDT)

Setting default flags

The REQUIRED-FLAGS and OPTIONAL-FLAGS can be specified in one of two ways:

1. As a command-line argument to the GNU Make utility

For example, if you wanted to build the GEOS-Chem executable using the KPP solver with Rosenbrock integrator and the SOA mechanism you could type:

       make -j4 MET=geosfp GRID=4x5 KPPSOLVER=rosenbrock CHEM=SOA

2. As an environment variable

If you don't wish to keep typing KPPSOLVER=rosenbrock CHEM=SOA every time, you could instead use:

       setenv KPPSOLVER rosenbrock
       setenv CHEM      SOA
       make -j4 MET=geosfp GRID=4x5

Specifying options as environment variables allows you to predefine settings so that you don't have to physically type them on the command line every time you build GEOS-Chem. The environment variables can also be placed in your ~/.cshrc or .bashrc so that they are initialized when you log in.

--Melissa Sulprizio 18:04, 7 April 2015 (EDT)

Compiling examples

(1) To build GEOS-Chem executable for the 4x5 grid with GEOS-FP meteorology, one can simply type:

   make -j4 MET=geosfp GRID=4x5

(2) To select a different compiler (e.g. PGI), type:

   make -j4 MET=geosfp GRID=4x5 COMPILER=pgi

• Depending on your PGI compiler setup, you may also have to add the option NONUMA=yes.

(3) The default is to compile for multi-processor. To turn off the parallelization and build the executable for a single-processor, type:

   make -j4 MET=geosfp GRID=4x5 OMP=no

(4) To compile GEOS-Chem (multi-processor) for use with a debugger (e.g. Totalview), type:

   make -j4 MET=geosfp GRID=4x5 DEBUG=yes FPE=yes OMP=yes

(5) To compile GEOS-Chem (single-processor) with the same debugging options, type:

   make -j4 MET=geosfp GRID=4x5 DEBUG=yes FPE=yes OMP=no

(6) To test for array-out-of-bounds errors, type:

   make -j4 MET=geosfp GRID=4x5 BOUNDS=yes

(7) If you wish to compile the code but not make the executable, you may type:

   make -j4 MET=geosfp GRID=4x5 BOUNDS=yes lib

(8) To print the GEOS-Chem help screen, type::

   make help

(9) To generate GEOS-Chem Reference Guide documentation in PostScript and PDF formats, type:

   make doc

(10) To remove all *.o, *.mod, and executable files from the source code directories (but not from the mod, lib, or bin directories), type:

   make clean

(11) And to remove everything and start over from scratch, type:

   make realclean

--Melissa Sulprizio 18:19, 7 April 2015 (EDT)

Advanced compilation techniques

Compiling with Unix shell scripts

It is possible to write a shell script to compile the GEOS–Chem code, such as the following:

   #!/bin/tcsh -f                       # Script definition line
   cd Code.v9-02                        # your code dir
   rm -f log                            # clear log file
   make -j4 MET=geos5 GRID=4x5 > log    # build the code
   exit(0)                              # exit normally

You can then run this script interactively, or submit it to the queue system on your computational cluster.

Compiling with a run-directory Makefile

The GNU Make utility is a powerful and flexible software development tool. With a little effort, you can design a Makefile that can be placed into your GEOS–Chem run directory that will create a GEOS–Chem executable and start a simulation. Here is an example:

# This makefile is a "router" makefile.  It calls the Makefile
# in the GEOS-Chem code directory (with various user-supplied options) 
# to build and run the GEOS-Chem model.

# Define variables
SHELL   = /bin/bash

# Make the development version "Code" the default directory
ifndef GEOSDIR
GEOSDIR = /home/bmy/Code.v10-01
endif

# Date string DDDHHMM
DATE = `date +%j%H%M`

#=============================================================================
# Makefile targets: type "make help" for a complete list!
#=============================================================================

.PHONY: all geos clean realclean fileclean doc docclean help logclean

all: geos

geos:
	@$(MAKE) -C $(GEOSDIR) 
	cp -f $(GEOSDIR)/bin/geos .
	@$(MAKE) logclean
 	./geos > log.$(DATE) &
	tail -f log.$(DATE)

clean:
	@$(MAKE) -C $(GEOSDIR) clean
	@$(MAKE) logclean

realclean:
	@$(MAKE) -C $(GEOSDIR) realclean
	@$(MAKE) fileclean
	rm -f geos

fileclean:
	@$(MAKE) logclean
	rm -f ctm.bpch*
	rm -f Ox.mass.*
	rm -f *2013080100*
	rm -f *.run.*o*

doc:
	@$(MAKE) -C $(GEOSDIR) doc

docclean: 
	@$(MAKE) -C $(GEOSDIR) docclean

help:
	@$(MAKE) -C $(GEOSDIR) help

logclean:
	rm -f *.log

At the heart of this Makefile are two important blocks of code, the first being:

# Make the development version "Code" the default directory
ifndef GEOSDIR
GEOSDIR = /home/bmy/Code.v10-01
endif

# Date string DDDHHMM
DATE = `date +%j%H%M`

This block defines the location of the GEOS–Chem source code directory, which is saved in a variable called GEOSDIR. If a value for GEOSDIR is not passed as a command-line argument, then GEOSDIR will be set to /home/bmy/Code.v10-01 (you can edit this location for your own setup). Also, a default date/time string is saved in the variable DATE. This date/time string will be appended to the GEOS–Chem log file output.

The second block of code is used to compile and run the GEOS–Chem code:

geos:
	@$(MAKE) -C $(GEOSDIR) 
	cp -f $(GEOSDIR)/bin/geos .
	@$(MAKE) logclean
 	./geos > log.$(DATE) &
	tail -f log.$(DATE)

The lines:

	@$(MAKE) -C $(GEOSDIR) 
	cp -f $(GEOSDIR)/bin/geos .

tell GNU Make to call down to the router makefile in the GEOS–Chem source code directory /home/bmy/Code.v10-01. GNU Make will compile GEOS–Chem and create the executable file /home/bmy/Code.v10-01/bin/geos. This file is then copied to the run directory.

The next 3 lines:

	@$(MAKE) logclean
	./geos > log.$(DATE) &
	tail -f log.$(DATE)

removes the log file if it already exists, runs the GEOS–Chem executable (thus starting the simulation), and follows the log file output with the Unix tail -f command.

--Melissa Sulprizio 13:46, 8 April 2015 (EDT)

Technical Issues

GNU Make is required

The makefiles described above are designed to be used with GNU Make. This is free, open-source software and is bundled with most of the popular Unix versions today (e.g. various Linux builds, Ubuntu, Fedora, etc.).

It is recommended to use GNU Make because not all Unix make utilities are compatible with each other. GNU Make is probably the most flexible make utility available.

If GNU Make is not already installed on your system, then you (or your IT guru) will have to install it. You can check to see if GNU Make is already installed by typing:

make --version

at the Unix prompt. GNU Make will display a version screen similar to this:

GNU Make 3.81
Copyright (C) 2006  Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.

This program built for x86_64-redhat-linux-gnu

Please also read the GNU Make Reference Document for more detailed information.

Install LaTeX utilities for auto documentation

Many of the updated GEOS-Chem source code files now use the ProTeX documentation headers. The protex script (included in the Code/doc subdirectory) strips the information from the documentation headers located in GEOS-Chem

• Modules
• Subroutines
• Functions
• Include files

and creates a LaTeX format (*.tex) file. The latex utility can then be used to produce output in PDF (*.pdf) and PostScript (*.ps) formats from the LaTeX file. This all happens automatically when you type:

make doc

at the Unix prompt. However, you must make sure to have the LaTeX utilities installed on your machine in order to be able to generate this automatic documentation. The LaTeX utilities include:

latex  

Converts LaTeX format files to Device-Independent format (*.dvi)

dvips  

Converts Device-Independent format to PostScript format

dvipdf 

Converts Device-Independent format to PDF format

The LaTeX utilities may be packaged with your version of Unix. Ask your IT guru for more information.

You may also want to install the following packages:

ghostview 

Reader for PostScript files

acroread  

Adobe Acrobat Reader for PDF files

--Bob Y. 09:48, 24 September 2009 (EDT)

Backup files not excluded from compilation

Maria Zatko wrote:

During compilation using version v8-02-04, the compilation cannot be completed because the .f~ file type is not recognized. The model runs successfully once I manually remove the .f~ files from the code.

Is there a way to run v8-02-04 while keeping the backup files in the code as done in v8-03-01 and v8-02-01?

Bob Yantosca wrote:

Yes...I think in some v8-02-xx versions of the code we may have had a wildcard that grepped for *.f* in the Makefile. This would have tried to compile the *.f~ files that are generated when you use Emacs to edit the code.

In the GeosCore/Makefile you can replace any lines that look like:

   # Source and object files
   SRC   = $(wildcard *.F*)

with

   # Source and object files
   SRC   = $(wildcard *.F) $(wildcard *.F90)

and this should exclude all *.f~ and *.f90~ files from the compilation. This fix has since been standardized in GEOS-Chem v8-03-01.

--Bob Y. 13:58, 27 August 2010 (EDT)

Compile-time options that can slow down GEOS-Chem

The following compile-time options can cause a GEOS-Chem job to run more slowly than usual:

DEBUG=yes

This is required for running GEOS-Chem in a debugger such as idb or Totalview. This will invoke the -g -O0 compiler options, which turn off all optimization.

BOUNDS=yes

This option turns on runtime array-out-of-bounds checking, which looks for instances of invalid array indices (i.e. If the A array only has 10 elements but you try to reference A(11).)

FPE=yes

This option turns on error checking for floating-point exceptions (i.e. div-by-zero, NaN, floating-invalid, and similar errors).

OMP=no

This option turns off the OpenMP parallelization (which is turned on by default).

If you have selected any of the above options, then try compiling GEOS-Chem from scratch, i.e.,

make realclean
make -j4 MET=____ GRID=____

and then re-run your GEOS-Chem job.

--Bob Y. 15:47, 2 October 2013 (EDT)

Known issues

Compiler cannot find certain files

This issue can occur in GEOS-Chem v9-01-01 and older versions. We have added this fix into GEOS-Chem v9-01-02 and higher versions.

Matthew Johnson wrote:

When looking into the core files I get error output such as this:

   Symbol file not found for main
   Core was generated by `./geos'.
   Program terminated with signal 11, Segmentation fault.
   #0  0x0000000000733bde in pjc_pfix_mod_mp_init_press_fix_ ()

Have you seen this portion of the code causing a seg fault before?

Bob Yantosca replied:

Try reordering the lines in GeosCore/Makefile from:

   lib: libkpp libutil libiso libcore

to:

   lib:                                               # Build all G-C libraries                  
        @$(MAKE) libkpp                        
        @$(MAKE) libutil                  
        @$(MAKE) libiso                   
        @$(MAKE) libcore                  

NOTE: All indented lines begin with a TAB!

This ordering will force GNU Make to compile all of the files in KPP/ first, then all of the files in GeosUtil/, etc. This ought to prevent the build sequence from referencing files that haven't yet been compiled.

--Bob Y. 14:02, 3 January 2013 (EST)

References

1. GNU Make Manual
2. Mecklenburg, Robert, Managing Projects with GNU Make, Third Edition,, O'Reilly and Associates, 2004.

--Bob Y. 12:07, 19 July 2011 (EDT)