Difference between revisions of "Compiling GCHP"

Revision as of 18:15, 1 March 2019

Previous | Next | User Manual Home | GCHP Home

1. Hardware and Software Requirements
2. Downloading Source Code
3. Obtaining a Run Directory
4. Setting Up the GCHP Environment
5. Compiling
6. Basic Example Run
7. Configuring a Run
8. Output Data
9. Developing GCHP
10. Run Configuration Files

Overview

This section provides instructions on how to clean and compile GCHP. There are several options for cleaning and compiling which allow clean and compilation of subsets of modules, thereby minimizing recompilation to only modules you have modified.

You are ready to compile GCHP after you have obtained the GCHP source code, set up your run directory, and set your environment. First time compilation can take a while (30 minutes or more) because it requires building large modules, including ESMF, MAPL, FV3 dycore advection, and GEOS-Chem. MAPL, which is short for Modeling, Analysis and Prediction Layer, is built on ESMF, aka the Earth System Modeling Framework. Both systems are highly flexible but highly complex, with significant compilation times. Fortunately, MAPL and ESMF do not need to be recompiled unless parts of their code have been modified which is usually not necessary. Subsequent compile times will therefore be significantly shorter, on the same order of magnitude as compiling GEOS-Chem Classic.

Initial Compilation

To compile GCHP from scratch, navigate to your run directory and check that your gchp.env symbolic link is pointing to the environment file you wish to use. Next, type the following within the run directory:

make clean
make build

These command will execute ./build.sh clean and ./build.sh build. All build information is sent to both the terminal window and to a log file called compile.log. Git information for both source code repositories at compile time are recorded in a separate file called lastbuild for future reference. Upon successful compilation, the following empty files are created in your GCHP source code directories:

• GCHP/ESMF/esmf.install
• GCHP/Shared/mapl.install
• GCHP/FVdycoreCubed_GridComp/fvdycore.install

These files indicate successful compilation per module and their presence prompts the GCHP source code Makefile to skip compiling each component during future builds unless you explicitly clean the source code. If you are ever uncertain on whether your ESMF, MAPL, and FV3 dycore were previously build, check if these files exist.

Subsequent Compilation

For subsequent GCHP compilations you have several options. You can see all options by typing the following at the prompt from within your run directory:

make help

This will display all make options to your screen. For compilation the make targets you are interested in are clean, clean_mapl, clean_gc, build, build_mapl, and build_gc. Here is a brief summary of these options.

Clean/compile ALL source code, including ESMF, MAPL, and FV3:

make clean
make build

Clean/compile all source code except ESMF:

make clean_mapl
make build_mapl

Clean/compile all source code except ESMF, MAPL, and FV3:

make clean_gc
make build_gc

Same as above but compiling GEOS-Chem with debug flags turned on (note that this will slow down the model)

make build_gc_debug

A typical workflow would look like you getting a run directory, compiling from scratch (make clean; make compile), editing GEOS-Chem code in GeosCore, and then recompiling without cleaning (make build_gc). Since this workflow is the most common the default make target, meaning the target used when none is specified, is set to build_gc. Therefore to rebuild GEOS-Chem after changing GEOS-Chem files or any high-level files in the GCHP directory you can simply type this:

make

Note that if you change your underlying libraries set in your environment you MUST do a full source code clean before rebuilding. Otherwise you will likely encounter a segmentation fault when running the model. For this reason it is recommended that you create a new run directory for each new environment you want to run in, e.g. if you want to test with both Intel and GNU fortran compilers.

Also note that these build targets were added in GCHP version 12.2.0. If you are using an earlier version type make help to view your make options.

Checking for Successful Compilation

If GCHP compiled without issues, you should see:

###################################
### GCHP compiled successfully! ###
###################################

Compiling GCHP produces a GCHP executable called geos that is stored in the bin/ sub-directory of your GEOS-Chem source code directory. Compiling within the run directory makes a copy of the executable locally for easy use. This copy is what you should use when you run.

Compiler output, including warnings and error messages, will be stored in a log file called compile.log. If you inspect your log file or watch the compile output real-time, you will notice that there are many warnings and error messages. Many of these errors are due to leftover components in MAPL which are no longer used but which still try to compile. As long as the compilation continues, these errors can almost always be ignored and they will have no effect on the operation of GCHP.

Beware that once you have an executable you must not rename your run directory or your source code directory or the executable will no longer work. This is because the GCHP executable will include paths to ESMF libraries that incorporate the CodeDir symbolic link in the run directory path. However, you can copy the executable elsewhere for use. If you do this then it is also a good idea to also copy lastbuild and compile.log to your new run directory to keep track of where your executable came from. If you encounter an error while running GCHP that an ESMF shared library is missing, check that the exact path in the error message is still valid.

If you run into problems please contact the GEOS-Chem Support Team for help.

Using Debug Flags

After the initial compilation you may turn on debug flags for debugging subsequent compilations of GEOS-Chem code. Do this with the following make option:

make compile_debug

Be aware that turning on debug flags will slow down your run. Just as for GEOS-Chem Classic, you should always recompile GCHP without debug flags prior to doing production runs.

Additional Makefile Options

The GCHP run directory Makefile contains several options for cleaning and compiling GCHP as well as other functionalities. Options are printed out by typing the following at the command line:

make help

This will output something like this to your screen.

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%%%    GCHP Run Directory Makefile Options    %%%
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

Usage: make TARGET [OPTIONAL-FLAGS]

Default threads for compiling is hard-coded as 6

----------------------------------------------------------
TARGET may be one of the following:
----------------------------------------------------------

To print settings to screen:
  printbuildinfo   Print out git info for GC and GCHP repos
  help             If you got this far, you know this one

To clean source code:
  clean_all        Clean all source code using build.sh
  clean_gc         Clean GEOS-Chem classic only using build.sh
  clean_mapl       Clean MAPL and FVdyCore using build.sh
  clean_nuclear    Clean everything using build.sh
  superclean       Remove exe, logs, output, and clean ALL code

To compile:
  compile_clean     Include everything
  compile_standard  Skip MAPL, FvDyCore, and ESMF; debug off
  compile_debug     Skip MAPL, FvDyCore, and ESMF; debug on
  compile_mapl      Skip ESMF

To remove run directory files:
  cleanup_output   Remove output data, log files, and executable
  cleanup_data     Remove output data
  cleanup_logs     Remove log files
  cleanup_exe      Remove executable

To run interactively:
  run_interactive

To clean, build, and run interactively:
  gchp_clean       Include everything
  gchp_standard    Skip MAPL, FvDyCore, and ESMF; debug off
  gchp_debug       Skip MAPL, FvDyCore, and ESMF; debug on
  gchp_mapl        Skip ESMF

Some very useful options include the following:

• make printbuildinfo: prints git information such as branch and last commit for the GEOS-Chem and GCHP repositories your CodeDir symbolic link points to.
• make cleanup_output: cleans your run directory to remove all traces of your last run, including log and output files, but will not remove your executable or last build information.

Advanced Compilation Topics

The Makefile in the run directory invokes a bash shell script called build.sh that includes changing into the source code directory and invoking source code Makefiles. This section details the capabilities of build.sh and source code make options for advanced users who wish to better understand how GCHP works. You may skip this section and return to it at a later date when you want to dive deeper into how GCHP is compiled.

build.sh

Whenever build.sh is executed it first checks if you have set up your environment. If it is not then you will be prompted to do this. If you have multiple terminal windows and cannot remember if you have loaded your environment, a quick check is to execute the help option of build.sh.

./build.sh help

If you have set up your environment then clean and compile options of build.sh will print to screen. For example, when working on the Harvard Odyssey compute cluster you output will look as follows:

MPI_ROOT is to /n/sw/fasrcsw/apps/Comp/intel/15.0.0-fasrc01/mvapich2/2.2-fasrc01
Script name:
  build.sh
Arguments:
  Accepts single argument indicating clean and/or compile settings.
  Currently implemented arguments include:
     clean_gc         - classic only
     clean_nuclear    - GCHP, ESMF, MAPL, FVdycore (be careful!)
     clean_all        - classic, GCHP, ESMF, MAPL, FVdycore (be careful!)
     clean_mapl       - mapl and fvdycore only
     compile_debug    - turns on debug flags, no cleaning
     compile_standard - no cleaning
     compile_mapl     - includes fvdycore
     compile_clean    - cleans and compiles everything (be careful!)
Example usage:
  ./build.sh compile_standard

The default compile commands for compile_clean, compile_standard, and compile_mapl are defined in build.sh as:

   make -j${SLURM_NTASKS} NC_DIAG=y   CHEM=standard EXTERNAL_GRID=y   \
                          DEBUG=y     DEVEL=y       TRACEBACK=y       \
                          MET=geos-fp GRID=4x5      NO_REDUCED=y      \
                          UCX=n       hpc

The compile command for compile_debug is the same as above with BOUNDS=yes and FPEX=yes added.

Source Code Makefiles

At its core, building GCHP simply requires the inclusion of an additional option, HPC=yes, in the source code directory make command beyond what is typically included in GEOS-Chem compilation. Appending HPC=yes causes the make procedure to look for the contents of GCHP, and bypasses much of the serial GEOS-Chem code. For example, base GEOS-Chem classic source code is cleaned using the following command:

make HPC=yes realclean

After the process completes, the GCHP/ subdirectory is completely cleaned by changing into that directory and doing:

make EXTERNAL_GRID=yes the_nuclear_option

Calling make with the_nuclear_option defined in GCHP/Makefile removes all compiled code in the directory. As implied by the name, this is not to be used lightly! The nuclear option is an alias, calling three commands in turn: wipeout_esmf, wipeout_mapl, and wipeout_fvdycore.

If you are interested in seeing the core make commands used for all options provided in the run directory Makefile, see the commands defined in build.sh. You can then trace the build sequence by looking at the Makefiles in the GEOS-Chem and GCHP repositories. Note that calling make always invokes the Makefile in the directory you are in and changing into the GEOS-Chem and GCHP source code directories occurs within build.sh. Also note that the GRID=4x5 argument has no effect on the resolution at which the model runs since GCHP functions on the cubed-sphere grid. However, passing GRID is required so that components of GEOS-Chem Classic are functional.

Previous | Next | User Manual Home | GCHP Home