|
|
(48 intermediate revisions by 2 users not shown) |
Line 1: |
Line 1: |
− | This page contains information about the Intel Fortran Compiler (aka "IFORT" compiler). The IFORT compiler is the preferred compiler for use with GEOS-Chem.
| + | '''''[[GNU Fortran compiler|Previous]] | [[Fortran language resources|Next]] | [[Guide to compilers for GEOS-Chem]]''''' |
| | | |
− | == Documentation == | + | #[[Supported compilers for GEOS-Chem]] |
| + | #[[GNU Fortran compiler|The GNU Fortran compiler (gfortran)]] |
| + | #<span style="color:blue">'''The Intel Fortran compiler (ifort, ifx)'''</span> |
| + | #[[Fortran language resources]] |
| | | |
− | NOTE: Recent Intel compiler versions (e.g. IFORT 12, IFORT 13) are now packaged and sold under the name ''Intel Fortran Composer XE'' (or something similar). Many GEOS-Chem users still use older versions (i.e. IFORT 11, IFORT 10).
| |
| | | |
− | You can find more information about the Intel Fortran Compiler here:
| + | This page contains information about the Intel Fortran compiler, which has recently been renamed from <tt>ifort</tt> to <tt>ifx</tt>. |
| | | |
− | #[http://acmg.seas.harvard.edu/geos/wiki_docs/compilers/main_for_lin_v11.pdf Version 11.1]
| + | '''''The Intel Fortran compiler is our recommended proprietary compiler for GEOS-Chem.''''' |
− | #[http://acmg.seas.harvard.edu/geos/wiki_docs/compilers/main_for_lin_v12.pdf Version 12.1]
| + | |
− | #[https://software.intel.com/en-us/compiler_15.0_ug_f Version 15.0]
| + | |
− | #[http://software.intel.com/en-us/articles/determining-root-cause-of-sigsegv-or-sigbus-errors/ Determining the cause of SIGSEGV or SIGBUS errors]
| + | |
− | #[http://software.intel.com/en-us/blogs/author/steve-lionel Dr. Fortran's Blog]
| + | |
| | | |
− | Also, normally when you installs the Intel Fortran compilers, you also will install the C and C++ compilers. These compilers are not needed for GEOS-Chem, but they will be needed if you [[Installing libraries for GEOS-Chem|install libraries (e.g. netCDF or HDF5)]] on your system.
| + | == Overview == |
| | | |
− | --[[User:Bmy|Bob Yantosca]] ([[User talk:Bmy|talk]]) 18:19, 3 November 2016 (UTC)
| + | === Documentation === |
| | | |
− | == Environment settings for Intel Fortran ==
| + | You can find more information about the Intel Fortran Compiler here: |
| | | |
− | Here is some information about how you can customize your Unix environment to use the Intel Fortran compiler.
| + | #[https://software.intel.com/en-us/fortran-compiler-developer-guide-and-reference|Intel Fortran 19 documentaton] |
| + | #[http://acmg.seas.harvard.edu/geos/wiki_docs/compilers/PDF_Fortran_Compiler_UG_17_0.pdf Intel Fortran 17] |
| | | |
− | === Using the module command to load Intel Fortran and related libraries ===
| + | Also, normally when you installs the Intel Fortran compilers, you also will install the C and C++ compilers. |
| | | |
− | On most Unix clusters, the <tt>module</tt> command is used to load the Intel Fortran compiler library into your Unix enviroment, along with any related libraries (e.g. netCDF, OpenMPI). For example, we use these commands on the Harvard cluster (odyssey.rc.fas.harvard.edu):
| + | --[[User:Bmy|Bob Yantosca]] ([[User talk:Bmy|talk]]) 19:44, 10 January 2019 (UTC) |
| | | |
− | # These commands load Intel Fortran 15
| + | === Intel Fortran Compiler versions that have been tested with GEOS-Chem === |
− | module load intel/15.0.0-fasrc01 openmpi/1.8.3-fasrc02 netcdf/4.1.3-fasrc04
| + | |
− |
| + | |
− | # These commands load Intel Fortran 13
| + | |
− | module load intel/13.0.079-fasrc01 openmpi/1.8.1-fasrc05 netcdf/4.1.3-fasrc06
| + | |
− |
| + | |
− | # These commands load Intel Fortran 11
| + | |
− | module load intel/11.1 GEOS-Chem-Libraries
| + | |
| | | |
− | You can ask your IT staff what the corresponding commands would be on your particular cluster.
| + | The [https://geoschem.github.io/support-team GEOS-Chem Support Team] has tested GEOS-Chem with the compiler versions listed below. But you should be able to use other Intel Fortran Compiler versions as well. |
| | | |
− | The <tt>module</tt> command should automatically define several environment variables for you:
| + | {| border=1 cellspacing=0 cellpadding=5 |
− | | + | |
− | {| border=1 cellspacing=0 cellpadding=5 | + | |
| |- bgcolor="#CCCCCC" | | |- bgcolor="#CCCCCC" |
− | !width="150px"|Variable | + | !width="100px"|Platform |
− | !width="150px"|Expected setting | + | !width="200px"|Compiler |
− | !width="600px"|Description | + | !width="450px"|Status |
| | | |
| |-valign="top" | | |-valign="top" |
− | |<tt>FC</tt> | + | |Linux |
− | |<tt>ifort</tt> | + | |ifort 23.0.0 |
− | |Name of the Intel Fortran compiler | + | |Supported |
| | | |
| |-valign="top" | | |-valign="top" |
− | |<tt>CC</tt> | + | |Linux |
− | |<tt>icc</tt> | + | |ifort 19.0.5.281 |
− | |Name of the Intel C compiler | + | |Supported |
| | | |
| |-valign="top" | | |-valign="top" |
− | |<tt>CXX</tt> | + | |Linux |
− | |<tt>icpc</tt> | + | |ifort 18.0.5 |
− | |Name of the Intel C++ compiler | + | |Supported |
| | | |
| |-valign="top" | | |-valign="top" |
− | |<tt>NETCDF_HOME</tt> | + | |Linux |
− | |System-dependent | + | |ifort 17.0.4 |
− | |Path to the root netCDF folder | + | |Supported |
| | | |
| |-valign="top" | | |-valign="top" |
− | |<tt>NETCDF_INCLUDE</tt> | + | |Linux |
− | |System-dependent | + | |ifort 15.0.0 and similar builds |
− | |Path to the netCDF include folder (e.g. <tt>$NETCDF_HOME/include</tt>) | + | |Supported |
| + | *NOTE: IFORT 15 has a compiler bug that causes errors when turning on array-out-of-bounds checking and optimization. |
| | | |
| |-valign="top" | | |-valign="top" |
− | |<tt>NETCDF_LIB</tt> | + | |Linux |
− | |System-dependent | + | |ifort 13.0.079 and similar builds |
− | |Path to the netCDF library folder (e.g. <tt>$NETCDF_HOME/lib</tt> or <tt>$NETCDF_HOME/lib64</tt>)
| + | |Supported |
| | | |
| |-valign="top" | | |-valign="top" |
− | |<tt>NETCDF_FORTRAN_HOME</tt> | + | |Linux |
− | |System-dependent | + | |ifort 12 |
− | |Path to the root netCDF Fortran folder | + | |Supported |
| + | *NOTE: A|compiler bug in ifort 12 and higher versions]] has forced us to add a workaround to HEMCO in v11-01. |
| | | |
| |-valign="top" | | |-valign="top" |
− | |<tt>NETCDF_FORTRAN_INCLUDE</tt> | + | |Linux |
− | |System-dependent | + | |ifort 11.1.069 and similar builds |
− | |Path to the netCDF Fortran include folder (e.g. <tt>$NETCDF_FORTRAN_HOME/include</tt>)
| + | |Supported |
− | | + | |
− | |-valign="top" | + | |
− | |<tt>NETCDF_FORTRAN_LIB</tt>
| + | |
− | |System-dependent
| + | |
− | |Path to the netCDF Fortran library folder (e.g. <tt>$NETCDF_FORTRAN_HOME/lib</tt> or <tt>$NETCDF_FORTRAN_HOME/lib64</tt>)
| + | |
| | | |
| |} | | |} |
| | | |
− | If these variables are not automatically set by the module command on your system (or if your system does not use the module command), then:
| + | == Environment settings for Intel Fortran == |
| | | |
− | *Set the <tt>FC</tt>, <tt>CC</tt>, and <tt>CXX</tt> variables manually in your startup script (e.g. <tt>.bashrc</tt> or <tt>.cshrc</tt>).
| + | Here is some information about how you can customize your Unix environment to use the Intel Fortran compiler. |
− | *Ask your IT staff where the netCDF library paths are located, and set the <tt>NETCDF_HOME</tt>, <tt>NETCDF_INCLUDE</tt>, and <tt>NETCDF_LIB</tt> environment variables accordingly.
| + | |
− | | + | |
− | Depending on your system, you may find that the netCDF Fortran libraries are installed into a different folder than the rest of the netCDF library files. If this is the case, then the <tt>module</tt> command should automatically define the <tt>NETCDF_FORTRAN_HOME</tt>, <tt>NETCDF_FORTRAN_INCLUDE</tt>, and <tt>NETCDF_FORTRAN_LIB</tt> environment variables. If not, then ask your IT staff what the proper paths are so that you can set these variables manually.
| + | |
− | | + | |
− | --[[User:Bmy|Bob Yantosca]] ([[User talk:Bmy|talk]]) 19:34, 4 October 2016 (UTC)
| + | |
− | | + | |
− | === Requesting sufficient stack memory for GEOS-Chem ===
| + | |
− | | + | |
− | In order to run GEOS-Chem with Intel Fortran, you must request the maximum amount of stack memory in your Unix environment. (The stack memory is where local automatic variables and temporary <code>!$OMP PRIVATE</code> variables will be created.) Add the following lines to your system startup file:
| + | |
− | | + | |
− | {| border=1 cellspacing=0 cellpadding=5
| + | |
− | |- bgcolor="#CCCCCC"
| + | |
− | !width="300px"|If you use bash.<br>add this to your .bashrc file
| + | |
− | !width="300px"|If you use csh or tcsh,<br>add this to your .cshrc file
| + | |
− | | + | |
− | |-valign="top"
| + | |
− | |
| + | |
− | ulimit -s unlimited
| + | |
− | export OMP_STACKSIZE=500m
| + | |
− | |
| + | |
− | limit stacksize unlimited
| + | |
− | setenv OMP_STACKSIZE 500m
| + | |
− | |}
| + | |
− | | + | |
− | The <tt>ulimit -s unlimited</tt> (for bash) or <tt>limit stacksize unlimited</tt> commands tell the Unix shell to use the maximum amount of stack memory available.
| + | |
− | | + | |
− | The environment variable <tt>OMP_STACKSIZE</tt> must also be set to a very large number. In this example, we are nominally requesting 500 MB of memory. But in practice, this will tell the GNU Fortran compiler to use the maximum amount of stack memory available on your system. The value <tt>500m</tt> is a good round number that is larger than the amount of stack memory on most computer clusters, but you can change this if you wish.
| + | |
− | | + | |
− | <span style="color:red">'''NOTE: Setting the <tt>OMP_STACKSIZE</tt> environment variable will make it easier to switch between different compilers on your system. The <tt>KMP_STACKSIZE</tt> environment variable only works with the Intel Fortran Compiler but not with GNU Fortran.'''</span>
| + | |
− | | + | |
− | --[[User:Bmy|Bob Yantosca]] ([[User talk:Bmy|talk]]) 21:35, 4 October 2016 (UTC)
| + | |
− | | + | |
− | == Performance ==
| + | |
− | | + | |
− | === 7-day time test results ===
| + | |
− | | + | |
− | Please see [[GEOS-Chem_performance#7-day_time_tests|this list of timing results]] done with [[GEOS-Chem v10-01]] and [[GEOS-Chem v11-01]].
| + | |
− | | + | |
− | === Timing results: IFORT 11 vs. IFORT 10 ===
| + | |
− | | + | |
− | [[Image:Obsolete.jpg]]
| + | |
− | | + | |
− | <span style="color:red">'''''The GEOS-Chem versions listed below are obsolete. Please see [[GEOS-Chem_performance#7-day_time_tests|this list of timing results]] done with more recent versions of GEOS-Chem.'''''</span>
| + | |
− | | + | |
− | ==== Description ====
| + | |
− | | + | |
− | The table below shows the wallclock time and mean OH for several GEOS-Chem simulations that were done in order to compare IFORT 10.1.013 vs. IFORT 11.1.069. The simulations had all these things in common:
| + | |
− | | + | |
− | # GEOS-Chem v8-02-04
| + | |
− | # 4x5 GEOS-5 met fields for month of 2008/07
| + | |
− | # 1-month of simulation (0 GMT 2005/07/01 to 0 GMT 2005/08/01)
| + | |
− | # Base compiler options: <tt>-cpp -w -O2 -auto -noalign -convert big_endian</tt>
| + | |
− | # KPP compiler turned on
| + | |
− | # Linoz stratospheric chemistry turned on
| + | |
− | # All simulations ran on virtual machines (kvm guests) with the following characteristics:
| + | |
− | | + | |
− | Machine Type x86_64
| + | |
− | Operating System Linux
| + | |
− | Operating System Release 2.6.18-128.1.6.el5_lustre.1.8.0
| + | |
− | CPU Count 8 CPUs
| + | |
− | CPU Speed 2659 MHz
| + | |
− | Memory Total 12008076.000 KB
| + | |
− | Swap Space Total 18415600.000 KB
| + | |
− | | + | |
− | ==== Results ====
| + | |
− | | + | |
− | {| border=1 cellpadding=5 cellspacing=0
| + | |
− | |- bgcolor="#CCCCCC" align="center"
| + | |
− | ! Run
| + | |
− | ! IFORT<br>version
| + | |
− | ! # CPUs
| + | |
− | ! Wall clock<br>(mm:ss)
| + | |
− | ! Parallel %
| + | |
− | ! Mean OH<br>(1e5 molec/cm3)
| + | |
− | |- align="center"
| + | |
− | |1
| + | |
− | |10.1.013
| + | |
− | |4
| + | |
− | |02:10:51
| + | |
− | |384.7%
| + | |
− | |12.5205894678448
| + | |
− | |- align="center" bgcolor="#CCFFFF"
| + | |
− | |2
| + | |
− | |11.1.069
| + | |
− | |4
| + | |
− | |02:09:14
| + | |
− | |382.7%
| + | |
− | |12.5217430768752
| + | |
− | |-align="center"
| + | |
− | |3
| + | |
− | |10.1.013
| + | |
− | |8
| + | |
− | |01:17:13
| + | |
− | |757.1%
| + | |
− | |12.5205894678448
| + | |
− | |- align="center" bgcolor="#CCFFFF"
| + | |
− | |4
| + | |
− | |11.1.069
| + | |
− | |8
| + | |
− | |01:18:44
| + | |
− | |753.4%
| + | |
− | |12.5213489686705
| + | |
− | |}
| + | |
− | | + | |
− | Here are some plots of surface ozone from the benchmarks simulations:
| + | |
− | | + | |
− | *[http://www.geos-chem.org/wiki_docs/machines/ifort11_4p_diff.gif IFORT 11 - IFORT 10 (4 processors)]
| + | |
− | *[http://www.geos-chem.org/wiki_docs/machines/ifort11_4p_diff.gif IFORT 11 - IFORT 10 (8 processors)]
| + | |
− | | + | |
− | NOTES:
| + | |
− | | + | |
− | # The wall times and parallel % are more or less identical when moving from IFORT 10 to IFORT 11.
| + | |
− | # The ideal parallelization percentages are 400% (on 4p) and 800% (on 8p).
| + | |
− | # The differences in the surface ozone is approximately between IFORT 10 and IFORT 11 is due to numerical differences in the libraries and optimization. The absolute magnitude of the differences is approximately 1 ppt of ozone.
| + | |
− | | + | |
− | --[[User:Bmy|Bob Y.]] 16:08, 25 March 2010 (EDT)
| + | |
− | | + | |
− | === Comparison between IFORT 9.1 and IFORT 10.1 ===
| + | |
− | | + | |
− | [[Image:Obsolete.jpg]]
| + | |
− | | + | |
− | <span style="color:red">'''''The GEOS-Chem versions listed below are obsolete. Please see [[GEOS-Chem_performance#7-day_time_tests|this list of timing results]] done with more recent versions of GEOS-Chem.'''''</span>
| + | |
− | | + | |
− | The table shows the wallclock time and mean OH for several GEOS-Chem simulations that were done in order to compare Intel Fortran Compiler (IFORT) v9.1 vs. v10.1.013. The simulations had all these things in common:
| + | |
− | | + | |
− | * GEOS-Chem v8-01-01
| + | |
− | * 4x5 GEOS-5 met fields
| + | |
− | * 1-week of simulation (0 GMT 2008/01/01 to 0 GMT 2008/01/08)
| + | |
− | * Base compiler options: <tt>-cpp -w -auto -noalign -convert big_endian</tt>
| + | |
− | * Runs were done on the Harvard "Ceres" cluster (OS type <tt>"linux-rhel5-x86_64"</tt>)
| + | |
− | | + | |
− | {| border=1 cellpadding=5 cellspacing=0
| + | |
− | |- bgcolor="#CCCCCC" align="center"
| + | |
− | ! Run
| + | |
− | ! IFORT<br>version
| + | |
− | ! # CPUs
| + | |
− | ! Optimization options
| + | |
− | ! Wall clock<br>(mm:ss)
| + | |
− | ! Speedup from<br>IFORT 9.1 to<br>IFORT 10.1
| + | |
− | ! Speedup from<br>4 to 8 CPUs w/<br>the same compiler
| + | |
− | ! Mean OH<br>(1e5 molec/cm3)
| + | |
− | |- align="center"
| + | |
− | | 1
| + | |
− | | 9.1
| + | |
− | | 4
| + | |
− | | -O2
| + | |
− | | 36:16
| + | |
− | |
| + | |
− | |
| + | |
− | | 11.2913755849576
| + | |
− | |- align="center" bgcolor="#CCFFFF"
| + | |
− | | 2
| + | |
− | | 10.1
| + | |
− | | 4
| + | |
− | | -O2
| + | |
− | | 33:55
| + | |
− | | 6.48%
| + | |
− | |
| + | |
− | | 11.2913755842197
| + | |
− | |- align="center"
| + | |
− | | 3
| + | |
− | | 9.1
| + | |
− | | 4
| + | |
− | | -O3
| + | |
− | | 37:26
| + | |
− | |
| + | |
− | |
| + | |
− | | 11.2913755849576
| + | |
− | |- align="center" bgcolor="#CCFFFF"
| + | |
− | | 4
| + | |
− | | 10.1
| + | |
− | | 4
| + | |
− | | -O3
| + | |
− | | 33:36
| + | |
− | | 10.24%
| + | |
− | |
| + | |
− | | 11.2913755838124
| + | |
− | |- align="center"
| + | |
− | | 5
| + | |
− | | 9.1
| + | |
− | | 8
| + | |
− | | -O2
| + | |
− | | 24:15
| + | |
− | |
| + | |
− | | 33.13%
| + | |
− | | 11.2913755849576
| + | |
− | |- align="center" bgcolor="#CCFFFF"
| + | |
− | | 6
| + | |
− | | 10.1
| + | |
− | | 8
| + | |
− | | -O2
| + | |
− | | 22:46
| + | |
− | | 6.12%
| + | |
− | | 32.88%
| + | |
− | | 11.2913755842197
| + | |
− | |- align="center"
| + | |
− | | 7
| + | |
− | | 9.1
| + | |
− | | 8
| + | |
− | | -O3
| + | |
− | | 23:36
| + | |
− | |
| + | |
− | | 36.95%
| + | |
− | | 11.2913755849576
| + | |
− | |- align="center" bgcolor="#CCFFFF"
| + | |
− | | 8
| + | |
− | | 10.1
| + | |
− | | 8
| + | |
− | | -O3
| + | |
− | | 22:31
| + | |
− | | 4.59%
| + | |
− | | 32.99%
| + | |
− | | 11.2913755838124
| + | |
− | |- align="center"
| + | |
− | | 9
| + | |
− | | 9.1
| + | |
− | | 8
| + | |
− | | -O3 -ipo -no-prec-div -static
| + | |
− | | 23:03
| + | |
− | |
| + | |
− | |
| + | |
− | | 11.2913764967223
| + | |
− | |- align="center" bgcolor="#CCFFFF"
| + | |
− | | 10
| + | |
− | | 10.1
| + | |
− | | 8
| + | |
− | | -O3 -ipo -no-prec-div -static
| + | |
− | | 21:56
| + | |
− | | 4.84%
| + | |
− | |
| + | |
− | | 11.0809209646817
| + | |
− | |}
| + | |
− | | + | |
− | NOTES about the table:
| + | |
− | # The column '''Speedup from IFORT 9.1 to IFORT 10.1''' compares the wall clock time of equivalent runs done with IFORT 9.1 and IFORT 10.1. For example, the 6.48% speedup listed for Run #2 is comparing Run #2 to Run #1. Similarly Run #4 is compared against Run #3, etc.
| + | |
− | # The column '''Speedup from 4 to 8 CPUs w/ the same compiler''' compares the wall clock time between runs with 4 CPUs and 8 CPUs for the same compiler (i.e. 4 CPUs on IFORT 9 vs 8 CPUs on IFORT 9, and ditto for IFORT 10). For example, the 33.13% speedup listed for Run #5 is comparing Run #5 to Run #1. Similarly, Run #6 is compared against Run #2, etc.
| + | |
− | # The compiler options <tt>-O3 -ipo -non-prec-div -static</tt> correspond to IFORT's <tt>-fast</tt> optimization option. Using this option results in a mean OH concentration that is different than with the simpler optimization options of -O2 and -O3. This is because the <tt>-fast</tt> option sacrifices numerical accuracy for speed.
| + | |
− | # With IFORT 9.1, switching from -O2 to -O3 does not change the mean OH concentration. Thus the bpch files of the runs were binary identical to each other.
| + | |
− | # With IFORT 10.1, switching from -O2 to -O3 changes the mean OH concentration slightly. This implies that there are slight differences in the chemistry. However all runs done with -O2 have the same mean OH, as do all runs done with -O3.
| + | |
− | | + | |
− | PLOTS:
| + | |
− | # [http://www.as.harvard.edu/ctm/geos/wiki_docs/machines/4p_O2_v9_v10.gif Run #2 vs Run #1 (i.e. IFORT 9.1 vs IFORT 10.1 w/ -O2 on 4 CPUs)]
| + | |
− | # [http://www.as.harvard.edu/ctm/geos/wiki_docs/machines/4p_O3_v9_v10.gif Run #3 vs Run #4 (i.e. IFORT 9.1 vs IFORT 10.1 w/ -O3 on 4 CPUs)]
| + | |
− | # [http://www.as.harvard.edu/ctm/geos/wiki_docs/machines/v9_8p_O2_fast.gif Run #5 vs Run #9 (i.e. -O2 vs -fast with IFORT 9.1)]
| + | |
− | # [http://www.as.harvard.edu/ctm/geos/wiki_docs/machines/v10_8p_O2_fast.gif Run #6 vs Run #10 (i.e. -O2 vs -fast with IFORT 10.1)]
| + | |
− | | + | |
− | TAKE-HOME MESSAGE:
| + | |
− | # IFORT 10.1 is '''always''' faster than the equivalent run with IFORT 9.1.
| + | |
− | #* IFORT 10.1 does indeed seem to optimize code better on machines with multi-core chipsets.
| + | |
− | #* For example: Run #6 (w/ IFORT 10) is 89 seconds faster per week than Run #5 (w/ IFORT 9) on 8 CPUs. This implies that a 52-week simulation with IFORT 10 on 8 CPUs would finish ~1hr 15m earlier than the equivalent IFORT 9 run.
| + | |
− | # Switching from 4 to 8 CPU's results in a ~33% speedup for both IFORT 9.1 and IFORT 10.1.
| + | |
− | # In general, switching from -O2 to -O3 (while using the same # of CPU's) does not result in a significant speedup. This is true for both IFORT 9.1 and IFORT 10.1.
| + | |
− | | + | |
− | OUR RECOMMENDATIONS:
| + | |
− | # If possible, use IFORT 10.1 instead of IFORT 9.1
| + | |
− | # Use the following compiler options (see Makefile.ifort):
| + | |
− | #*<tt>FFLAGS = -cpp -w -O2 -auto -noalign -convert big_endian</tt>
| + | |
− | | + | |
− | --[[User:Bmy|Bob Y.]] 16:46, 16 April 2008 (EDT)
| + | |
| | | |
− | Upgrading to IFORT 10.1 does not seem to fix the stacksize problem listed below. You still need to manually reset the stacksize limit to a large positive number for both [[#Resetting stacksize for Linux|Linux]] and [[#Resetting stacksize for Altix|Altix]] platforms.
| + | Here is some information about how you can customize your Linux environment to use the GNU Fortran compiler. This information was recently migrated to our [https://geos-chem.readthedocs.io geos-chem.readthedocs.io] manual. |
| | | |
− | --[[User:Bmy|Bob Y.]] 12:41, 25 April 2008 (EDT) | + | * [https://geos-chem.readthedocs.io/en/latest/getting-started/login-env-files-intel.html Create an environment file for Intel compilers] |
| + | * [https://geos-chem.readthedocs.io/en/latest/getting-started/login-env-compilers.html Set environment variables for compilers] |
| + | * [https://geos-chem.readthedocs.io/en/latest/getting-started/login-env-parallel.html Set environment variables for parallelization] |
| | | |
| == Optimization == | | == Optimization == |
Line 364: |
Line 94: |
| === Optimization options === | | === Optimization options === |
| | | |
− | Here is a quick reference table of IFORT's optimization options (taken from the online [http://software.intel.com/sites/products/documentation/hpc/compilerpro/en-us/fortran/lin/compiler_f/index.htm Intel Fortran Compiler User and Reference Guides]. | + | Here is a quick reference table of optimization options (taken from the online [http://software.intel.com/sites/products/documentation/hpc/compilerpro/en-us/fortran/lin/compiler_f/index.htm Intel Fortran Compiler User and Reference Guides]. |
| | | |
| {| border=1 cellspacing=0 cellpadding=5 | | {| border=1 cellspacing=0 cellpadding=5 |
Line 707: |
Line 437: |
| | | |
| --[[User:Bmy|Bob Yantosca]] ([[User talk:Bmy|talk]]) 14:50, 28 March 2017 (UTC) | | --[[User:Bmy|Bob Yantosca]] ([[User talk:Bmy|talk]]) 14:50, 28 March 2017 (UTC) |
− |
| |
− | == Known issues ==
| |
− |
| |
− | The following issues occur with certain versions of the Intel Fortran Compiler. Several of these issues have now been resolved in the most recent GEOS-Chem versions. We recommend upgrading to the most recent public release if possible.
| |
− |
| |
− | === Cannot compile GEOS-Chem v10-01 with Intel Fortran Compiler v17 ===
| |
− |
| |
− | <span style="color:green">'''''This issue is now resolved in [[GEOS-Chem v11-01]].'''''</span>
| |
− |
| |
− | '''''Myroslav Hordiichuk wrote:'''''
| |
− |
| |
− | <blockquote>When compiling with <tt>make -j2 MET=geosfp GRID=4x5 UCX=y CHEM=benchmark</tt> there is a problem, I can't fix. The screenshots are attached.
| |
− |
| |
− | [[Image:Ifort 17 error.png]]
| |
− |
| |
− | [[Image:Ifort 17 error2.png]]
| |
− | </blockquote>
| |
− |
| |
− | '''''[[User:Bmy|Bob Yantosca]] wrote:'''''
| |
− |
| |
− | <blockquote>Thanks for writing. [Because...] you [are] using a very new version of the Intel compiler (v2017)... then that may be not able to parse the module interfaces in <tt>NcdfUtil/ncdf_mod.F90</tt> and in <tt>HEMCO/Core/hco_diagn_mod.F90</tt>. We recently encountered this issue when trying to port [[GEOS-Chem v11-01]] to the [[GNU Fortran compiler]]. I ended up rewriting code in these modules to avoid this issue.
| |
− |
| |
− | What I think is happening is that, like GNU Fortran, Intel Fortran 2017 is by default using the newer Fortran 2003 or Fortran 2008 standard. This standard is more strict than the Fortran-90 language standard when it comes to the module interfaces. Long story short: I had to rewrite the <code>DIAGN_UPDATE</code> interface in <tt>HEMCO/Core/hco_diagn_mod.F90</tt> to remove the <code>OPTIONAL</code> array arguments. This got the code to compile with GNU Fortran. Basically, instead of having only 2 subroutines contained in the <code>DIAGN_UPDATE</code> module interface, I had to have 6. The arguments <code>Scalar</code>, <code>Array2D</code>, <code>Array3D</code> cannot be <code>OPTIONAL</code> arguments in this case. This syntax used to be OK in the original Fortran-90 standard but evidently not in the newer F2003 or F2008 standards. We also had to modify a similar module interface in <tt>NcdfUtil/ncdf_mod.F90</tt> accordingly.
| |
− |
| |
− | If you go to [[GNU_Fortran_compiler#Modifications_made_to_GEOS-Chem_for_GNU_Fortran|this wiki link]], you will see several updates that we had to make to GEOS-Chem v11-01 in order to get it to work with GNU Fortran in the various sub-folders of the GC source code. These updates are already included in our v11-01 development code, and thus should let v11-01 (when it is released) to compile with Intel Fortran 2017 as well.</blockquote>
| |
− |
| |
− | --[[User:Bmy|Bob Yantosca]] ([[User talk:Bmy|talk]]) 20:47, 20 January 2017 (UTC)
| |
− |
| |
− | === Resetting stacksize for Linux ===
| |
− |
| |
− | Overall machine memory limits are set with the Unix <tt>limit</tt> command. If you use <tt>csh</tt> or <tt>tcsh</tt>, you can set the following commands in your <tt>~/.cshrc</tt> file:
| |
− |
| |
− | # Max out machine limits
| |
− | limit cputime unlimited # NOTE: "Unlimited" is not truly unlimited. It
| |
− | limit filesize unlimited # will set the given limit to the maximum value
| |
− | limit datasize unlimited # determined by your hardware configuration
| |
− | limit stacksize unlimited
| |
− | limit coredumpsize unlimited
| |
− | limit memoryuse unlimited
| |
− | limit vmemoryuse unlimited
| |
− | limit descriptors unlimited
| |
− | limit memorylocked unlimited
| |
− | limit maxproc unlimited
| |
− |
| |
− | Or if you use <tt>bash</tt>, you can add these to commands your <tt>~/.bashrc</tt> file:
| |
− |
| |
− | # Max out machine limits
| |
− | ulimit -t unlimited # cputime
| |
− | ulimit -f unlimited # filesize
| |
− | ulimit -d unlimited # datasize
| |
− | ulimit -s unlimited # stacksize
| |
− | ulimit -c unlimited # coredumpsize
| |
− | ulimit -m unlimited # memoryuse
| |
− | ulimit -v unlimited # vmemoryuse
| |
− | ulimit -n unlimited # descriptors
| |
− | ulimit -l unlimited # memorylocked
| |
− | ulimit -u unlimited # maxproc
| |
− |
| |
− | ''NOTE: depending on your particular OS build (Linux, CentOS, Fedora, Ubuntu), not all of these limits will be used.''
| |
− |
| |
− | It is important to set the stacksize memory to the maximum value, because this will determine the amount of memory available for temporary variables, which are:
| |
− |
| |
− | * variables that are not declared with the SAVE attribute
| |
− | * variables that are not located at the top of a module
| |
− | * variables that are not located in a common block.
| |
− |
| |
− | However, one quirk is that the stacksize memory for child processes (i.e. processes spawned by CPUs within <code>!$OMP PARALLEL DO</code> loops) are not set by the <tt>stacksize</tt> limit, but instead by the <tt>OMP_STACKSIZE</tt> environment variable. If <tt>OMP_STACKSIZE</tt> is not set with a high enough value, then your GEOS-Chem simulation may think it doesn't have enough memory to proceed, and may die with a [[Common GEOS-Chem error messages#Segmentation faults|segmentation fault error]].
| |
− |
| |
− | The fix for this situation is to make sure that you set <tt>OMP_STACKSIZE</tt> to a high value. It's OK if the value you give to <tt>OMP_STACKSIZE</tt> is larger than the largest amount of memory on your system. As long as it's set to a high positive number it will work.
| |
− |
| |
− | If you use <tt>csh</tt> or <tt>tcsh</tt>, you can add this command to your <tt>~/.cshrc</tt> file.
| |
− |
| |
− | # Reset the child stack size to a large positive number
| |
− | # (It's OK if this is larger than the max value, it's just a kludge)
| |
− | setenv OMP_STACKSIZE 500m
| |
− |
| |
− | Or if you use bash, add this command to your <tt>~/.bashrc</tt> file:
| |
− |
| |
− | # Reset the child stack size to a large positive number
| |
− | # (It's OK if this is larger than the max value, it's just a kludge)
| |
− | export OMP_STACKSIZE=500m
| |
− |
| |
− | Resetting the <tt>OMP_STACKSIZE</tt> environment variable in this manner usually will correct the following errors:
| |
− |
| |
− | *[[Common GEOS-Chem error_messages#Segmentation fault encountered after TPCORE initialization|Segmentation fault encountered after TPCORE initialization]]
| |
− | *[[Common GEOS-Chem error_messages#Stack overflow|Stack overflow]]
| |
− | *[[Common GEOS-Chem error_messages#forrtl: error .2876.29: IOT trap signal|forrtl: error (76): IOT trap signal]]
| |
− |
| |
− | <span style="color:red">'''NOTE: We now recommend that you use the <tt>OMP_STACKSIZE</tt> variable instead of the <tt>KMP_STACKSIZE</tt> variable. This will make it easier to switch between compilers on your system. <tt>OMP_STACKSIZE</tt> works with all compilers, but <tt>KMP_STACKSIZE</tt> only works with Intel Fortran.'''</span>
| |
− |
| |
− | --[[User:Bmy|Bob Y.]] 14:05, 5 November 2014 (EST)
| |
− |
| |
− | === Out of memory asking for NNNNN===
| |
− |
| |
− | '''''[mailto:dkweis@seas.harvard.edu Debra Weisenstein] wrote:'''''
| |
− |
| |
− | :I'm trying to compile GEOS-CHEM (the stratospheric beta version that Seb Eastham at MIT is working on) on hpc at Harvard, and though it compiled for me early before, I've now been getting an "out of memory" error every time it tries to compile <tt>GeosCore/diag49_mod.F</tt>. I changed the compile option for mcmodel to large. Here is the compile line and error.
| |
− |
| |
− | ifort -cpp -w -O2 -auto -noalign -convert big_endian -vec-report0
| |
− | -mcmodel=large -i-dynamic -fp-model source -openmp -Dmultitask
| |
− | -I../Headers -module ../mod -I/home/dkweis/include -c diag49_mod.F
| |
− | Fatal compilation error: Out of memory asking for 36864.
| |
− |
| |
− | :I've gone back and tried compiling older versions that I've compiled and run before and get a similar error:
| |
− |
| |
− | ifort -cpp -w -O2 -auto -noalign -convert big_endian -vec-report0
| |
− | -fp-model source -openmp -Dmultitask -DAPM
| |
− | -I../Headers -module ../mod -c dao_mod.F
| |
− | Fatal compilation error: Out of memory asking for 20480.
| |
− |
| |
− | :Any idea what is going on? The "top" command shows 301780k of free memory (out of 8GB) and 906164k of free swap.
| |
− |
| |
− | '''''[mailto:yantosca@seas.harvard.edu Bob Yantosca] wrote:'''''
| |
− |
| |
− | :I googled the error message (“Out of memory asking for ….”) and I found [http://software.intel.com/en-us/forums/topic/268149 this post online]. The person who replied to this post gave 3 suggestions:
| |
− |
| |
− | ::#Use <tt>-O2</tt>.
| |
− | ::#Split up the source into smaller files if possible (may be difficult if it's a module.)
| |
− | ::#Raise your datasize limit
| |
− | ::I'd start with the last. If you can't resolve it, please submit a test case to Intel Premier Support.
| |
− |
| |
− | :So it may be that your datasize limit is not maxed out in your computational environment. You can check the setting of the system limits by typing:
| |
− |
| |
− | limit
| |
− |
| |
− | :at the Unix prompt. You’ll get a screen like this:
| |
− |
| |
− | [54 bmy]% limit
| |
− | datasize unlimited
| |
− | stacksize unlimited
| |
− | ... etc ...
| |
− |
| |
− | :Depending on your platform (i.e. combination of hardware + operating system) you may not have all of these limits available. The two most important limits for GEOS-Chem are <tt>datasize</tt> and <tt>stacksize</tt>. These should be set to <tt>unlimited</tt>. For good measure, you can also set the other available limits for your platform to unlimited as well. Follow the [[#Resetting_stacksize_for_Linux|instructions in this wiki post]]. One caveat: if you try to set a limit that does not exist on your platform to unlimited, you will get a warning message from the Unix shell. You can then delete the offending limit setting from your .cshrc or .bashrc file.
| |
− |
| |
− | --[[User:Bmy|Bob Y.]] 14:58, 25 July 2013 (EDT)
| |
− |
| |
− | === Bug fix for GEOS-Chem compiled with Intel Fortran Compiler 12 ===
| |
− |
| |
− | '''''[mailto:psk9@duke.edu Prasad Kasibhatla] wrote:'''''
| |
− |
| |
− | :I am experiencing the [[Intel Fortran Compiler|IFORT 12]] compile failure of [[GEOS-Chem v9-01-03]] with nested grid options turned on in <tt>define.h</tt>. The compile seems to be failing in <tt>strat_chem_mod.F90</tt> (see end of this msg). And to add - the compile succeeds for [[GEOS-Chem_v9-01-03_benchmark_history#v9-01-03l|v9-01-03l]] with IFORT 12. I noticed that <tt>strat_chem_mod.F90</tt> appeared post-v9-01-03l.
| |
− |
| |
− | ifort -cpp -w -O2 -auto -noalign -convert big_endian -vec-report0 -mcmodel=medium -i-dynamic
| |
− | -fp-model source -openmp -Dmultitask -I../Headers -module ../mod -I/opt/geos-netcdf-4/include
| |
− | -c -free strat_chem_mod.F90
| |
− | : catastrophic error: **Internal compiler error: segmentation violation signal raised**
| |
− | Please report this error along with the circumstances in which it occurred in a Software Problem Report.
| |
− | Note: File and line given may not be explicit cause of this error.
| |
− | compilation aborted for strat_chem_mod.F90 (code 1)
| |
− | make[3]: *** [strat_chem_mod.o] Error 1
| |
− | make[3]: Leaving directory `/nfs/fire/psk9/Code.v9-01-03/GeosCore'
| |
− | make[2]: *** [lib] Error 2
| |
− | make[2]: Leaving directory `/nfs/fire/psk9/Code.v9-01-03/GeosCore'
| |
− | make[1]: *** [all] Error 2
| |
− | make[1]: Leaving directory `/nfs/fire/psk9/Code.v9-01-03/GeosCore'
| |
− | make: *** [all] Error 2
| |
− |
| |
− | '''''[mailto:yantosca@seas.harvard.edu Bob Yantosca] wrote:'''''
| |
− |
| |
− | :I have this version of IFORT installed locally:
| |
− |
| |
− | [67 bmy Code.v9-02]% ifort -V
| |
− | Intel(R) Fortran Intel(R) 64 Compiler XE for applications running on Intel(R) 64, Version 12.1.3.293 Build 20120212
| |
− | Copyright (C) 1985-2012 Intel Corporation. All rights reserved.
| |
− | FOR NON-COMMERCIAL USE ONLY
| |
− |
| |
− | :And I got the exact same error as you did. I looked on the internet and it seemed that the issue is related to how IFORT 12 deals w/ OpenMP DO loops (click [http://software.intel.com/en-us/forums/topic/271372 HERE] and [http://software.intel.com/en-us/forums/topic/271164 HERE]).
| |
− |
| |
− | :A little brute-force debugging revealed the offending parallel DO loop in <tt>strat_chem_mod.F90</tt>, in routine <tt>CALC_STE</tt>:
| |
− |
| |
− | ! Determine mean tropopause level for the period
| |
− | !$OMP PARALLEL DO &
| |
− | !$OMP DEFAULT( SHARED ) &
| |
− | !$OMP PRIVATE( I, J )
| |
− | DO J = 1,JJPAR
| |
− | DO I = 1,IIPAR
| |
− | LTP(I,J) = NINT( TPauseL(I,J) / TPauseL_Cnt )
| |
− | ENDDO
| |
− | ENDDO
| |
− | !$OMP END PARALLEL DO
| |
− |
| |
− | :I also noticed that this DO loop is only executed for global simulations. Immediately preceding it there is an <tt>#if</tt> statement that exits out of the routine if you are doing an nested grid run:
| |
− |
| |
− | #if defined( NESTED_NA ) || defined( NESTED_CH ) || defined( NESTED_EU )
| |
− | ! This method only works for a global domain.
| |
− | ! It could be modified for nested domains if the total mass flux across the
| |
− | ! boundaries during the period is taken into account.
| |
− | RETURN
| |
− | #endif
| |
− |
| |
− | :So given that this code doesn’t get used for nested runs, I added an #else block so that the code isn’t compiled in when you compile for nested grids:
| |
− |
| |
− | #if defined( NESTED_NA ) || defined( NESTED_CH ) || defined( NESTED_EU )
| |
− | ! This method only works for a global domain.
| |
− | ! It could be modified for nested domains if the total mass flux across the
| |
− | ! boundaries during the period is taken into account.
| |
− | RETURN
| |
− | !------------------------------------------------------------------------------
| |
− | ! Prior to 10/5/12:
| |
− | ! Since the rest of this code isn't needed for the nested grid, wrap it
| |
− | ! in an #else statement. This might help the code to compile for IFORT 12.
| |
− | !#endif
| |
− | !------------------------------------------------------------------------------
| |
− | #else
| |
− |
| |
− | ! Determine mean tropopause level for the period
| |
− | !$OMP PARALLEL DO &
| |
− | !$OMP DEFAULT( SHARED ) &
| |
− | !$OMP PRIVATE( I, J )
| |
− | DO J = 1,JJPAR
| |
− | DO I = 1,IIPAR
| |
− | LTP(I,J) = NINT( TPauseL(I,J) / TPauseL_Cnt )
| |
− | ENDDO
| |
− | ENDDO
| |
− | !$OMP END PARALLEL DO
| |
− |
| |
− | ... etc ...
| |
− |
| |
− | #endif
| |
− |
| |
− | END SUBROUTINE Calc_STE
| |
− |
| |
− | :I’ve compiled this with the NA nested grid, the CH nested grid, and for 2 x 2.5 and 4 x 5 global GEOS-5 simulations, and the code does not get that compilation error. I’ve pushed this to the GEOS-Chem repository as a last-minute bug fix. I’m not sure why the error happens in the first place w/ the IFORT 12 compiler, but in any case, this fixes it.
| |
− |
| |
− | --[[User:Bmy|Bob Y.]] 13:52, 5 October 2012 (EDT)
| |
− |
| |
− | === Error compiling with IFORT 12 and Mac OS X ===
| |
− |
| |
− | '''''[mailto:djl101000@utdallas.edu David Lary] wrote:'''''
| |
− |
| |
− | :I am using Intel(R) 64, Version 12.1.3.289 Build 20120130 on Mac OS X 10.7.4. When I typed <tt>make</tt>, I get the following error:
| |
− |
| |
− | ld: library not found for -lcrt1.10.6.o
| |
− | make[2]: *** [exe] Error 1
| |
− | make[1]: *** [all] Error 2
| |
− | make: *** [all] Error 2
| |
− |
| |
− | '''''[mailto:yantosca@seas.harvard.edu Bob Yantosca] replied:'''''
| |
− |
| |
− | :I think this is an issue particular to IFORT running on Mac OS X. It is probably not finding the proper library. I found a couple of posts ([http://lists.apple.com/archives/xcode-users/2007/Oct/msg00696.html post #1] and [http://lists.apple.com/archives/xcode-users/2007/Oct/msg00689.html post #2]) on the internet that describe this error.
| |
− |
| |
− | :Post #2 recommends adding the following path to your <tt>LD_LIBRARY_PATH</tt> environment variable:
| |
− |
| |
− | /Developer/SDKs/MacOSX10.4u.sdk/usr/lib.
| |
− | (i.e. $(SDKROOT)/usr/lib)
| |
− |
| |
− | --[[User:Bmy|Bob Y.]] 10:43, 26 June 2012 (EDT)
| |
− |
| |
− | === Compilation error with IFORT 12 ===
| |
− |
| |
− | '''''[mailto:geertvinken@hotmail.com Geert Vinken] wrote:'''
| |
− |
| |
− | :I was trying to compile the [[GEOS-Chem v9-01-03 benchmark history#v9-01-03k|GEOS-Chem v9-01-03k]] version, but I was getting a strange error when trying to do this with ifort 12.1.2:
| |
− |
| |
− | biofuel_mod.F(363): error #5082: Syntax error, found IDENTIFIER 'X1' when expecting one of: ( % [ : . = =>
| |
− | GENERIC8_1x1 = GENERIC_1x1
| |
− | ----------------------------^
| |
− | biofuel_mod.F(363): error #6404: This name does not have a type, and must have an explicit type. [GENERIC8_1]
| |
− | GENERIC8_1x1 = GENERIC_1x1
| |
− | ------------------^
| |
− | biofuel_mod.F(363): error #6460: This is not a field name that is defined in the encompassing structure. [X1]
| |
− | GENERIC8_1x1 = GENERIC_1x1
| |
− | ----------------------------^
| |
− | biofuel_mod.F(363): error #6366: The shapes of the array expressions do not conform. [X1]
| |
− | GENERIC8_1x1 = GENERIC_1x1
| |
− | ----------------------------^
| |
− | compilation aborted for biofuel_mod.F (code 1)
| |
− |
| |
− | :Compiling with ifort 11.1 shows no problems at this module. Any of you ever heard of this problem or know a way around it? If I uncomment this line the model compiles, so it's just this line (and apparently the combination of 8_1x1 =) that's causing the problem.
| |
− |
| |
− | :Also, renaming the <tt>GENERIC8_1x1</tt> array to <tt>GENERIC8</tt> seems to solve the problem.
| |
− |
| |
− | '''''[mailto:yantosca@seas.harvard.edu Bob Yantosca] replied:'''
| |
− |
| |
− | :I think the default behavior of IFORT 12.1 has changed to adopt one of the more modern Fortran standards (F2003 or F2008). I found [http://scc.ustc.edu.cn/zlsc/sugon/intel/compiler_f/main_for/hh_goto.htm#lref_for/source_files/rfcrealr.htm this entry in the Intel Fortran Compiler 12.1 manual] which describes how one specifies floating-point constants.
| |
− |
| |
− | :As shown in the entry above, the IFORT 12 compiler now interprets an underscore immediately preceded and followed by numbers as an alternate way to specify numbers in scientific notation. For example, the traditional way of representing a <tt>REAL*4</tt> and <tt>REAL*8</tt> constants:
| |
− |
| |
− | REAL*4, PARAMETER :: PI = 3.14159e0
| |
− | REAL*8, PARAMETER :: PI = 3.141592658979323d0
| |
− |
| |
− | :can now also be represented as:
| |
− |
| |
− | REAL*4, PARAMETER :: PI = 3.14159e0_4
| |
− | REAL*8, PARAMETER :: PI = 3.141592658979323e0_8
| |
− |
| |
− | :Therefore, the string:
| |
− |
| |
− | GENERIC8_1x1
| |
− |
| |
− | :is probably now parsed by the compiler as an exponential. This more than likely separates the string into substrings <tt>GENERIC8_1x1</tt> and <tt>x1</tt>, which is what generates the error.
| |
− |
| |
− | :I have looked for a compiler switch [http://scc.ustc.edu.cn/zlsc/sugon/intel/compiler_f/main_for/index.htm in the IFORT 12.1 manual] that would disable this feature, but have not been successful. The short-term solution is to rename the variables so that it does not have an underscore surrounded by two numbers (e.g. from <tt>GENERIC8_1x1</tt> to <tt>GENERIC_8_1x1</tt>).
| |
− |
| |
− | --[[User:Bmy|Bob Y.]] 18:13, 23 May 2012 (EDT)
| |
− |
| |
− | === Relocation truncated to fit error ===
| |
− |
| |
− | If your code uses many large arrays, or if you are compiling an ultra-fine resolution version of GEOS-Chem (e.g. a 0.25° x 0.3125° [[GEOS-FP]] nested grid), then you may see this type of error:
| |
− |
| |
− | Relocation truncated to fit: R_X86_64_32S against `.bss' Error"
| |
− |
| |
− | The wording you get may differ slightly than the example shown above.
| |
− |
| |
− | Long story short: IFORT is telling you that your program is trying to use more than 2GB of statically-allocated data (i.e. data space that is not declared with an ALLOCATABLE statement) at compile time. The default setting in IFORT is to expect to use less than 2GB of memory, so you are hitting the upper limit.
| |
− |
| |
− | The solution is simple: recompile your code with the following compiler flags:
| |
− |
| |
− | -mcmodel=medium -shared-intel
| |
− |
| |
− | The <tt>-mcmodel=medium</tt> flag will tell IFORT that you expect to use more than 2GB of statically-allocated memory in your program. However, this also requires that you use link using dynamic libraries instead of the normal shared libraries. Using the <tt>-shared-intel</tt> flag will turn on the dynamic library linking. (Starting with [[GEOS-Chem v9-01-03]], these compiler flags will be applied to the build sequence automatically.)
| |
− |
| |
− | '''''IMPORTANT NOTE!''''' If your code links to any libraries such as HDF or netCDF, then you MUST rebuild each library, making sure that the Fortran and C compilers use the <tt>-mcmodel=medium</tt> option. Please see our [[Installing libraries for GEOS-Chem]] page for examples.
| |
− |
| |
− | [[GEOS-Chem v9-01-03]] and higher will automatically set these flags for you.
| |
− |
| |
− | For more information, please see the following links:
| |
− |
| |
− | # [http://amiatypist.blogspot.com/2010/05/relocation-truncated-to-fit-rx866432s.html Typist vs. Programmer blog]
| |
− | # [http://mitgcm.org/pipermail/mitgcm-support/2005-October/003505.html MITGCM support blog]
| |
− | # [http://software.intel.com/en-us/forums/showthread.php?t=43717 Software.intel.com blog]
| |
− |
| |
− | --[[User:Bmy|Bob Y.]] ([[User talk:Bmy|talk]]) 18:24, 25 September 2015 (UTC)
| |
− |
| |
− | === Problems with IFORT 11.0.xxx ===
| |
− |
| |
− | You should use GEOS-Chem with IFORT 11.1.058 or higher versions. Please see the discussion below about problems in the earlier versions of IFORT 11.0.xxx:
| |
− |
| |
− | '''''[mailto:May.Fu@polyu.edu.hk Tzung-May Fu] wrote:'''''
| |
− |
| |
− | :I tested the Intel Fortran v11.0.074 compiler, but found that it is incompatible with the GC code. This is related to the [[Bugs and fixes#Error message in partition.f|<tt>partition.f</tt> bug that I reported earlier]]. (Actually, I'm not sure there is a bug in partition.f any more, unless you have also run into it with IFORT v10).
| |
− |
| |
− | :I ran a 1-day simulation, using Bob's v8-01-03 standard run release, with no change at all. Using Intel Fortran v10.1.015, I was able to replicate Bob's standard run. However, when I switched to Intel Fortran v11.0.074, I ran into the error in partition.f, due to the CONCNOX-SUM1 < 0d0 check. Here's the error message in log:
| |
− |
| |
− | ===============================
| |
− | GEOS-CHEM ERROR: STOP 30000
| |
− | STOP at partition.f
| |
− | ===============================
| |
− |
| |
− | :I then tried [[Bugs and fixes#Error message in partition.f|Bob's fix to partition.f]]. This time the run finishes, warning the user about the CONCNOX-SUM1 < 0d0 issue. But the output result is completely wacky!!! Below you can compare the surface Ox concentrations, using
| |
− |
| |
− | :* (A) [http://www.as.harvard.edu/ctm/geos/wiki_docs/machines/A_Ox_sfc_20050701_ifort10.gif IFORT v10]
| |
− | :* (B) [http://www.as.harvard.edu/ctm/geos/wiki_docs/machines/B_Ox_sfc_20050701_ifort11_partition_fix.gif IFORT v11 and the <tt>partition.f</tt> fix]
| |
− |
| |
− | :The (B) spatial pattern is completely off. NOx is also affected and shows the similar weird pattern.
| |
− |
| |
− | :I'm pretty sure the problem is in the chemistry part. I've tried turning off the optimization but the problem persists. Perhaps there is some problem with the way IFORTv11 treats floating points? Also, I am not sure if IFORTv11 caused the weird model result, or if IFORTv11 caused some issues in chemistry, and the <tt>partition.f</tt> 'fix' subsequently lead to the weird result.
| |
− |
| |
− | :Long story short, it seems like IFORTv11 is not a good choice for now, and that the 'fix' to partition.f should not be implemented.
| |
− |
| |
− | '''''[mailto:plesager@seas.harvard.eud Philippe Le Sager] wrote:'''''
| |
− |
| |
− | :Thanks for testing Ifort11. We did run into the partition bug with Ifort10 after fixing tpcore. So I doubt that the weird result is related to that partition fix, and it is probably just a problem with IFORT 11.
| |
− |
| |
− | '''''[mailto:yantosca@seas.harvard.edu Bob Yantosca] wrote:'''''
| |
− |
| |
− | :You might have to go thru the IFORT 11 manuals to see if any default behavior has changed (i.e. optimization, compiler options, etc). It may not just be the concnox thing but something else in the numerics that is particular to IFORT 11.
| |
− |
| |
− | :There is usually a "What's new" document w/ every Intel compiler release. Maybe that has some more information, you could look at it.
| |
− |
| |
− | '''''[mailto:yantosca@seas.harvard.edu Bob Yantosca] wrote:'''''
| |
− |
| |
− | :I've also heard from some folks @ NASA that IFORT 11.0 was problematic. They claim that IFORT 11.1 is much better. You may want to look into this in the meantime.
| |
− |
| |
− | --[[User:Bmy|Bob Y.]] 16:50, 7 October 2009 (EDT)
| |
− |
| |
− | '''''[mailto:esofen@atmos.washington.edu Eric Sofen] wrote:'''''
| |
− |
| |
− | :Both Becky Alexander and I have run into problems with IFORT 11.1. When either of us run offline aerosol simulations compiled on IFORT 11.1, the simulation compiles and runs without errors, but the sulfur budgets are way off. The problems seem to be occurring in the deposition code, as Becky's simulations end up with very little deposition, but at the same time, the S burdens are too low. In my case, the deposition ends up being an order of magnitude too high. Changing back to IFORT 10 fixed both of these problems.
| |
− |
| |
− | --[[User:Esofen|Eric Sofen]] 13:32, 22 October 2009
| |
− |
| |
− | '''''[mailto:yxw@mail.tsinghua.edu.cn Yuxuan Wang] wrote:'''''
| |
− |
| |
− | :From our interaction with the Intel people, <tt>ifort 11.1.056</tt> should work for GEOS-Chem. The GC version we tested at Tsinghua is v8-02-01 (nested-grid China with GEOS-5 meteorology). The platform we tested is Nehalem from Intel, with the following compilation options:
| |
− |
| |
− | -cpp -w -static -fno-alias -O2 -safe_cray_ptr -no-prec-sqrt -no-prec-div -auto -noalign -convert big_endian
| |
− |
| |
− | :Not sure whether these options will work for Mac OSX. From the testing, we found that codes compiled with <tt>ifort 11.1.056</tt> ran at 2% faster than <tt>ifort 10.1.008</tt>.
| |
− |
| |
− | --[[User:Bmy|Bob Y.]] 14:59, 4 November 2009 (EST)
| |
− |
| |
− | === Problem with IFORT 11 and GEOS-Chem adjoint ===
| |
− |
| |
− | '''''[mailto:N.Bousserez@dal.ca Nicolas Bousserez] wrote:'''''
| |
− |
| |
− | :We have been struggling for some time with the following problem when running GC adjoint (v8-02-01):
| |
− |
| |
− | "OMP abort: Initializing libguide.so,
| |
− |
| |
− | :but found libguide.so already initialized".
| |
− |
| |
− | :After some investigations it seems like it is a linker error generated when different parts of the program try to link both static and dynamic verions of the OpenMP runtime. There is an option in ifort 11 to have openmp linked statically, which theoretically should fix this problem.
| |
− |
| |
− | :But using ifort 11 for GC seems to cause other problems and this compilation option doesn't exist with ifort 10. The fact is that Daven Henze, who is using ifort 10 and a linux platform similar to ours never got the above problem. Has anyone got this error before? My platform configuration is the following:
| |
− |
| |
− | Linux node9 2.6.9-89.0.23.ELsmp #1 SMP Wed Mar 17 06:49:21 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux
| |
− |
| |
− | :If anyone running GC adjoint has a similar configuration please let me know what is your Makefile (using netcdf libraries) configuration and which version of ifort you're using so that I can do some testing.
| |
− |
| |
− | --[[User:Bmy|Bob Y.]] 09:43, 8 April 2011 (EDT)
| |
− |
| |
− | === Incompatibility between IFORT 11 and OS version ===
| |
− |
| |
− | If you are using a the Intel Fortran Compiler version 11, you may encounter some incompatibilities with your operating system, which might require an OS upgrade.
| |
− |
| |
− | '''''[mailto:N.Bousserez@dal.ca Nicolas Bousserez] wrote:'''''
| |
− |
| |
− | :We have been struggling for some time with the following problem when running GC adjoint (v8-02-01). We get this error:
| |
− |
| |
− | "OMP abort: Initializing libguide.so, but we but found libguide.so already initialized".
| |
− |
| |
− | :After some investigations it seems like it is a linker error generated when different parts of the program try to link both static and dynamic verions of the OpenMP runtime. There is an option in ifort 11 to have openmp linked statically, which theoretically should fix this problem. But using ifort 11 for GC seems to cause other problems and this compilation option doesn't exist with ifort 10. The fact is that Daven Henze, who is using ifort 10 and a linux platform similar to ours never got the above problem. Has anyone got this error before? My platform configuration is the following:
| |
− |
| |
− | Linux node9 2.6.9-89.0.23.ELsmp #1 SMP Wed Mar 17 06:49:21 EDT 2010
| |
− | x86_64 x86_64 x86_64 GNU/Linux
| |
− |
| |
− | '''''[mailto:N.Bousserez@dal.ca Nicolas Bousserez] wrote:'''''
| |
− |
| |
− | :For what it's worth, this is the oldest OS we're using:
| |
− |
| |
− | Linux terra-01.vpn.as.harvard.edu 2.6.18-194.3.1.el5_lustre.1.8.4 #1 SMP
| |
− | Fri Jul 9 21:55:24 MDT 2010 x86_64 x86_64 x86_64 GNU/Linux
| |
− |
| |
− | :and this is the newest:
| |
− |
| |
− | Linux kvm-12.s.as.harvard.edu 2.6.18-194.32.1.el5 #1 SMP
| |
− | Wed Jan 5 17:52:25 EST 2011 x86_64 x86_64 x86_64 GNU/Linux
| |
− |
| |
− | :What is relevant is the 2.6.9 and the 2.6.18, not the compilation dates. It means you're running something equivalent to a RHEL4/CentOS-4 kernel instead of RHEL5/CentOS-5, which has implications for your libraries, compatibility, bugs, security, etc. I would guess that you've been updating RHEL4 (first released 2005) or equivalent for several years, and ifort11 was released during the era of RHEL5 (first released 2007), so it wouldn't be too surprising if there were a library incompatibility. I don't know whether that is the cause of your symptom, but it might be. (RHEL6 was released late last year and v12 of the Intel compilers have also been released. CentOS-6 will be out soon.)
| |
− |
| |
− | :You can continue to use the older compiler with the older OS, but I'd recommend upgrading the OS, which is worth doing anyway.
| |
− |
| |
− | --[[User:Bmy|Bob Y.]] 10:25, 13 April 2011 (EDT)
| |
− |
| |
− | === Error in partition.f when compiling with IFORT 10 ===
| |
− |
| |
− | [[Image:Obsolete.jpg]]
| |
− |
| |
− | <span style="color:red">'''''NOTE: File <tt>partition.F</tt> was removed from [[GEOS-Chem v11-01|v11-01]] and higher versions, along with the rest of the SMVGEAR solver. We shall leave this information here for reference.'''''</span>
| |
− |
| |
− | Prasad Kasibhatla reported [[GEOS-Chem v8-01-04#Error message in partition.f|an error in routine <tt>partition.f</tt>]] which caused a [[GEOS-Chem v9-01-01]] execution to halt. Upon further investigation, we found that this error only occurs when compiling GEOS-Chem with the [[Intel Fortran Compiler#IFORT 10|Intel Fortran Compiler version 10]] (aka IFORT 10) when selecting <tt>-O2</tt> optimization and <tt>-openmp</tt> parallelization.
| |
− |
| |
− | We recommend that all GEOS-Chem users upgrade to [[#IFORT 11|Intel Fortran Compiler version 11]] (aka IFORT 11). If you must use IFORT 10, then we recommend that you compile the entire code with the <tt>-O1</tt> optimization option. For more information, please see [[GEOS-Chem v8-01-04#Instance of this error when compiling with IFORT 10|this wiki post]].
| |
− |
| |
− | --[[User:Bmy|Bob Y.]] 09:17, 29 June 2011 (EDT)
| |
− |
| |
− | === KPP not compatibile with IFORT 9.1===
| |
− |
| |
− | Please see [[GEOS-Chem_v8-02-03#KPP_is_not_compatible_with_IFORT_9.1|this wiki post]] about how problems compiling the KPP solver with IFORT 9.1.
| |
− |
| |
− | === Speedup With Hyperthreading on Nehalem chips ===
| |
− |
| |
− | Hyperthreading is when a job uses more threads than there are actual CPU cores. I've noticed that using 16 threads ($OMP_NUM_THREADS = 16) on an 8-core system (2 x quad core Intel Nehalem X5570's) leads to a 15% speedup over using 8 threads. These tests were with GEOS-Chem v8-02-03, full chemistry, 2x2.5, ifort 10.1.021, and
| |
− |
| |
− | FFLAGS = -cpp -w -O3 -auto -noalign -convert big_endian -g -traceback -CB -vec-report0.
| |
− |
| |
− | This does not have a positive impact when using earlier generations of Intel chips (Harpertown or Clovertown).
| |
− |
| |
− | --[[User:daven|Daven Henze]] 1:42, 16 December 2009 (MDT)
| |
− |
| |
− | === Performance bottleneck caused by inefficient subroutine calls ===
| |
− |
| |
− | Special care has to be taken when passing pointer arrays or sub-fields of dervied type objects to subroutines. If this is done incorrectly, it can cause a huge performance slowdown. Please see the discussion on our [[Passing array arguments efficiently in GEOS-Chem]] wiki page for full details.
| |
− |
| |
− | --[[User:Bmy|Bob Y.]] 10:49, 10 June 2013 (EDT)
| |
− |
| |
− | === Bugs in the IFORT compiler cause HEMCO to segfault ===
| |
− |
| |
− | The [[GEOS-Chem Support Team]] has recently determined that bugs in the Intel Fortran Compiler versions 14 and 15 have caused the [[HEMCO|Harvard-NASA Emissions Component (aka HEMCO)]] to halt with segmentation faults. For more information, please see these wiki posts:
| |
− |
| |
− | #[[HEMCO#IFORT_15_error_when_using_array-out-of-bounds_error_checking|IFORT 15 error when using array-out-of-bounds error checking]]
| |
− | #[[HEMCO#IFORT_13/IFORT_14_segmentation_fault_error|IFORT 13/IFORT 14 segmentation fault error]]
| |
− |
| |
− | --[[User:Bmy|Bob Y.]] ([[User talk:Bmy|talk]]) 20:38, 25 August 2015 (UTC)
| |
Also, normally when you installs the Intel Fortran compilers, you also will install the C and C++ compilers.
Here is some information about how you can customize your Unix environment to use the Intel Fortran compiler.
Here is some information about how you can customize your Linux environment to use the GNU Fortran compiler. This information was recently migrated to our geos-chem.readthedocs.io manual.
In this section we present information about the various optimization options available in the Intel Fortran Compiler.
In this section, we present information about the compilation and optimization options that are invoked when you compile a GEOS-Chem simulation.
whereas a debugging run (meant to execute in a debugger such as TotalView) will typically use these flags:
You can use the following Intel Fortran Compiler options to select how aggressively you would like to optimize floating-point operations.
Using these semantics, the following shows some possible ways the compiler may interpret the original code:
Using these semantics, the following shows a possible way the compiler may interpret the original code:
These options are especially efficient to handle the transport. So in simulations with a faster chemistry (like tagged tracers simulations), we expect to see a higher gain in time. For example, the time for a methane run is shorten by about 30 %.
If you would like to run your code in a debugger, such as Totalview, you must use the following compiler switches:
The standard GEOS-Chem build sequence does not include any optimization flags that are specific to a certain type of CPU. If you are interested, you can certainly experiment for yourself. But be aware that this may invoke certain chip-level optimizations that could potentially change the simulation output.