Difference between revisions of "Machine issues & portability"

Revision as of 17:40, 16 April 2008

On this page we list the compiler-dependent and platform-dependent issues that we have recently encountered.

IFORT Compiler

Comparison between IFORT 9.1 and IFORT 10.1

The table shows the wallclock time and mean OH for several GEOS-Chem simulations that were done in order to compare Intel Fortran Compiler (IFORT) v9.1 vs. v10.1. The simulations had all these things in common:

• GEOS-Chem v8-01-01
• 4x5 GEOS-5 met fields
• 1-week of simulation (0 GMT 2008/01/01 to 0 GMT 2008/01/08)
• Base compiler options: -cpp -w -auto -noalign -convert big_endian

NOTES:

1. The compiler options -O3 -ipo -non-prec-div -static correspond to the -fast optimization option.
2. With IFORT 9.1, switching from -O2 to -O3 does not change the mean OH concentration. The bpch files of the runs were binary identical to each other.
3. IFORT 10.1 is always faster than the equivalent run in IFORT 9.1. IFORT 10.1 seems to optimize code better for multi-core chipsets.
4. In general, on the same # of CPU's, switching from -O2 to -O3 does not result in a significant speedup. This is true for both IFORT 9.1 and IFORT 10.1
5. With IFORT 10.1,

26-Mar-2008

Harvard users need to also make a similar fix as described below. Copy this code into your .cshrc file under the "Linux" section:

   #--------------------------------------------------------------------------
   # Due to a limitation in the glibc library that is used by the Intel IFORT 
   # v9.x compilers, you must do the following in order to avoid potential 
   # memory problems with OpenMP:
   #
   # (1) Explicitly set the "stacksize" limit to a large positive number
   #      instead of to "unlimited".
   #
   # (2) Explicitly set the "KMP_STACKSIZE" environment variable to a large
   #      positive number (but not so large that you get an error msg.)
   #
   # For more information see the Intel IFORT release notes:
   #  http://archimede.mat.ulaval.ca/intel/fc/9.1.036/doc/Release_Notes.htm
   #
   # The symptom will be that GEOS-Chem will appear to be out of memory and 
   # will die with a segmentation fault.
   #
   # Only reset the stacksize on Ceres & Tethys, since these are the only
   # 2 machines on which we will be running GEOS-Chem.
   #
   # (bmy, 3/31/08)
   #--------------------------------------------------------------------------

   # Test if this is Ceres or Tethys (regardless of .as.harvard.edu etc.)
   set resetstack = `perl -e '$a=qx(uname -n); if ($a=~"ceres" or $a=~"tethys") {print 1;} else {print 0;}'`

   # Only reset stacksize limits on Ceres or Tethys
   if ( $resetstack == 1 ) then
      limit  stacksize     10000000000
      setenv KMP_STACKSIZE 100000000
   endif                   

   # Undefine 
   unset resetstack

07-Jan-2008

(1) The IFORT compiler has an error that can cause the GEOS-Chem to appear that it is running out of memory when it actually isn't. The symptom that we have noticed is that it seems to choke right when the TPCORE is called. This may tend to happen more often IFORT v9 or v10 on Linux Boxes, but it can also happen on Altix/Itanium systems.

If GEOS-Chem still crashes with the thiserror, then you may need to set the stacksize variable to a large positive # instead of unlimited. This is a known issue with the POSIX glibc library that is used by IFORT.

Try adding this code to your .cshrc file as well under the "Altix" section:

   #--------------------------------------------------------------------------
   # Due to a limitation in the glibc library that is used by the Intel IFORT 
   # v9.x compilers, you must do the following in order to avoid potential 
   # memory problems with OpenMP:
   #
   # (1) Explicitly set the "stacksize" limit to 2097152 kbytes (which is
   #      the max allowable value) instead of to "unlimited".
   #
   # (2) Explicitly set the "KMP_STACKSIZE" environment variable to a 
   #      large positive number (e.g. 209715200).
   # 
   # For more information see the Intel IFORT release notes:
   #  http://archimede.mat.ulaval.ca/intel/fc/9.1.036/doc/Release_Notes.htm
   #
   # The symptom will be that GEOS-Chem will appear to be out of memory and 
   # will die with a segmentation fault.  This may happen especially if you
   # are running GEOS-Chem with GEOS-5 met on Altix or Titan.
   #
   # (bmy, 8/16/07, 3/13/08)
   #--------------------------------------------------------------------------
   limit  stacksize     2097152 kbytes
   setenv KMP_STACKSIZE 209715200

The 2097152 is the maximum allowable stacksize on the Harvard Altix/Itanium system. This may be different on your system. You can find out the maximum stacksize on your machine by typing "limit" at the Unix prompt. On your machine the number may vary. Then just cut-n-paste this number and replace the "2097152 kbytes" in the text above and put that into your .cshrc or .bashrc.

(2) If you are using the IFORT 10.x compilers, then you might also need to tell the compiler to put automatic arrays into heap memory instead of on the stack.

Mike Seymour (seymour@atmosp.physics.utoronto.ca) wrote:

I found this Intel page regarding stack sizes and ifort >=8.0:

http://www.intel.com/support/performancetools/fortran/sb/cs-007790.htm.

It suggests for ifort 10.0 to use the heap for temporary storage with -heap-arrays <size>, where arrays known at compile-time to be larger than <size> are allocated on the heap instead of the stack.

However, setting <size> to be 1000 does not change things. I don't know if smaller values will have an effect, or if there will be performance issues.

PGI Compiler

02-Jul-2007

Win Trivitayanurak (win@cmu.edu) wrote:

In short, TRIM and ADJUSTL or ADJUSTR do not work together properly when compiled with Portland Group Fortran. I propose removing TRIM inside the subroutine StrSqueeze. This is not urgent and relevant to only the few PGI users.

So if you are using the PGI compiler, then you will have to modify the code in routine STRSQUEEZE "charpak_mod.f" such that the statements

STR = ADJUSTR( TRIM( STR ) )
STR = ADJUSTL( TRIM( STR ) )

are now replaced with

STR = ADJUSTR( STR )
STR = ADJUSTL( STR )

and this will solve the problem. We will incorporate this into a future release of GEOS-Chem.

SGI-MIPS Compiler

Sun Studio Compiler