Passing array arguments efficiently in GEOS-Chem

On this page we provide strategies to help you write GEOS-Chem code that passes arrays between subroutines in the most efficient manner.

Overview

In many areas of GEOS-Chem, we pass arrays as arguments from one routine to another. But if this is not done properly, it can cause GEOS-Chem to use an excessive amount of memory and take longer to run, especially at very fine resolutions. The following sections explain this issue in more depth:

Technical Description

This very technical description of how Fortran passes arrays to subroutines is taken from the Intel Fortran Compiler Version 11 Manual, p. 1628-1630:

In Fortran, there are two general types of array arguments:

1. Explicit-shape arrays (introduced with Fortran 77); for example, A(3,4) and B(0:*)
   • These arrays have a fixed rank and extent that is known at compile time.
   • Other dummy argument (receiving) arrays that are not deferred-shape (such as assumed-size arrays) can be grouped with explicit-shape array arguments.
2. Deferred-shape arrays (introduced with Fortran 95/90); for example, C(:.:)
   • Types of deferred-shape arrays include array pointers and allocatable arrays.
   • Assumed-shape array arguments generally follow the rules about passing deferred-shape array arguments.

When passing arrays as arguments, either the starting (base) address of the array or the address of an array descriptor is passed:

1. When using explicit-shape (or assumed-size) arrays to receive an array, the starting address of the array is passed.
2. When using deferred-shape (or assumed-shape) arrays to receive an array, the address of the array descriptor is passed (the compiler creates the array descriptor).

Passing an assumed-shape array or array pointer to an explicit-shape array can slow run-time performance. This is because the compiler needs to create an array temporary for the entire array. The array temporary is created because the passed array may not be contiguous and the receiving (explicit-shape) array requires a contiguous array. When an array temporary is created, the size of the passed array determines whether the impact on slowing run-time performance is slight or severe.

The following table summarizes what happens with the various combinations of array types. The amount of run-time performance inefficiency depends on the size of the array.

	Dummy Argument Array Types (i.e. declared in the routine being called)
Actual Argument Array Type (i.e. declared in the calling routine)	EXPLICIT-SHAPE ARRAYS	DEFERRED-SHAPE and ASSUMED-SHAPE ARRAYS
EXPLICIT-SHAPE ARRAYS	Result when using this combination: VERY EFFICIENT. Does not use an array temporary. Does not pass an array descriptor. Interface block optional.	Result when using this combination: EFFICIENT. Only allowed for assumed-shape arrays (not deferred-shape arrays). Does not use an array temporary. Passes an array descriptor. Requires an interface block.
DEFERRED-SHAPE and ASSUMED-SHAPE ARRAYS	Result when using this combination: When passing an allocatable array, VERY EFFICIENT. Does not use an array temporary. Does not pass an array descriptor. Interface block optional. When not passing an allocatable array: NOT EFFICIENT. Instead, use allocatable arrays whenever possible. Uses an array temporary. Does not pass an array descriptor. Interface block optional.	Result when using this combination: EFFICIENT. Requires an assumed-shape or array pointer as dummy argument. Does not use an array temporary. Passes an array descriptor. Requires an interface block.

--Bob Y. 13:47, 6 June 2013 (EDT)

In plain English

The discussion in the preceding section is very technical. We attempt to explain it in more understandable terms below.

Allocatable arrays

Many arrays in GEOS-Chem are declared with the ALLOCATABLE attribute, such as:

  REAL*8, ALLOCATABLE :: XMID(:,:,:)

In this example, we tell the compiler to expect a 3-dimensional array named XMID. But we also tell the compiler not to give XMID any memory at compile time. Instead, we do this after GEOS-Chem has started exectuting by means of an ALLOCATE statement:

  ALLOCATE( XMID( IIPAR, JJPAR, LLPAR ), STAT=RC )
  IF ( RC /= 0 ) CALL ALLOC_ERR( 'XMID' )
  XMID = 0d0

The above statements tell the compiler to make XMID of size IIPAR, JJPAR, LLPAR, which have been declared previously. We also check to make sure that there is enough memory before we give memory to XMID. You will see these statements in many modules of GEOS-Chem.

Derived-type objects

In GEOS-Chem v9-02k and higher versions, we have replaced many common allocatable arrays with fields from derived type objects. We did this to better enable GEOS-Chem to run within an external GCM, such as NASA's GEOS-5 GCM.

In particular:

1. The tracer concentration array STT is now replaced with the State_Chm%Tracers field, and
2. The CSPEC_FULL chemical species array is now replaced with State_Chm%Species field

State_Chm is a derived type object that holds information needed for GEOS-Chem's chemistry solver. A derived-type object is like a "bucket-o-variables", where you can store scalar and array variables of different types (INTEGER, REAL*4, REAL*8, CHARACTER, etc.) in a single container.

One important thing to note is that the Fortran-90 standard requires array fields of a derived-type object to be declared as with the POINTER attribute instead of with the ALLOCATABLE attribute, as you might otherwise expect. For this reason, State_Chm%Tracers and State_Chm%Species are declared as POINTER fields. In other words, the declaration of these fields looks like this:

  TYPE ChmState
      ... etc ... 
     REAL*8, POINTER :: Tracers(:,:,:,:)
     REAL*8, POINTER :: Species(:,:,:,:)
      ... etc ...  
   END TYPE ChmState

(Similarly, all GEOS-Chem arrays that hold meteorological fields are now declared as POINTER arrays of the State_Met derived-type object.)

Using POINTER instead of ALLOCATABLE here may seem like a very insignificant distinction, but as you will see in the next subsection, it has important consequences for the overall efficiency of GEOS-Chem.

Passing subsections of POINTER arrays to other routines

GEOS-Chem contains a significant fraction of third-party routines, such as TPCORE, FAST-J, etc. These routines were often not originally intended to work with GEOS-Chem. As such, many of these routines accept inputs such as the array of tracer concentrations via the argument list. Prior to GEOS-Chem v9-02k, this would have looked like:

    SUBROUTINE MY_GC_SUB

    !---------------------------
    ! Example calling routine
    !---------------------------

    ! Get variables from modules
    USE CMN_SIZE_MOD, ONLY : IIPAR, JJPAR, LLPAR  ! Dimensions
    USE TRACER_MOD,   ONLY : STT                  ! Tracer array
    USE TRACER_MOD,   ONLY : N_TRACERS            ! # of tracers
     
    ... etc ...
 
    ! Call another subroutine
    CALL THIRD_PARTY_SUB( IIPAR, JJPAR, LLPAR, STT )

    ... etc ...

    END SUBROUTINE MY_GC_SUB

    SUBROUTINE THIRD_PARTY_SUB( IX, JX, LX, NX, Q )

    !-----------------------------------
    ! Example 3rd-party routine 
    ! that gets called from GEOS-Chem 
    !-----------------------------------

    ! Arguments
    INTEGER, INTENT(IN) :: IX, JX, LX, NX
    REAL*8,  INTENT(IN) :: Q(IX,JX,LX,NX)

    END SUBROUTINE MY_OTHER_GC_SUB