The NcdfUtilities package
The NcdfUtilities package contains Fortran modules that you can use to write data to and read data from netCDF files. This package is contained within GEOS-Chem (in the NcdfUtil/ folder, but may also be downloaded as a separate standalone package.
List of modules
NcdfUtilities within GEOS-Chem
Hello
NcdfUtilities as a standalone distribution
The following sections describe how you can download and run the NcdfUtilities as a standalone package that you can incorporate into your own Fortran programs.
Setting enviroment variables for the standalone NcdfUtilities package
The NcdfUtilities library requires that you set the following environment variables listed below in your system startup file (e.g. .bashrc or .cshrc).
NOTE: If you load a netCDF library into your Unix environment with the module command, then very often the root path to the netCDF library will be automatically set for you. Then you can use this to define the variables listed below. Ask your IT staff for more information.
| Variable | Description |
|---|---|
| NETCDF_BIN | The bin/ folder of the netCDF installation, where utilities such as nc-config, ncdump, etc. are stored. |
| NETCDF_INCLUDE | The include/ folder of the netCDF installation, where the netcdf.inc and netcdf_mod.F90 are found. |
| NETCDF_LIB | The lib/ or lib64/ folder of the netCDF installation, where the netCDF library files (e.g. libnetcdf.a) are found. |
NOTE: In netCDF-4.2 and higher versions, the netCDF Fortran libraries are built from a separate distribution. If on your system, the netCDF-Fortran libraries have been installed into a different folder than the rest of the netCDF libaries, you will also need to set these environment variables in your system startup file:
| Variable | Description |
|---|---|
| NETCDF_FORTRAN_BIN | The bin/ folder of the netCDF-Fortran installation, where utilities such as nf-config is stored. |
| NETCDF_FORTRAN_INCLUDE | The include/ folder of the netCDF-Fortran installation, where the netcdf.inc and netcdf_mod.F90 are found. |
| NETCDF_FORTRAN_LIB | The lib/ or lib64/ folder of the netCDF-Fortran installation, where the netCDF-Fortran library files (e.g. libnetcdff.a) are found. |
--Bob Yantosca (talk) 18:43, 8 March 2017 (UTC)
Downloading the NcdfUtilities standalone package
You can download a copy of the NcdfUtilities standalone package with the Git source code management system. The master NcdfUtilities code repository is hosted on Bitbucket.org. To download the code, type:
git clone https://bitbucket.org/gcst/ncdfutilities.git NcdfUtil
This will download the NcdfUtilities into a folder named NcdfUtil in your disk space.
--Bob Yantosca (talk) 18:56, 8 March 2017 (UTC)
Directory structure of the NcdfUtilities standalone package
If you download the NcdfUtilities as a standalone package, the root-level directory will contain the following sub-directories:
| Sub-directory | Description |
|---|---|
| Code/ | The Fortran source code files (*.F *.F90) reside here. |
| bin/ | The TestNcdfUtilities.x executable will be created here. |
| doc/ | The NcdfUtilities documentation will be created here. |
| lib/ | The NetCdfUtilities library file (libNcUtils.a) will be created here. |
| mod/ | Compiled module files (*.mod) will be created here. |
| perl/ | Several perl scripts that can be useful in creating netCDF files are contained here. |
System requirements for using NcdfUtilities
In order to use NcdfUtilities, you will first have to check to see if the netCDF library is installed on your system. You may find that there are several netCDF library versions to select from. Or you can ask your IT staff to build you a version.
In order to build the reference documents (described below), you must have the LaTeX utilities (i.e. latex, dvips, dvipdf) installed on your system.
Compiling the NcdfUtilities code
The NcdfUtilities/Code directory contains the Fortran source code modules as well as two Makefiles (named Makefile and Makefile_header.mk).
The file Makefile_header.mk is a sub-makefile which is used to define the compilation options for different compilers. At present, the Intel Fortran Compiler, GNU Fortran compiler, and PGI Fortran compiler are supported.
Once you have set the proper environment variables for your system (as described above), you are ready to build the executable. Make sure you are in the Code/ subdirectory and type:
make lib
This should start building the source code and create a library file named libNcUtils.a in the lib/ subdirectory.
If you would like to build the NcdfUtilities for a HPC-environment, then type:
make lib HPC=yes
This will compile the code using the mpif90 compiler wrapper rather than the underlying compilers themselves. This will activate the various MPI settings.
--Bob Yantosca (talk) 18:55, 8 March 2017 (UTC)
Testing the NcdfUtilities Library
Once the libNcUtils.a file has been created in the lib/ subdirectory, you can test to see if the library was created (and can link to) the netCDF library correctly. Type:
make check
This will create an executable file named TestNcdfUtilities.x in the bin subdirectory, and will also execute the file. If the libNcUtils.a library was installed correctly you should see the following output:
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% %%% Testing libNcdfUtilities.a %%% %%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%% === Begin netCDF file creation test === Writing time (dim ) to netCDF file Writing lev (dim ) to netCDF file Writing lat (dim ) to netCDF file Writing lon (dim ) to netCDF file Writing cdim1 (dim ) to netCDF file Writing cdim2 (dim ) to netCDF file Testing re-opening of define mode Writing lon (1D array) to netCDF file Writing lat (1D array) to netCDF file Writing lev (1D array) to netCDF file Writing time (1D array) to netCDF file Writing PS (3D array) to netCDF file Writing T (4D array) to netCDF file Writing DESC (2D char ) to netCDF file === End netCDF file creation test === === Begin netCDF file reading test === Reading lon (dim ) back from netCDF file...........PASSED Reading lat (dim ) back from netCDF file...........PASSED Reading lev (dim ) back from netCDF file...........PASSED Reading time (dim ) back from netCDF file...........PASSED Reading cdim1 (dim ) back from netCDF file...........PASSED Reading cdim2 (dim ) back from netCDF file...........PASSED Reading lon (array) back from netCDF file...........PASSED Reading lat (array) back from netCDF file...........PASSED Reading lev (array) back from netCDF file...........PASSED Reading time (array) back from netCDF file...........PASSED Reading PS back from netCDF file...........PASSED Reading PS:units back from netCDF file...........PASSED Reading PS:long_name back from netCDF file...........PASSED Reading PS:_FillValue back from netCDF file...........PASSED Reading PS:valid_range back from netCDF file...........PASSED Reading T back from netCDF file...........PASSED Reading T:units back from netCDF file...........PASSED Reading T:long_name back from netCDF file...........PASSED Reading T:_FillValue back from netCDF file...........PASSED Reading T:valid_range back from netCDF file...........PASSED Reading DESC back from netCDF file...........PASSED Reading DESC:units back from netCDF file...........PASSED Reading DESC:long_name back from netCDF file...........PASSED Reading title back from netCDF file...........PASSED Reading start_date back from netCDF file...........PASSED Reading start_time back from netCDF file...........PASSED === End of netCDF file read test! ===
If all of the tests return with "PASSED" then the libNcUtils.a file was created correctly.
Setting up automatic checks for several netCDF library installations
The GEOS-Chem Support Team has created a set of scripts to check several netCDF configurations by using the TestNcdfUtilities.x program. For more information, please see the README file at our NcdfUnitTest code repository on Bitbucket].
--Bob Yantosca (talk) 19:00, 8 March 2017 (UTC)
Generating reference documentation for the standalone NcdfUtilities package
The NcdfUtilities Fortran source code and Makefiles use the ProTeX automatic documentation system. This enables you to create reference documents in *.pdf and *.ps format from the comments in the subroutine headers.
To build the reference documents, make sure you are in the doc/ subdirectory, then type:
make doc
This will create the following documents in the doc/ subdirectory:
NcdfUtilities.pdf NcdfUtilities.ps NcdfUtilities.tex
-- Reference document for the NcdfUtilities Fortran code
in *.pdf, *.ps, and LaTeX formats
NcdfUtilities_Makefiles.pdf NcdfUtilities_Makefiles.ps NcdfUtilities_Makefiles.tex
-- Reference document for the NcdfUtilities Makefiles
in *.pdf, *.ps, and LaTeX formats
The reference documents contain a description of each subroutine and function, the variables that are passed to it as input & output arguments, and the revision history. The Makefile reference document displays the full text of the Makefiles. These documents will come in handy if you need to modify or update the Fortran code or Makefiles.
If you wish to remove the NcdfUtilities reference documentation files, then make sure you are in the doc directory and type:
make clean
--Bob Yantosca (talk) 18:45, 8 March 2017 (UTC)
Cleaning up
To remove all of the *.o, *.mod and executable file in the Code/ subdirectory only, type:
make clean
However, if you wish to also remove the contents of the bin/ and lib/ subdirectories (as well as removing the *.ps, *.pdf, and *.txt files from the doc/ subdirectory), then type:
make realclean
--Bob Yantosca (talk) 18:46, 8 March 2017 (UTC)
Previous issues that have since been resolved
Enable compression in netCDF-4 output files
This update will be included in v11-02a.
Chris Holmes wrote:
I noticed that netCDF-4 files created with GEOS-Chem were not using compression, which is one of the major benefits of netCDF-4. I don’t know if that was intentional, but I added this feature.
Along the way I found that the files called netCDF-4 in the HEMCO comments were actually netCDF-3 files with 64bit support (i.e. large file support), so I made the files real netCDF-4 classic model then added compression support.
I have sent a patch to the GCST (applied to v11-01-public-release) that enables these changes. NetCDF3 users should see no effect.
With the lowest level of compression enabled, the restart files are about half of their previous size, so a big benefit.
Enable compression for netCDF-4 files
NetCDF-4 files created by HEMCO now have lossless compression enabled.
Uses lowest compression level (deflate_level=1).
Informal testing and netCDF discussion forums suggest that higher compression
provides little additional benefit, but slower file writing.
Restart files are about 50% smaller.
Write time increases about 1 second out of about 5 seconds total.
Non-fatal errors are displayed if compression doesn't work.
No error is displayed for netCDF-3 files that don't support compression.
The GCST has added an extra check on top of Chris Holmes' update in order to prevent errors if the netCDF library cannot support file compression.
Bob Yantosca replied:
Some netCDF-4 library installations might not have been built with compression enabled. We now first check the include file netcdf.inc (which is in the netCDF include folder) to see if the function nf_def_var_deflate is defined. If it is, then we set a C-preprocessor switch named NC_HAS_COMPRESSION, which will activate the code to compress the netCDF output files. Otherwise the compression code is left disabled. This workaround was necessary in order to avoid compile-time errors.
Tests with the geosfp_4x5_standard simulation show a decrease in file size from approx. 200 MB (uncompressed) to 120 MB (compressed).
We also display a message at the top of the log file indicating if this netCDF library build supports file compression.
--Bob Yantosca (talk) 21:44, 1 March 2017 (UTC)