Difference between revisions of "Downloading GEOS-Chem data directories"

Revision as of 22:37, 16 February 2011

This page describes where you can obtain the GEOS-Chem source code and required data files.

What you need to download before you can run GEOS-Chem

When setting up GEOS–Chem on your system, you will need install the following components:

1. The GEOS–Chem source code directory. This is the directory where the Fortran-90 source code files (i.e. *.f, *.f90 files) and Makefiles reside. Your Fortran compiler will create an executable from these source code files.
2. A GEOS–Chem run directory. Here is where you will run the compiled GEOS–Chem executable. Each run directory contains:
   1. Various input files that you can modify in order to select different options for your GEOS–Chem simulation
   2. Files that define the GEOS–Chem's chemical and photolysis mechanisms
   3. "Restart files" that hold the initial conditions for your GEOS–Chem simulation
3. The GEOS–Chem shared data directories. This is the directory tree where the following types of data are stored:
   1. Meteorological data (a.k.a. the "met fields) used to drive GEOS–Chem
   2. Emissions inventories for GEOS–Chem
   3. Scale factors to used to scale emissions from a base year to a given year
   4. Oxidant (OH, O3) concentrations for both full-chemistry and offline simulations
   5. IPCC future scenarios (for GCAP simulatons)
   6. Other GEOS–Chem specific data files.

The GEOS–Chem source code and run directories are small enough to download directly to your own disk space in your Unix account. You can download these with the Git version control software.

On the other hand, the GEOS–Chem shared data directories contain many large files that probably cannot all fit into your own personal disk quota. Therefore, you (or your sysadmin) should download the shared data directories to a common disk space where all GEOS–Chem users in your group can access them. The volume of data contained in the shared data directories precludes using Git; you must instead download these files via FTP, wget, or similar file transfer programs.

For complete downloading instructions, please see:

1. GEOS-Chem User's Guide: Chapter 2.2, Downloading the GEOS-Chem source code
2. GEOS-Chem User's Guide: Chapter 2.3, Downloading the GEOS-Chem run directories
3. GEOS-Chem User's Guide: Chapter 2.4, Downloading the GEOS-Chem shared data directories

For more information on the files contained in each of these directories, please see:

1. GEOS-Chem User's Guide: Chapter 3, Compiling the GEOS-Chem source code
2. GEOS-Chem User's Guide: Chapter 4, The GEOS-Chem shared data directories
3. GEOS-Chem User's Guide: Chapter 5, The GEOS-Chem run directories

--Bob Y. 11:14, 10 February 2011 (EST)

TARBALL downloads are no longer supported

In the past, the GEOS–Chem source code and run directories were distributed to the user community as a series of TARBALL (i.e. *.tar.gz) files via anonymous FTP. The advantage of this method was that one would only have to download a single file. However, as the number of GEOS–Chem users (and submitted source code modifications) grew, this method became unwieldy. For example, if only a single file needed to be updated, the entire TARBALL file would have to be regenerated. This often became a source of confusion and error.

Given the large number of user code submissions, robust source code management techniques must be employed in order to ensure the integrity of the GEOS–Chem code. Therefore, the GEOS–Chem Support Team has selected the Git version control software for managing the GEOS–Chem source code and run directories.

--Bob Y. 11:14, 10 February 2011 (EST)

Data Directory Access

The GEOS–Chem shared data directories contain the various met fields, emissions, and other data that GEOS–Chem will read in during the course of a simulation. You must download the shared data directories via FTP or a similar utility (e.g. wget, FireFTP, SecureFX, etc.) The large volume of data makes it impossible to track this directory structure with Git.

If there are several GEOS–Chem users in your research group, then chances are someone (perhaps your sysadmin) has already downloaded the GEOS–Chem shared directory structure to a common disk space on your servers. Nevertheless, you should still check to make sure that the most recent subdirectories are present. See Chapter 4 for complete details about the contents of the GEOS–Chem shared data directories.

There are currently two data archives from which you may download the GEOS–Chem shared data directories:

1. Harvard archive (ftp.as.harvard.edu)
2. Dalhousie archive (rain.ucis.dal.ca)

The Dalhousie archive is not as comprehensive as the Harvard archive. However, the Dalhousie archive stores data files for the various nested grids and global 1° x 1.25° grids that are not available on the Harvard archive.

You may access either of these archives via the following methods:

1. wget (Recommended)
2. Anonymous FTP

Primary download site

The primary GEOS-Chem data download site is located at:

ftp ftp.as.harvard.edu
cd pub/geos-chem

We recommend that you use the wget utility to download these directories instead of anonymous FTP. Wget allows you to download multiple directories at once.

Downtime

The Harvard FTP archive will be unavailable during the weekly maintenance period every Monday between 7-10 AM ET. Please plan your data downloads accordingly.

Directory structure

The pub/geos-chem directory is further divided into the following subdirectories:

1month_plots/
1yr_benchmarks/
NRT/
NRT-ARCTAS/
NRT_archive/
beta_releases/
dao/
data/
downloads/
mean_OH/
patches/
public_releases/

The data/ subdirectory is the most important. This is the root under which the GEOS–Chem shared data directories reside. This is where you will find the GEOS-Chem met fields and emissions data.

Here is a quick look at the contents of these subdirectories of pub/geos-chem/

Please view the catalog of met data at the Harvard archive to determine if the data period you wish to download is available.

--Bob Y. 15:39, 10 February 2011 (EST)

Obsolete data directories

The following data directories contain iles that are now considered obsolete.

--Bob Y. 15:39, 10 February 2011 (EST)

Alternative download site

The GEOS-Chem data and meteorological fields used by Dalhousie University are also available via anonymous FTP from:

ftp rain.ucis.dal.ca

We recommend that you use the wget utility to download these directories instead of anonymous FTP. Wget allows you to download multiple directories at once.

Directory structure

The Dalhousie archive overlaps with many of the above directories from the Harvard site, but it is not as extensive. This site, however, additionally hosts the following unique datasets:

A catalog of available data may be found HERE.

--Bob Y. 11:49, 10 February 2011 (EST)

For GCAP users

For those users who wish to run GEOS-Chem with the GISS/GCAP met fields, please contact Loretta Mickley.

--Bob Y. 09:52, 8 March 2010 (EST)

Question about directory structure

Shanna Shaked wrote:

We are working again on trying to run GEOS-Chem. However, we are encountering some errors that may be due to the directory structure. We find a discrepancy between the directory structure described in the GEOS-Chem manual and that available on the ftp site.

The GEOS-Chem manual describes a directory structure of:

   data/GEOS_4x5/GEOS_5/YYYY/MM

However, on the ftp site, we find a directory structure with an extra '.d':

   data/GEOS_4x5.d/GEOS_5/YYYY/MM

(the GEOS_5 folder is in GEOS_4x5.d rather than GEOS_4x5). There does exist a GEOS_4x5 that contains many of the emissions data, but does not contain GEOS_5.

If we leave the structure as is, and enter ../data/GEOS_4x5/ as our root data directory in input.geos, we get a file not found error when it looks for GEOS_5 within this directory (obviously).

If we instead enter ../data/GEOS_4x5.d as our root data directory, we get a file not found error when the program looks for emissions within this directory (lightning NOx emissions, in this case).

QUESTION: To solve this problem, we have moved the GEOS_5 folder into the GEOS_4x5 directory. [Is this] okay?

Bob Yantosca replied:

The only difference on our system between e.g. GEOS_4x5 and GEOS_4x5.d is that our sysadmin (Jack Yatteau) set up the ".d" directories separately so that they only contain met data (which is much larger than the emissions etc. data). That way he could separate the disks that just had met data from the disk that have the emissions data to facilitate our configuration here. There are symbolic links from GEOS_4x5 to GEOS_4x5.d etc. (i.e. the directory GEOS_4x5/GEOS_5 is actually a symbolic link to the corresponding directory in GEOS_4x5.d/GEOS_5/ and etc. for the other met field resolutions & directories).

You don't necessarily have to do this on your end, but this is what we did here. You can just make the GEOS_4x5/GEOS_5 etc. real subdirectories and not symbolic links and store the data there. The solution you picked above is OK.

Also to facilitate FTP file transfer, you could do the following:

• Write a script or an FTP macro
• Use a 3rd-party GUI program like FireFTP in Mozilla Firefox.
• Or even better yet, use the Unix wget utility (see below)

Each user is responsible for their own file transfers.

--Bob Y. 11:04, 5 February 2009 (EST)

Using wget to download files

Probably the simplest way to download the GEOS-Chem emissions and met field data is to use the Unix wget utility. This allows you to download multiple files and directories at a time.

The wget utility is free and open-source (published by GNU), and comes standard with pretty much all builds of *nix (Linux, Ubuntu, Fedora, Centos, etc.). You can check out the user manual for more information.

Syntax

Most of the time, the syntax you will use to download multiple directories is as follows:

Downloading data from Harvard:

wget -r -nH "ftp://ftp.as.harvard.edu/pub/geos-chem/data/DIRECTORY_NAME/"

Downloading data from Dalhousie:

wget -r -nH "ftp://rain.ucis.dal.ca/DIRECTORY_NAME/"

The options to wget are as follows:

-r   Specifies recursive directory transfer (i.e. will download all subdirectories)
-nH  Will store all directories and subdirectories in DIRECTORY_NAME, not ftp.as.harvard.edu/DIRECTORY_NAME

If you wish to trim the name of the downloaded directory (i.e., so it downloads as DIRECTORY_NAME, not pub/geos-chem/data/DIRECTORY_NAME), then use the --cut-dirs option:

wget -r -nH --cut-dirs=X "ftp://ftp.as.harvard.edu/pub/geos-chem/data/DIRECTORY_NAME/"

where X is the number of directories to trim.

NOTE: The URL must be enclosed in quotes for file transfer to occur. If you omit the quotes then wget will just return a directory listing in a file named index.html without any files being downloaded.

Examples

1. Download all emissions files in the GEOS_2x2.5/ data directory structure:

wget -r -nH "ftp://ftp.as.harvard.edu/pub/geos-chem/data/GEOS_2x2.5/" &

The & character will make sure the file transfer happens in the Unix background.

2. Download all available GEOS-Chem 2° x 2.5° met field data files in the GEOS_2x2.5.d directory structure:

wget -r -nH "ftp://ftp.as.harvard.edu/pub/geos-chem/data/GEOS_2x2.5.d/" &

NOTE: Due to the huge volume of data involved, this is not recommended, as the file downloads may swamp your system. It's better to do this:

3. Download all GEOS-5 met data at 2° x 2.5° resolution:

wget -r -nH "ftp://ftp.as.harvard.edu/pub/geos-chem/data/GEOS_2x2.5.d/GEOS_5/" &

And it may even be better to download one year at a time:

4. Download all GEOS-5 met data at 2° x 2.5° resolution for 2008:

wget -r -nH "ftp://ftp.as.harvard.edu/pub/geos-chem/data/GEOS_2x2.5.d/GEOS_5/2008/" &

5. Download the lightning_NOx_201101 folder to an existing local data directory /data/gc/GEOS_4x5 using the --cut option:

cd /data/gc/GEOS_4x5 
wget -r -nH --cut-dirs=4 "ftp://ftp.as.harvard.edu/pub/geos-chem/data/GEOS_4x5/lightning_NOx_201101"

etc.

--Bob Y. 14:27, 8 September 2010 (EDT) --Daven 17:35, 16 February 2011 (EST)