Difference between revisions of "Downloading GEOS-Chem data directories"

From Geos-chem
Jump to: navigation, search
(Primary download site)
 
(357 intermediate revisions by 9 users not shown)
Line 1: Line 1:
== Data Directory Access ==
+
__FORCETOC__
 +
'''''[[GEOS-Chem directory structure|Previous]] | [[Creating GEOS-Chem run directories|Next]] | [[Getting Started with GEOS-Chem]]'''''
 +
#[[Minimum system requirements for GEOS-Chem|Minimum system requirements (and software installation)]]
 +
#[[Configuring your computational environment]]
 +
#[[Downloading GEOS-Chem source code|Downloading source code]]
 +
#<span style="color:blue">'''Downloading data directories'''</span>
 +
#[[Creating GEOS-Chem run directories|Creating run directories]]
 +
#[[GEOS-Chem input files|Configuring runs]]
 +
#[[Compiling GEOS-Chem|Compiling]]
 +
#[[Running GEOS-Chem|Running]]
 +
#[[GEOS-Chem output files|Output files]]
 +
#[[Python tools for use with GEOS-Chem]]
 +
#[[GEOS-Chem_coding_and_debugging|Coding and debugging]]
 +
#[[GEOS-Chem_overview#Further_reading|Further reading]]
  
The GEOS-Chem source code, data and meteorological field directories may be accessed by anonymous FTP from one of the following FTP sites.
 
  
=== Primary download site ===
+
This content has been migrated to the [https://geos-chem.readthedocs.io/en/latest/gcc-guide/04-data/download-data.html '''Download input data''' chapter of <tt>geos-chem.readthedocs.io</tt>].
  
The primary GEOS-Chem data download site is located at:
 
  
ftp ftp.as.harvard.edu
+
----
cd pub/geos-chem
+
'''''[[GEOS-Chem directory structure|Previous]] | [[Creating GEOS-Chem run directories|Next]] | [[Getting Started with GEOS-Chem]]'''''
 
+
The geos-chem directory is further divided into the following subdirectories:
+
 
+
HDF/
+
data/
+
evaluation/
+
beta_releases/
+
mean_OH/
+
patches/
+
public_releases/
+
 
+
Here is a quick look at the contents of these subdirectories:
+
 
+
{| border=1 cellpadding=5 cellspacing=0
+
|- bgcolor="#CCCCCC"
+
! Directory                         
+
! Description                               
+
|-
+
| <tt>/data</tt>                             
+
| Root Data Directory
+
|-                         
+
| <tt>/data/GEOS_1x1</tt>                     
+
| 1x1 global grid emissions etc. files       
+
|-
+
| <tt>data/GEOS_1x1_CH</tt>                 
+
| 1x1 China-nested grid emissions etc files 
+
|-
+
| <tt>/data/GEOS_1x1_CH/GEOS_3</tt>         
+
| 1x1 China-nested grid GEOS-3 met fields 
+
|- 
+
| <tt>/data/GEOS_1x1_NA/</tt>               
+
| 1x1 NA-nested grid emissions etc files     
+
|-
+
| <tt>/data/GEOS_1x1_NA/GEOS_3</tt>         
+
| 1x1 NA-nested grid GEOS-3 met fields       
+
|-
+
| &nbsp;
+
| &nbsp;
+
|-
+
| <tt>/data/GEOS_2x2.5/</tt>                 
+
| 2x2.5 emissions and other data files 
+
|-     
+
| <tt>/data/GEOS_2x2.5/GEOS_1/YYYY/MM</tt>   
+
| GEOS-1 2x2.5 met data               
+
|-
+
| <tt>/data/GEOS_2x2.5/GEOS_S/YYYY/MM</tt>   
+
| GEOS-STRAT 2x2.5 met data               
+
|-
+
| <tt>/data/GEOS_2x2.5/GEOS_3/YYYY/MM</tt>   
+
| GEOS-3 2x2.5 met data               
+
|-
+
| <tt>/data/GEOS_2x2.5/GEOS_4_v4/YYYY/MM</tt> 
+
| GEOS-4 2x2.5 met data (late-look)   
+
|-
+
| <tt>/data/GEOS_2x2.5/GEOS_4_flk/YYYY/MM</tt>
+
| GEOS-4 2x2.5 met data (1st-look)
+
|-
+
| <tt>/data/GEOS_2x2.5/GEOS_4_5/YYYY/MM</tt>
+
| GEOS-5 2x2.5 met data 
+
|-
+
| &nbsp;
+
| &nbsp;
+
|-
+
| <tt>/data/GEOS_4x5/</tt>               
+
| 4x5 emissions and other data files         
+
|-
+
| <tt>/data/GEOS_4x5/GEOS_1/YYYY/MM</tt>     
+
| GEOS-1 4x5 met data               
+
|-
+
| <tt>/data/GEOS_4x5/GEOS_S/YYYY/MM</tt>   
+
| GEOS-STRAT 4x5 met data               
+
|-
+
| <tt>/data/GEOS_4x5/GEOS_3/YYYY/MM</tt>
+
| GEOS-3 4x5 met data               
+
|-
+
| <tt>/data/GEOS_4x5/GEOS_4_v4/YYYY/MM</tt> 
+
| GEOS-4 4x5 met data (late-look)   
+
|-
+
| <tt>/data/GEOS_4x5/GEOS_4_flk/YYYY/MM</tt> 
+
| GEOS-4 4x5 met data (1st-look) 
+
|-
+
| <tt>/data/GEOS_4x5/GEOS_5/YYYY/MM</tt> 
+
| GEOS-5 4x5 met data
+
|-
+
| &nbsp;
+
| &nbsp; 
+
|-
+
| <tt>/evaluation</tt>                       
+
| Plots from 1-yr benchmark simulations
+
|-     
+
| <tt>/HDF</tt>                             
+
| Contains user code for reading HDF and HDF-EOS data files
+
|-     
+
| <tt>/NRT-ARCTAS</tt>                             
+
| Contains output from the [http://coco.atmos.washington.edu/cgi-bin/ion-p?page=arctas.ion GEOS-Chem Near-Real-Time simulations for ARCTAS]
+
|-
+
| <tt>/public_releases</tt>
+
| Directory containing TAR file source code and run directories for GEOS-Chem public releases
+
|-
+
| <tt>/beta_releases</tt> 
+
| Directory containing TAR file source code and run directories for beta GEOS-Chem releases
+
|-
+
| <tt>patches/</tt>
+
| Directory containing bug-fix software patches (if necessary)
+
|-
+
| <tt>mean_OH/</tt>
+
| Directory containing 3-D mean OH fields archived from previous GEOS-Chem simulations.
+
|}
+
 
+
Please view the [http://acmg.seas.harvard.edu/geos/geos_sim.html catalog of met data at the Harvard archive] to determine if the data period you wish to download is available.
+
 
+
=== Alternative download site ===
+
 
+
The GEOS-Chem data and meteorological fields used by Dalhousie University are also available via anonymous FTP from:
+
 
+
ftp rain.ucis.dal.ca
+
 
+
This site has overlap with many of the above directories from the Harvard site, but it is not as extensive.  This site, however, additionally hosts the following unique datasets:
+
 
+
{| border=1 cellpadding=5 cellspacing=0
+
|- bgcolor="#CCCCCC"
+
! Directory                         
+
! Description                               
+
|-
+
| <tt>/GEOS_0.5x0.666_EU/</tt>                             
+
| 1/2 x 2/3 European nested grid emission etc files
+
|-                         
+
| <tt>/GEOS_0.5x0.666_EU.d/</tt>                     
+
| 1/2 x 2/3 European nested grid met fields (GEOS-5)
+
|-
+
| <tt>/GEOS_0.5x0.666_NA/</tt>                 
+
| 1/2 x 2/3 North American nested grid emission etc files
+
|-
+
| <tt>/GEOS_0.5x0.666_NA.d/</tt>         
+
| 1/2 x 2/3 North American nested grid met fields (GEOS-5)
+
|-
+
| &nbsp;
+
| &nbsp;
+
|- 
+
| <tt>/GEOS_1x1.25/</tt>               
+
| 1 x 1.25 Global GEOS4 emission etc files     
+
|-
+
| <tt>/GEOS_1x1.25.d/</tt>         
+
| 1 x 1.25 Global GEOS4 met fields
+
|}
+
 
+
A catalog of available data may be found [http://fizz.phys.dal.ca/~atmos/g47.swf?jump=6.2 HERE].
+
 
+
=== Question about directory structure ===
+
 
+
'''''Shanna Shaked (shaked@umich.edu) wrote:'''''
+
 
+
:We are working again on trying to run GEOS-Chem.  However, we are encountering some errors that may be due to the directory structure.  We find a discrepancy between the directory structure described in the GEOS-Chem manual and that available on the ftp site.
+
 
+
:The GEOS-Chem manual describes a directory structure of:
+
+
    data/GEOS_4x5/GEOS_5/YYYY/MM
+
 
+
:However, on the ftp site, we find a directory structure with an extra '.d':
+
 
+
    data/GEOS_4x5.d/GEOS_5/YYYY/MM
+
 
+
:(the <tt>GEOS_5</tt> folder is in <tt>GEOS_4x5.d</tt> rather than <tt>GEOS_4x5</tt>).  There does exist a  <tt>GEOS_4x5</tt> that contains many of the emissions data, but does not contain <tt>GEOS_5</tt>.
+
 
+
:If we leave the structure as is, and enter <tt>../data/GEOS_4x5/</tt> as our root data directory in <tt>input.geos</tt>, we get a file not found error when it looks for <tt>GEOS_5</tt> within this directory (obviously).
+
 
+
:If we instead enter <tt>../data/GEOS_4x5.d</tt> as our root data directory, we get a file not found error when the program looks for emissions within this directory (lightning NOx emissions, in this case).
+
 
+
:QUESTION: To solve this problem, we have moved the <tt>GEOS_5</tt> folder into the <tt>GEOS_4x5</tt> directory.  [Is this] okay?
+
 
+
'''''Bob Yantosca (yantosca@seas.harvard.edu) replied:'''''
+
 
+
:The only difference on our system between e.g. <tt>GEOS_4x5</tt> and <tt>GEOS_4x5.d</tt> is that our sysadmin (Jack Yatteau) set up the ".d" directories separately so that they only contain met data (which is much larger than the emissions etc. data).  That way he could separate the disks that just had met data from the disk that have the emissions data to facilitate our configuration here.  There are symbolic links from <tt>GEOS_4x5</tt> to <tt>GEOS_4x5.d</tt> etc. (i.e. the directory <tt>GEOS_4x5/GEOS_5</tt> is actually a symbolic link to the corresponding directory in <tt>GEOS_4x5.d/GEOS_5/</tt> and etc. for the other met field resolutions & directories). 
+
 
+
:You don't necessarily have to do this on your end, but this is what we did here.  You can just make the <tt>GEOS_4x5/GEOS_5</tt> etc. real subdirectories and not symbolic links and store the data there.  The solution you picked above is OK.
+
 
+
:Also to facilitate FTP file transfer, you could do the following:
+
:* Write a script or an FTP macro
+
:* Use a 3rd-party GUI program like FireFTP in Mozilla Firefox.
+
:* Or even better yet, use the Unix <tt>wget</tt> utility (see below)
+
 
+
:Each user is responsible for their own file transfers.
+
 
+
--[[User:Bmy|Bob Y.]] 11:04, 5 February 2009 (EST)
+
 
+
== Using wget to download files ==
+
 
+
Probably the simplest way to download the GEOS-Chem emissions and met field data is to use the Unix <tt>wget</tt> utility.  This allows you to download multiple files and directories at a time.
+
 
+
The <tt>wget</tt> utility is free and open-source (published by GNU), and comes standard with pretty much all builds of *nix (Linux, Ubuntu, Fedora, Centos, etc.).  You can check out the [http://www.editcorp.com/Personal/Lars_Appel/wget/wget_1.html user manual] for more information.
+
 
+
=== Syntax ===
+
 
+
Most of the time, the syntax you will use to download multiple directories is as follows:
+
 
+
<nowiki>wget -r -nH "ftp://ftp.as.harvard.edu/pub/geos-chem/data/DIRECTORY_NAME/"</nowiki>
+
 
+
The options to <tt>wget</tt> are as follows:
+
 
+
-r  Specifies recursive directory transfer (i.e. will download all subdirectories)
+
-nH  Will store all directories and subdirectories in DIRECTORY_NAME, not ftp.as.harvard.edu/DIRECTORY_NAME
+
 
+
'''''NOTE: The URL must be enclosed in quotes for file transfer to occur. If you omit the quotes then <tt>wget</tt> will just return a directory listing in a file named <tt>index.html</tt> without any files being downloaded.'''''
+
 
+
=== Examples ===
+
 
+
1. Download all emissions files in the GEOS_2x2.5 data directory structure:
+
 
+
<nowiki>wget -r -nH "ftp://ftp.as.harvard.edu/pub/geos-chem/data/GEOS_2x2.5/" &</nowiki>
+
 
+
The <tt>&</tt> character will make sure the file transfer happens in the Unix background.
+
 
+
 
+
2. Download all available GEOS-Chem 2 x 2.5 met field data files in the GEOS_2x2.5.d directory structure:
+
 
+
<nowiki>wget -r -nH "ftp://ftp.as.harvard.edu/pub/geos-chem/data/GEOS_2x2.5.d/" &</nowiki>
+
 
+
NOTE: Due to the huge volume of data involved, this is not recommended, as the file downloads may swamp your system.  It's better to do this:
+
 
+
 
+
3. Download all GEOS-5 met data at 2 x 2.5 resolution:
+
 
+
<nowiki>wget -r -nH "ftp://ftp.as.harvard.edu/pub/geos-chem/data/GEOS_2x2.5.d/GEOS_5/" &</nowiki>
+
 
+
And it may even be better to download one year at a time:
+
 
+
 
+
4. Download all GEOS-5 met data at 2 x 2.5 resolution for 2008:
+
 
+
<nowiki>wget -r -nH "ftp://ftp.as.harvard.edu/pub/geos-chem/data/GEOS_2x2.5.d/GEOS_5/2008/" &</nowiki>
+
 
+
etc.
+
 
+
--[[User:Bmy|Bob Y.]] 11:01, 3 December 2009 (EST)
+

Latest revision as of 14:56, 4 August 2022

Previous | Next | Getting Started with GEOS-Chem

  1. Minimum system requirements (and software installation)
  2. Configuring your computational environment
  3. Downloading source code
  4. Downloading data directories
  5. Creating run directories
  6. Configuring runs
  7. Compiling
  8. Running
  9. Output files
  10. Python tools for use with GEOS-Chem
  11. Coding and debugging
  12. Further reading


This content has been migrated to the Download input data chapter of geos-chem.readthedocs.io.



Previous | Next | Getting Started with GEOS-Chem