Difference between revisions of "Downloading data from WashU"

From Geos-chem
Jump to: navigation, search
(Created page with " == Compute Canada directory structure == If you are downloading the GEOS-FP or MERRA-2 met data, then please note that you must download the "CN" (constant) data files for e...")
 
(Compute Canada directory structure)
Line 2: Line 2:
 
== Compute Canada directory structure ==
 
== Compute Canada directory structure ==
  
If you are downloading the GEOS-FP or MERRA-2 met data, then please note that you must download the "CN" (constant) data files for each horizontal grid that you are using. 
+
The GEOS-Chem shared data directories may be downloaded from the Compute Canada server:
  
For GEOS-FP these are timestamped for 2011/01/01 and are found in these data directories:
+
http://geoschemdata.computecanada.ca
*<tt>ExtData/GEOS_0.25x0.3125/GEOS_FP/2011/01/GEOSFP.20110101.CN.025x03125.nc</tt>
+
*<tt>ExtData/GEOS_0.25x0.3125_AS/GEOS_FP/2011/01/GEOSFP.20110101.CN.025x03125.AS.nc</tt>
+
*<tt>ExtData/GEOS_0.25x0.3125_CH/GEOS_FP/2011/01/GEOSFP.20110101.CN.025x03125.CH.nc</tt>
+
*<tt>ExtData/GEOS_0.25x0.3125_EU/GEOS_FP/2011/01/GEOSFP.20110101.CN.025x03125.EU.nc</tt>
+
*<tt>ExtData/GEOS_0.25x0.3125_NA/GEOS_FP/2011/01/GEOSFP.20110101.CN.025x03125.NA.nc</tt>
+
*<tt>ExtData/GEOS_2x2.5/GEOS_FP/2011/01/GEOSFP.20110101.CN.2x25.nc</tt>
+
*<tt>ExtData/GEOS_4x5/GEOS_FP/2011/01/GEOSFP.20110101.CN.4x5.nc</tt>
+
 
+
For MERRA-2 these are timestamped for 2015/01/01 and are found in these data directories :
+
*<tt>ExtData/GEOS_0.5x0.625/MERRA2/2015/01/MERRA2.20150101.CN.05x0625.nc4</tt>
+
*<tt>ExtData/GEOS_0.5x0.625_AS/MERRA2/2015/01/MERRA2.20150101.CN.05x0625.AS.nc4</tt>
+
*<tt>ExtData/GEOS_0.5x0.625_CH/MERRA2/2015/01/MERRA2.20150101.CN.05x0625.CH.nc4</tt>
+
*<tt>ExtData/GEOS_0.5x0.625_EU/MERRA2/2015/01/MERRA2.20150101.CN.05x0625.EU.nc4</tt>
+
*<tt>ExtData/GEOS_0.5x0.625_NA/MERRA2/2015/01/MERRA2.20150101.CN.05x0625.NA.nc4</tt>
+
*<tt>ExtData/GEOS_2x2.5/MERRA2/2015/01/MERRA2.20150101.CN.2x25.nc4</tt>
+
*<tt>ExtData/GEOS_4x5/MERRA2/2015/01/MERRA2.20150101.CN.4x5.nc4</tt>
+
 
+
Additional notes:
+
*Prior to downloading GEOS-FP data, please be aware of caveats regarding use of GEOS-FP. See the [[GEOS-FP|GEOS-FP]] wiki page for more information.  
+
  
 +
which has the following directory structure:
  
 
{| border=1 cellpadding=5 cellspacing=0
 
{| border=1 cellpadding=5 cellspacing=0
Line 116: Line 98:
  
 
|-valign="top"   
 
|-valign="top"   
|<tt>gcgrid/data/ExtData/GEOS_4x5/MERRA2/YYYY/MM/</tt>   
+
|<tt>ExtData/GEOS_4x5/MERRA2/YYYY/MM/</tt>   
 
|[[GEOS-Chem horizontal grids#GMAO 4 x 5 grid|4&deg; x 5&deg;]] [[MERRA-2]] global met fields
 
|[[GEOS-Chem horizontal grids#GMAO 4 x 5 grid|4&deg; x 5&deg;]] [[MERRA-2]] global met fields
 
|}
 
|}
  
 +
=== GEOS-FP and MERRA-2 constant data files ===
 +
 +
If you are downloading the GEOS-FP or MERRA-2 met data, then please note that you must also download the "CN" (constant) data files for each horizontal grid that you are using. 
 +
 +
For GEOS-FP these are timestamped for 2011/01/01 and are found in these data directories:
 +
*<tt>ExtData/GEOS_0.25x0.3125/GEOS_FP/2011/01/GEOSFP.20110101.CN.025x03125.nc</tt>
 +
*<tt>ExtData/GEOS_0.25x0.3125_AS/GEOS_FP/2011/01/GEOSFP.20110101.CN.025x03125.AS.nc</tt>
 +
*<tt>ExtData/GEOS_0.25x0.3125_CH/GEOS_FP/2011/01/GEOSFP.20110101.CN.025x03125.CH.nc</tt>
 +
*<tt>ExtData/GEOS_0.25x0.3125_EU/GEOS_FP/2011/01/GEOSFP.20110101.CN.025x03125.EU.nc</tt>
 +
*<tt>ExtData/GEOS_0.25x0.3125_NA/GEOS_FP/2011/01/GEOSFP.20110101.CN.025x03125.NA.nc</tt>
 +
*<tt>ExtData/GEOS_2x2.5/GEOS_FP/2011/01/GEOSFP.20110101.CN.2x25.nc</tt>
 +
*<tt>ExtData/GEOS_4x5/GEOS_FP/2011/01/GEOSFP.20110101.CN.4x5.nc</tt>
 +
 +
For MERRA-2 these are timestamped for 2015/01/01 and are found in these data directories :
 +
*<tt>ExtData/GEOS_0.5x0.625/MERRA2/2015/01/MERRA2.20150101.CN.05x0625.nc4</tt>
 +
*<tt>ExtData/GEOS_0.5x0.625_AS/MERRA2/2015/01/MERRA2.20150101.CN.05x0625.AS.nc4</tt>
 +
*<tt>ExtData/GEOS_0.5x0.625_CH/MERRA2/2015/01/MERRA2.20150101.CN.05x0625.CH.nc4</tt>
 +
*<tt>ExtData/GEOS_0.5x0.625_EU/MERRA2/2015/01/MERRA2.20150101.CN.05x0625.EU.nc4</tt>
 +
*<tt>ExtData/GEOS_0.5x0.625_NA/MERRA2/2015/01/MERRA2.20150101.CN.05x0625.NA.nc4</tt>
 +
*<tt>ExtData/GEOS_2x2.5/MERRA2/2015/01/MERRA2.20150101.CN.2x25.nc4</tt>
 +
*<tt>ExtData/GEOS_4x5/MERRA2/2015/01/MERRA2.20150101.CN.4x5.nc4</tt>
 +
 +
Additional notes:
 +
*Prior to downloading GEOS-FP data, please be aware of caveats regarding use of GEOS-FP. See the [[GEOS-FP|GEOS-FP wiki page]] for more information.
  
See the [[#Directory structure|table below]] for the <tt>DIRECTORY_NAME</tt> options. The <tt>-R "*.html"</tt> option will reject HTML files.
+
--[[User:Bmy|Bob Yantosca]] ([[User talk:Bmy|talk]]) 19:20, 11 December 2019 (UTC)
  
 
== Data download commands ==
 
== Data download commands ==

Revision as of 19:20, 11 December 2019

Compute Canada directory structure

The GEOS-Chem shared data directories may be downloaded from the Compute Canada server:

http://geoschemdata.computecanada.ca

which has the following directory structure:

Directory Description
ExtData/ Root data directory containing all meteorlogy fields, emissions data, and chemistry input data.
ExtData/CHEM_INPUTS/ Contains non-emissions data for GEOS-Chem chemistry modules
ExtData/HEMCO/ Contains emissions data for the HEMCO emissions component
ExtData/GEOSCHEM_RESTARTS/ Contains sample restart files uses to initialize GEOS-Chem simulations.
0.25° x 0.3125° Data Directories Description
ExtData/GEOS_0.25x0.3125/GEOS_FP/YYYY/MM/ 0.25° x 0.3125° GEOS-FP global met fields
ExtData/GEOS_0.25x0.3125_AS/GEOS_FP/YYYY/MM/ 0.25° x 0.3125° GEOS-FP met fields cropped to the Asia domain
ExtData/GEOS_0.25x0.3125_CH/GEOS_FP/YYYY/MM/ 0.25° x 0.3125° GEOS-FP met fields cropped to the China domain
ExtData/GEOS_0.25x0.3125_EU/GEOS_FP/YYYY/MM/ 0.25° x 0.3125° GEOS-FP met fields cropped to the Europe domain
ExtData/GEOS_0.25x0.3125_NA/GEOS_FP/YYYY/MM/ 0.25° x 0.3125° GEOS-FP met fields cropped to the North America domain
0.5° x 0.625° Data Directories Description
ExtData/GEOS_0.5x0.625/MERRA2/YYYY/MM/ 0.5° x 0.625° MERRA-2 global met fields
ExtData/GEOS_0.5x0.625_AS/MERRA2/YYYY/MM/ 0.5° x 0.625° MERRA-2 met fields cropped to the Asia domain
ExtData/GEOS_0.5x0.625_CH/MERRA2/YYYY/MM/ 0.5° x 0.625° MERRA-2 met fields cropped to the China domain
ExtData/GEOS_0.5x0.625_EY/MERRA2/YYYY/MM/ 0.5° x 0.625° MERRA-2 met fields cropped to the Europe domain
ExtData/GEOS_0.5x0.625_NA/MERRA2/YYYY/MM/ 0.5° x 0.625° MERRA-2 met fields cropped to the North America domain
2° x 2.5° Data Directories Description
ExtData/GEOS_2x2.5/GEOS_FP/YYYY/MM 2° x 2.5° GEOS-FP global met fields
ExtData/GEOS_2x2.5/MERRA2/YYYY/MM 2° x 2.5° MERRA-2 global met fields
4° x 5° Data Directories Description
ExtData/GEOS_4x5/GEOS_FP/YYYY/MM/ 4° x 5° GEOS-FP global met fields
ExtData/GEOS_4x5/MERRA2/YYYY/MM/ 4° x 5° MERRA-2 global met fields

GEOS-FP and MERRA-2 constant data files

If you are downloading the GEOS-FP or MERRA-2 met data, then please note that you must also download the "CN" (constant) data files for each horizontal grid that you are using.

For GEOS-FP these are timestamped for 2011/01/01 and are found in these data directories:

  • ExtData/GEOS_0.25x0.3125/GEOS_FP/2011/01/GEOSFP.20110101.CN.025x03125.nc
  • ExtData/GEOS_0.25x0.3125_AS/GEOS_FP/2011/01/GEOSFP.20110101.CN.025x03125.AS.nc
  • ExtData/GEOS_0.25x0.3125_CH/GEOS_FP/2011/01/GEOSFP.20110101.CN.025x03125.CH.nc
  • ExtData/GEOS_0.25x0.3125_EU/GEOS_FP/2011/01/GEOSFP.20110101.CN.025x03125.EU.nc
  • ExtData/GEOS_0.25x0.3125_NA/GEOS_FP/2011/01/GEOSFP.20110101.CN.025x03125.NA.nc
  • ExtData/GEOS_2x2.5/GEOS_FP/2011/01/GEOSFP.20110101.CN.2x25.nc
  • ExtData/GEOS_4x5/GEOS_FP/2011/01/GEOSFP.20110101.CN.4x5.nc

For MERRA-2 these are timestamped for 2015/01/01 and are found in these data directories :

  • ExtData/GEOS_0.5x0.625/MERRA2/2015/01/MERRA2.20150101.CN.05x0625.nc4
  • ExtData/GEOS_0.5x0.625_AS/MERRA2/2015/01/MERRA2.20150101.CN.05x0625.AS.nc4
  • ExtData/GEOS_0.5x0.625_CH/MERRA2/2015/01/MERRA2.20150101.CN.05x0625.CH.nc4
  • ExtData/GEOS_0.5x0.625_EU/MERRA2/2015/01/MERRA2.20150101.CN.05x0625.EU.nc4
  • ExtData/GEOS_0.5x0.625_NA/MERRA2/2015/01/MERRA2.20150101.CN.05x0625.NA.nc4
  • ExtData/GEOS_2x2.5/MERRA2/2015/01/MERRA2.20150101.CN.2x25.nc4
  • ExtData/GEOS_4x5/MERRA2/2015/01/MERRA2.20150101.CN.4x5.nc4

Additional notes:

  • Prior to downloading GEOS-FP data, please be aware of caveats regarding use of GEOS-FP. See the GEOS-FP wiki page for more information.

--Bob Yantosca (talk) 19:20, 11 December 2019 (UTC)

Data download commands

We recommend that you use the wget utility to download data from Compute Canada.

For example, this command will download an entire directory (and its subdirectories) from Compute Canada to your current folder:

wget -r -np -nH -R "*.html" "http://geoschemdata.computecanada.ca/ExtData/DIRECTORY_NAME"

Or if you wish to download the folder to a different directory:

wget -r -np -nH -R "*.html" -P /your/data/root "http://geoschemdata.computecanada.ca/ExtData/DIRECTORY_NAME"

The options to wget are as follows:

-r   Specifies recursive directory transfer (i.e. will download all subdirectories)
-np Will not allow ascent to the parent directory 
-nH  Will store all directories and subdirectories in DIRECTORY_NAME, not geoschemdata.computecanada.ca/DIRECTORY_NAME
-R "*.html" Will reject any file ending in .html

If you wish to trim the name of the downloaded directory (i.e., so it downloads as DIRECTORY_NAME, not pub/geos-chem/data/DIRECTORY_NAME), then use the --cut-dirs option:

wget -r -np -nH -R "*.html" --cut-dirs=X "http://geoschemdata.computecanada.ca/DIRECTORY_NAME/"

where X is the number of directories to trim.

NOTE: The URL must be enclosed in quotes for file transfer to occur. If you omit the quotes then wget will just return a directory listing in a file named index.html without any files being downloaded.

--Bob Yantosca (talk) 19:15, 11 December 2019 (UTC)