Difference between revisions of "Downloading data from WashU"

From Geos-chem
Jump to: navigation, search
(Basic syntax)
 
(27 intermediate revisions by 3 users not shown)
Line 1: Line 1:
 
+
This content has been migrated to the [https://geos-chem.readthedocs.io/en/latest/gcc-guide/04-data/download-data.html '''Download input data''' chapter of <tt>geos-chem.readthedocs.io</tt>].
== Compute Canada directory structure ==
+
 
+
The GEOS-Chem shared data directories may be downloaded from the Compute Canada server:
+
 
+
http://geoschemdata.computecanada.ca
+
 
+
which has the following directory structure:
+
 
+
{| border=1 cellpadding=5 cellspacing=0
+
|-bgcolor="#CCCCCC"
+
!width="400px"|Directory
+
!width="600px"|Description                               
+
 
+
|-valign="top"
+
|<tt>ExtData/</tt>                             
+
|Root data directory containing all meteorlogy fields, emissions data, and chemistry input data.
+
 
+
|-valign="top"
+
|<tt>ExtData/CHEM_INPUTS/</tt>                     
+
|Contains non-emissions data for GEOS-Chem chemistry modules
+
 
+
|-valign="top"                         
+
|<tt>ExtData/HEMCO/</tt>                 
+
|Contains [[HEMCO_data_directories|emissions data]] for the [[HEMCO|HEMCO emissions component]]
+
 
+
|-valign="top"                         
+
|<tt>ExtData/GEOSCHEM_RESTARTS/</tt>                 
+
|Contains sample [[GEOS-Chem_restart_files|restart files]] uses to initialize GEOS-Chem simulations.
+
 
+
|-valign="top" bgcolor="#CCCCCC"                       
+
!0.25&deg; x 0.3125&deg; Data Directories                 
+
!Description     
+
 
+
|-valign="top"
+
|<tt>ExtData/GEOS_0.25x0.3125/GEOS_FP/YYYY/MM/</tt>
+
|0.25&deg; x 0.3125&deg; [[GEOS-FP]] global met fields
+
 
+
|-valign="top"                                                 
+
|<tt>ExtData/GEOS_0.25x0.3125_AS/GEOS_FP/YYYY/MM/</tt>
+
|0.25&deg; x 0.3125&deg; [[GEOS-FP]] met fields cropped to the [[GEOS-Chem horizontal grids#GMAO 0.5 x 0.625 AS nested grid|Asia domain]]
+
 
+
|-valign="top"                                                 
+
|<tt>ExtData/GEOS_0.25x0.3125_CH/GEOS_FP/YYYY/MM/</tt>
+
|0.25&deg; x 0.3125&deg; [[GEOS-FP]] met fields cropped to the [[GEOS-Chem horizontal grids#GMAO 0.25 x 0.3125 CH nested grid|China domain]]
+
 
+
|-valign="top"                                                 
+
|<tt>ExtData/GEOS_0.25x0.3125_EU/GEOS_FP/YYYY/MM/</tt>
+
|0.25&deg; x 0.3125&deg; [[GEOS-FP]] met fields cropped to the [[GEOS-Chem horizontal grids#GMAO 0.25 x 0.3125 EU nested grid|Europe domain]]
+
 
+
|-valign="top"                                                 
+
|<tt>ExtData/GEOS_0.25x0.3125_NA/GEOS_FP/YYYY/MM/</tt>
+
|0.25&deg; x 0.3125&deg; [[GEOS-FP]] met fields cropped to the [[GEOS-Chem horizontal grids#GMAO 0.25 x 0.3125 NA nested grid|North America domain]]
+
 
+
|-valign="top" bgcolor="#CCCCCC"                       
+
!0.5&deg; x 0.625&deg; Data Directories                 
+
!Description
+
 
+
|-valign="top"
+
|<tt>ExtData/GEOS_0.5x0.625/MERRA2/YYYY/MM/</tt>
+
|0.5&deg; x 0.625&deg; [[MERRA-2]] global met fields
+
 
+
|-valign="top"                                                 
+
|<tt>ExtData/GEOS_0.5x0.625_AS/MERRA2/YYYY/MM/</tt>
+
|0.5&deg; x 0.625&deg; [[MERRA-2]] met fields cropped to the [[GEOS-Chem horizontal grids#GMAO 0.5 x 0.625 AS nested grid|Asia domain]]
+
 
+
|-valign="top"                                                 
+
|<tt>ExtData/GEOS_0.5x0.625_CH/MERRA2/YYYY/MM/</tt>
+
|0.5&deg; x 0.625&deg; [[MERRA-2]] met fields cropped to the [[GEOS-Chem horizontal grids#GMAO 0.25 x 0.3125 CH nested grid|China domain]]
+
 
+
|-valign="top"                                                 
+
|<tt>ExtData/GEOS_0.5x0.625_EY/MERRA2/YYYY/MM/</tt>
+
|0.5&deg; x 0.625&deg; [[MERRA-2]] met fields cropped to the [[GEOS-Chem horizontal grids#GMAO 0.5 x 0.625 EU nested grid|Europe domain]]
+
 
+
|-valign="top"                                                 
+
|<tt>ExtData/GEOS_0.5x0.625_NA/MERRA2/YYYY/MM/</tt>
+
|0.5&deg; x 0.625&deg; [[MERRA-2]] met fields cropped to the [[GEOS-Chem horizontal grids#GMAO 0.5 x 0.625 NA nested grid|North America domain]]
+
 
+
|-bgcolor="#CCCCCC"                       
+
!2&deg; x 2.5&deg; Data Directories               
+
!Description
+
 
+
|-valign="top" 
+
|<tt>ExtData/GEOS_2x2.5/GEOS_FP/YYYY/MM</tt>
+
|[[GEOS-Chem horizontal grids#GMAO 2 x 2.5 grid|2&deg; x 2.5&deg;]] [[GEOS-FP]] global met fields
+
 
+
|-valign="top" 
+
|<tt>ExtData/GEOS_2x2.5/MERRA2/YYYY/MM</tt>
+
|[[GEOS-Chem horizontal grids#GMAO 2 x 2.5 grid|2&deg; x 2.5&deg;]] [[MERRA-2]] global met fields
+
 
+
|-valign="top" bgcolor="#CCCCCC"                       
+
!4&deg; x 5&deg; Data Directories               
+
!Description
+
 
+
|-valign="top" 
+
|<tt>ExtData/GEOS_4x5/GEOS_FP/YYYY/MM/</tt> 
+
|[[GEOS-Chem horizontal grids#GMAO 4 x 5 grid|4&deg; x 5&deg;]] [[GEOS-FP]] global met fields
+
 
+
|-valign="top" 
+
|<tt>ExtData/GEOS_4x5/MERRA2/YYYY/MM/</tt> 
+
|[[GEOS-Chem horizontal grids#GMAO 4 x 5 grid|4&deg; x 5&deg;]] [[MERRA-2]] global met fields
+
|}
+
 
+
--[[User:Bmy|Bob Yantosca]] ([[User talk:Bmy|talk]]) 19:21, 11 December 2019 (UTC)
+
 
+
=== GEOS-FP and MERRA-2 constant data files ===
+
 
+
If you are downloading the GEOS-FP or MERRA-2 met data, then please note that you must also download the "CN" (constant) data files for each horizontal grid that you are using. 
+
 
+
For GEOS-FP these are timestamped for 2011/01/01 and are found in these data directories:
+
*<tt>ExtData/GEOS_0.25x0.3125/GEOS_FP/2011/01/GEOSFP.20110101.CN.025x03125.nc</tt>
+
*<tt>ExtData/GEOS_0.25x0.3125_AS/GEOS_FP/2011/01/GEOSFP.20110101.CN.025x03125.AS.nc</tt>
+
*<tt>ExtData/GEOS_0.25x0.3125_CH/GEOS_FP/2011/01/GEOSFP.20110101.CN.025x03125.CH.nc</tt>
+
*<tt>ExtData/GEOS_0.25x0.3125_EU/GEOS_FP/2011/01/GEOSFP.20110101.CN.025x03125.EU.nc</tt>
+
*<tt>ExtData/GEOS_0.25x0.3125_NA/GEOS_FP/2011/01/GEOSFP.20110101.CN.025x03125.NA.nc</tt>
+
*<tt>ExtData/GEOS_2x2.5/GEOS_FP/2011/01/GEOSFP.20110101.CN.2x25.nc</tt>
+
*<tt>ExtData/GEOS_4x5/GEOS_FP/2011/01/GEOSFP.20110101.CN.4x5.nc</tt>
+
 
+
For MERRA-2 these are timestamped for 2015/01/01 and are found in these data directories :
+
*<tt>ExtData/GEOS_0.5x0.625/MERRA2/2015/01/MERRA2.20150101.CN.05x0625.nc4</tt>
+
*<tt>ExtData/GEOS_0.5x0.625_AS/MERRA2/2015/01/MERRA2.20150101.CN.05x0625.AS.nc4</tt>
+
*<tt>ExtData/GEOS_0.5x0.625_CH/MERRA2/2015/01/MERRA2.20150101.CN.05x0625.CH.nc4</tt>
+
*<tt>ExtData/GEOS_0.5x0.625_EU/MERRA2/2015/01/MERRA2.20150101.CN.05x0625.EU.nc4</tt>
+
*<tt>ExtData/GEOS_0.5x0.625_NA/MERRA2/2015/01/MERRA2.20150101.CN.05x0625.NA.nc4</tt>
+
*<tt>ExtData/GEOS_2x2.5/MERRA2/2015/01/MERRA2.20150101.CN.2x25.nc4</tt>
+
*<tt>ExtData/GEOS_4x5/MERRA2/2015/01/MERRA2.20150101.CN.4x5.nc4</tt>
+
 
+
Additional notes:
+
*Prior to downloading GEOS-FP data, please be aware of caveats regarding use of GEOS-FP. See the [[GEOS-FP|GEOS-FP wiki page]] for more information.
+
 
+
--[[User:Bmy|Bob Yantosca]] ([[User talk:Bmy|talk]]) 19:20, 11 December 2019 (UTC)
+
 
+
== Data download commands ==
+
 
+
We recommend that you use the free and open-source [https://www.gnu.org/software/wget wget utility] to download data from Compute Canada.  Most modern Unix systems have wget already installed.
+
 
+
=== Basic syntax ===
+
 
+
The basic formula to download data from Compute Canada to your local server is:
+
 
+
<nowiki>wget OPTIONS "http://geoschemdata.computecanada.ca/ExtData/DIRECTORY_NAME"</nowiki>
+
 
+
Commonly used options with wget are:
+
 
+
{| border=1 cellpadding=5 cellspacing=0
+
|-bgcolor="#CCCCCC"
+
!width="100px"|wget option
+
!width="900px"|Description                               
+
                         
+
|-valign="top"
+
|<tt>-np</tt>
+
|Will not allow ascent to the parent directory
+
 
+
|-valign="top"
+
|<tt>-nH</tt>
+
|Omits the remote root directory name from the local directory name.
+
*i.e. Downloads <tt>geoschemdata.computecanada.ca/ExtData/DIRECTORY_NAME</tt> to local folder <tt>ExtData/DIRECTORY_NAME</tt>.
+
 
+
|-valign="top"
+
|<tt>-N</tt>
+
|Downloads only those files having newer timestamps than any local copies.
+
 
+
|-valign="top"
+
|<tt>-P path</tt>
+
|Copies data to the specified directory
+
*e.g. Specifying <tt>-P /home/data</tt> will copy <tt>geoschemdata.computecanada.ca/ExtData/DIRECTORY_NAME</tt><br>to <tt>/home/data/ExtData/DIRECTORY_NAME</tt>, etc.
+
 
+
|-valign="top"
+
|<tt>-r</tt>
+
|Specifies recursive directory transfer (i.e. will download all subdirectories).
+
 
+
|-valign="top"
+
|<tt>-R "*.html"</tt>
+
|Skips downloading files ending in <tt>*.html</tt>.
+
 
+
|}
+
 
+
=== Example 1 ===
+
 
+
For example, this command will download an entire directory (and its subdirectories) from Compute Canada to your current folder:
+
 
+
<nowiki>wget -r -np -nH -R "*.html" "http://geoschemdata.computecanada.ca/ExtData/DIRECTORY_NAME"</nowiki>
+
 
+
Or if you wish to download the folder to a different directory:
+
 
+
<nowiki>wget -r -np -nH -R "*.html" -P /your/data/root "http://geoschemdata.computecanada.ca/ExtData/DIRECTORY_NAME"</nowiki>
+
 
+
The options to <tt>wget</tt> are as follows:
+
 
+
-r  Specifies recursive directory transfer (i.e. will download all subdirectories)
+
-np Will not allow ascent to the parent directory
+
-nH  Will store all directories and subdirectories in <tt>DIRECTORY_NAME</tt>, not <tt>geoschemdata.computecanada.ca/DIRECTORY_NAME</tt>
+
-R "*.html" Will reject any file ending in .html
+
-P specifies a local directory prefix
+
-N tells wget to only download files having newer timestamps rather than all files.
+
 
+
If you wish to trim the name of the downloaded directory (i.e., so it downloads as <tt>DIRECTORY_NAME</tt>, not <tt>pub/geos-chem/data/DIRECTORY_NAME</tt>), then use the <tt>--cut-dirs</tt> option:
+
+
<nowiki>wget -r -np -nH -R "*.html" --cut-dirs=X "http://geoschemdata.computecanada.ca/DIRECTORY_NAME/"</nowiki>
+
 
+
where <tt>X</tt> is the number of directories to trim.
+
 
+
<span style="color:red">'''''NOTE: The URL must be enclosed in quotes for file transfer to occur. If you omit the quotes then <tt>wget</tt> will just return a directory listing in a file named <tt>index.html</tt> without any files being downloaded.'''''</span>
+
 
+
Prasad Kasibhatla wrote:
+
 
+
    Maybe this is common knowledge, but I just discovered that using the -N option in wget ensures that only files with newer timestamps than what resides on my local machines are downloaded - found this very useful to update my shared data directories.
+
 
+
--[[User:Bmy|Bob Yantosca]] ([[User talk:Bmy|talk]]) 19:15, 11 December 2019 (UTC)
+

Latest revision as of 20:39, 3 August 2022

This content has been migrated to the Download input data chapter of geos-chem.readthedocs.io.