View source for Downloading data with the GEOS-Chem dry-run option

__FORCETOC__
'''''[[Downloading GEOS-Chem data directories|Previous]] | [[Downloading data from Compute Canada|Next]] | [[Getting Started with GEOS-Chem]]'''''
#[[Minimum system requirements for GEOS-Chem|Minimum system requirements]]
#[[Configuring your computational environment]]
#[[Downloading GEOS-Chem source code|Downloading source code]]
#[[Downloading GEOS-Chem data directories|Downloading data directories]]
#*<span style="color:blue">'''... with the GEOS-Chem dry-run option'''</span>
#*[[Downloading data from Compute Canada|... from Compute Canada]]
#*[[Downloading data from Amazon Web Services cloud storage|... from Amazon Web Services cloud storage]]
#[[Creating GEOS-Chem run directories|Creating run directories]]
#[[GEOS-Chem configuration files|Configuring runs]]
#[[Compiling GEOS-Chem|Compiling]]
#[[Running GEOS-Chem|Running]]
#[[GEOS-Chem output files|Output files]]
#[[Guide_to_visualization_and_analysis_tools_for_GEOS-Chem|Visualizing and processing output]]
#[[GEOS-Chem_coding_and_debugging|Coding and debugging]]
#[[GEOS-Chem_overview#Further_reading|Further reading]]


== Overview ==

The GEOS-Chem dry-run option will be available in [[GEOS-Chem 12#12.7.0|GEOS-Chem 12.7.0]] and later versions.  This version should be released by mid-to-late January 2020 (pending successful benchmarking).

=== What is a GEOS-Chem dry-run? ===

A "dry-run" is a is a GEOS-Chem "Classic" simulation that steps through time, but does not perform computations or read data files from disk.  Instead, a dry-run simulation prints a list of all data files that a regular GEOS-Chem simulation would have read.  The dry-run output also denotes whether each data file was found on disk, or if it is missing.

=== Why should I perform a GEOS-Chem dry-run? ===

A GEOS-Chem dry-run is a good way for you to check if you have [[Configuring your computational environment|properly configured your computational environment]].  This is especially important if you are porting GEOS-Chem to run on a new computer system, or on the [http://cloud.geos-chem.org AWS cloud].

For example:

{| border=1 cellpadding=5 cellspacing=0
|-bgcolor="#CCCCCC"
!width="350px"|Problem encountered in dry-run
!width="650px"|What this can indicate
                           
|-valign="top"
|GEOS-Chem "Classic" does not compile
|
*Your system does not have a supported compiler
*Your system is missing one or more required software libraries
*You have invalid settings in your Unix startup scripts (e.g. <tt>.bashrc</tt>, etc.)

|-valign="top"
|GEOS-Chem "Classic" compiles but does not run
|
*Your GEOS-Chem configuration files (<tt>input.geos</tt>, <tt>HEMCO_Config.rc</tt>, etc.) contain errors
*You have invalid settings in your Unix startup scripts (e.g. <tt>.bashrc</tt>, etc.)

|} 

Once you have can successfully finished a GEOS-Chem dry-run, you can have confidence that GEOS-Chem can run on your system.

More importantly, output from the GEOS-Chem dry-run simulation can be used to download required met field and emissions data for a GEOS-Chem simulation (see next section).

=== What can I do with the output of a GEOS-Chem dry-run? ===

When you run GEOS-Chem in dry-run mode, you must pipe the output to a log file (as you would do for any other GEOS-Chem simulation).  [[GEOS-Chem_log_files#GEOS-Chem_log_file_with_dry-run_output|The log file containing dry-run output]] looks similar to a regular GEOS-Chem log file, but will also contain text such as:

 ...
 HEMCO: Opening /path/to/ExtData/HEMCO/EDGARv43/v2016-11/EDGAR_v43.BC.POW.0.1x0.1.nc
 ...
 HEMCO: REQUIRED FILE NOT FOUND /path/to/ExtData/HEMCO/EDGARv43/v2016-11/EDGAR_v43.BC.POW.0.1x0.1.nc
 ...

''NOTE: <tt>/path/to/ExtData</tt> denotes the full pathname of the <tt>ExtData</tt> folder. This is the root of the directory tree containing all GEOS-Chem met fields and emissions data.  This will of course be different on each system.''

This text lets you know if GEOS-Chem was able to find each input file on disk or not.  This information can be parsed by a Python script (<tt>download_data.py</tt>, which is included in each run directory) to produce:

# A unique list of required data files (with all duplicates removed).  This can be useful for documentation purposes.
# A bash script that will download all MISSING data files from one of the GEOS-Chem data repositories:
#*Compute Canada repository
#*Amazon Web Services s3://gcgrid repository

In the following sections, we will describe the commands that are needed to execute a complete GEOS-chem dry-run workflow.

=== Which GEOS-Chem simulations can execute in dry-run mode? ===

Any of the supported [[Guide_to_GEOS-Chem_simulations|GEOS-Chem "Classic" simulations]] (e.g. standard, benchmark, complexSOA, CH4, TransportTracers, etc.) can be executed in dry-run mode.

As of this writing (Jan 2020), [[GEOS-Chem HP|GCHP]] cannot execute in dry-run mode.  Dry-run functionality may be added in the future, but this will require modifications to the NASA MAPL software library. Because GCHP uses the same data files as GEOS-Chem "Classic", you can use a GEOS-Chem "Classic" dry-run to facilitate downloading met fields and emissions data for a GCHP simulation.

--[[User:Bmy|Bob Yantosca]] ([[User talk:Bmy|talk]]) 17:00, 6 January 2020 (UTC)

== Executing GEOS-Chem in dry-run mode ==

Follow these steps to perform a GEOS-Chem dry-run:

#<p>Create a run directory for the type of GEOS-Chem simulation that you wish to perform (e.g. <tt>geosfp_4x5_standard</tt>, <tt>merra2_2x25_tropchem</tt>, etc..  For detailed instructions, please see our [[Creating_GEOS-Chem_run_directories|our ''Creating run directories'' chapter]].</p>
#<p>Change to the run directory (e.g. <tt>cd geosfp_4x5_standard</tt>, etc.) and compile GEOS-Chem as you normally would.  For detailed instructions, please see our [[Compiling GEOS-Chem|our ''Compiling'' chapter]].</p>
#<p>Check to see if these settings in [[The input.geos file#Simulation|the SIMULATION MENU]] and [[The input.geos file#Grid|the GRID MENU]] of the <tt>input.geos</tt> file are as you expect:<blockquote>[[Image:Config4.png]]</blockquote></p>
#<p>Make sure that the ROOT and METDIR settings in HEMCO_Config.rc use the same root data directory as is specified in <tt>input.geos</tt>:<blockquote>[[Image:Config5.png]]</blockquote></p>
#<p>Make sure to select the emission inventories and extensions that you wish to use for your simulation:<blockquote>[[Image:Config3.png]]<br>etc...</blockquote>You can reduce the amount of data that needs to be downloaded for your GEOS-Chem simulation by turning off inventories that you don't need.</p>
#<p>Run GEOS-Chem with the <tt>--dryrun</tt> command-line argument, and pipe the output to a log file.  You can type either:<blockquote><tt>./geos --dryrun > log.dryrun</tt></blockquote>which will send the ouptut to the <tt>log.dryrun</tt> file, or:<blockquote><tt>./geos --dryrun tee log.dryrun</tt></blockquote>which will not only will pipe the output to <tt>log.dryrun</tt>, but will also show the output on the screen.</p><p>The <tt>log.dryrun</tt> file will look somewhat like a regular GEOS-Chem log file but will also contain [[GEOS-Chem_log_files#GEOS-Chem_log_file_with_dry-run_output|a list of files and whether each file was found on disk or not]].</p><p>Also note, you can use whatever name you like for the dry-run output log file (we prefer <tt>log.dryrun</tt>).</p>

--[[User:Bmy|Bob Yantosca]] ([[User talk:Bmy|talk]]) 17:04, 6 January 2020 (UTC)

== Downloading data from dry-run output ==

Once you have successfully executed a GEOS-Chem dry-run ([[#Executing GEOS-Chem in dry-run mode|see previous section]]), you can use the output from the dry-run (contained in the <tt>log.dryrun</tt> file) to download the data files that GEOS-Chem will need to perform the corresponding "production" simulation.  Follow one of these three options:

=== Downloading data from Compute Canada ===

If you are using GEOS-Chem on your institutional computer cluster, we recommend that you '''download data from the Compute Canada data repository''' (<tt>http://geoschemdata.computecanada.ca</tt>). From the GEOS-Chem run directory, run the Python program <tt>download_data.py</tt> as follows:

 ./download_data.py log.dryrun --cc

The <tt>download_data.py</tt> is included in each GEOS-Chem run directory that you create (for [[GEOS-Chem 12#12.7.0|GEOS-Chem 12.7.0]] and later versions).  It uses base Python 3 packages, and does not need to be run from within a Conda environment.  This Python program creates and executes a temporary bash script containing the appropriate <tt>wget</tt> commands to download the data files.  (We have found that this is the fastest method.)

The <tt>download_data.py</tt> program will also generate a '''log of unique data files''' (i.e. with all duplicate listings removed), which looks similar to this:

 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
 !!! LIST OF (UNIQUE) FILES REQUIRED FOR THE SIMULATION
 !!! Start Date       : 20160701 000000
 !!! End Date         : 20160701 010000
 !!! Simulation       : standard
 !!! Meteorology      : GEOSFP
 !!! Grid Resolution  : 4.0x5.0
 !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
 ./GEOSChem.Restart.20160701_0000z.nc4 --> /n/holylfs/EXTERNAL_REPOS/GEOS-CHEM/gcgrid/data/ExtData/GEOSCHEM_RESTARTS/v2018-11/initial_GEOSChem_rst.4x5_standard.nc
 ./HEMCO_Config.rc
 ./HEMCO_Diagn.rc
 ./HEMCO_restart.201607010000.nc
 ./HISTORY.rc
 ./input.geos
 /n/holylfs/EXTERNAL_REPOS/GEOS-CHEM/gcgrid/data/ExtData/CHEM_INPUTS/FAST_JX/v2019-10/FJX_j2j.dat
 /n/holylfs/EXTERNAL_REPOS/GEOS-CHEM/gcgrid/data/ExtData/CHEM_INPUTS/FAST_JX/v2019-10/FJX_spec.dat
 /n/holylfs/EXTERNAL_REPOS/GEOS-CHEM/gcgrid/data/ExtData/CHEM_INPUTS/FAST_JX/v2019-10/dust.dat
 /n/holylfs/EXTERNAL_REPOS/GEOS-CHEM/gcgrid/data/ExtData/CHEM_INPUTS/FAST_JX/v2019-10/h2so4.dat
 /n/holylfs/EXTERNAL_REPOS/GEOS-CHEM/gcgrid/data/ExtData/CHEM_INPUTS/FAST_JX/v2019-10/jv_spec_mie.dat
 ... etc ...

This name of this "unique" log file will be the same as the log file with dryrun ouptut, with <tt>.unique</tt> appended. In our above example, we passed <tt>log.dryrun</tt> to <tt>download_data.py</tt>, so the "unique" log file will be named <tt>log.dryrun.unique</tt>.   This "unique" log file can be very useful for documentation purposes.

--[[User:Bmy|Bob Yantosca]] ([[User talk:Bmy|talk]]) 17:47, 6 January 2020 (UTC)

=== Downloading data from AWS s3://gcgrid ===

If you are running GEOS-Chem on the Amazon Web Services cloud, you can quickly '''download the necessary data for your GEOS-Chem simulation from the <tt>s3://gcgrid</tt> bucket''' to the Elastic Block Storage (EBS) volume attached to your cloud instance.  Change to your GEOS-Chem run directory and type:

 ./download_data.py log.dryrun --aws

This will start the data download process using the <tt>aws s3 cp</tt> commands, which should execute much more quickly than if you were to download the data from Compute Canada.  It will also produce the '''log of unique data files''' [[#Downloading data from Compute Canada|as described in the previous section]].

<span style="color:red">'''NOTE: Copying data from s3://gcgrid to the EBS volume of an AWS cloud instance is always free.  But if you download data from <tt>s3://gcgrid</tt> to your own local computer cluster, you will incur an egress fee (~ $90/TB).  Use with caution!'''</span>

--[[User:Bmy|Bob Yantosca]] ([[User talk:Bmy|talk]]) 18:02, 6 January 2020 (UTC)

=== Create the log of unique data files without downloading data ===

If you wish to only produce the '''log of unique data files''' ([[#Downloading data from Compute Canada|as described above]]) without downloading any data, then type the following command from within your GEOS-Chem run directory:

 ./download_data.py log.dryrun --skip-download

or for short

 ./download_data.py log.dryrun --skip

This can be useful if you already have the necessary data downloaded to your system but wish to create the log of unique files for documentation purposes (such as for benchmark simulations, etc.)

--[[User:Bmy|Bob Yantosca]] ([[User talk:Bmy|talk]]) 18:05, 6 January 2020 (UTC)

== Further reading ==
# [http://cloud.geos-chem.org Using GEOS-Chem on the AWS cloud ('''cloud.geos-chem.org''')]


----
'''''[[Downloading GEOS-Chem data directories|Previous]] | [[Downloading data from Compute Canada|Next]] | [[Getting Started with GEOS-Chem]]'''''