Difference between revisions of "GCHP Main Page"

From Geos-chem
Jump to: navigation, search
(GCHP Wiki Guide)
(Featured News)
 
(48 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 +
'''''[[Main_Page|GEOS-Chem Main Page]]'''''
 +
 
== About High Performance GEOS-Chem ==
 
== About High Performance GEOS-Chem ==
  
GEOS-Chem with the high performance option (GCHP) is the next generation of GEOS-Chem. It uses a distributed memory capability designed to enable efficient scaling across many cores, and thus enable finer resolution simulations. GCHP has the advantage of using a cubed sphere geometry that enables more accurate transport and eliminates the polar singularity inherent to lat-lon grids. Despite these advances you can still run the model on a single machine at coarse resolution if you choose to do so.
+
GEOS-Chem with the high performance option (GCHP) is the next generation of GEOS-Chem. It uses a distributed memory capability designed to enable efficient scaling across many cores, and thus enable finer resolution simulations. GCHP has the advantage of using a cubed sphere geometry that enables more accurate transport and eliminates the polar singularity inherent to lat-lon grids. With these advances you can still run the model on a single machine at coarse resolution if you choose to do so.
  
If you are working on a project using GCHP, please add your name and project description to the project table on the GEOS-Chem High Performance Working Group wiki page (see link below). If you would like to stay informed of GCHP developments, please join the GEOS-Chem High Performance Working Group mailing list. We also encourage you to join our GCHP Slack workspace.
+
If you are working on a project using GCHP, please add your name and project description to the [[GEOS-Chem_High_Performance_Working_Group#Current_GCHP_Projectss|GCHP projects list]]. Also help the GCHP Working Group pool performance information across systems by contributing your GCHP run information on the [[GCHP_Timing_Tests|GCHP Timing Tests page]].  
  
Please contact the [[GEOS-Chem_Support_Team|GEOS-Chem Support Team]] with GCHP questions and feedback.
+
If you would like to stay informed of GCHP developments, please join the [[GEOS-Chem_High_Performance_Working_Group#Email|GEOS-Chem High Performance Working Group mailing list]] and subscribe to [https://help.github.com/en/articles/watching-and-unwatching-repositories GCHP notifications] on [https://github.com/geoschem/gchp GitHub]. You may also join the [[GEOS-Chem_High_Performance_Working_Group#Slack|GCHP Slack workspace]] to interact with other users. If you would like receive help or participate in conversations about GCHP, please use the [https://github.com/geoschem/gchp/issues GCHP GitHub issue tracker]. You may also contact the [[GEOS-Chem_Support_Team|GEOS-Chem Support Team]] if you have questions or need assistance, or reach out to the [[GEOS-Chem_High_Performance_Working_Group#Email|GCHP Working Group Co-Chairs]].
  
 
== GCHP Wiki Guide ==
 
== GCHP Wiki Guide ==
  
'''Getting Started with GCHP'''
+
'''Getting Started'''
*[[Getting_Started|GCHP Online Manual]]
+
*[[Getting_Started_With_GCHP|Online Manual]]
'''Version History and Validation'''
+
*[[Getting_Started_with_GCHP#Recent_Changes|Recent Changes]]
*[[GEOS-Chem_HP_v11-02|GEOS-Chem HP v11-02/12.0.0]]
+
'''Version Information'''
*[[GEOS-Chem_v11-02_benchmark_history|v11-02 Benchmarks]]
+
*[[GEOS-Chem_12|Current Version]]
*[[GEOS-Chem_model_development_priorities|GEOS-Chem Model Development Priorities]]
+
*[[GEOS-Chem_newsletters|Newsletters]]
 +
*[[GEOS-Chem_model_development_priorities|Model Development Priorities]]
 +
*[[GCHP_v11-02|Older Versions (v11-02 and prior)]]
 +
'''Validation and Performance'''
 +
*[[GEOS-Chem_Benchmarks|Benchmarks]]
 +
*[[GCHP_Timing_Tests|Performance Information]] (add your run info here!)
 +
*[[GEOS-Chem_v11-02_benchmark_history|Validation of early versions]]
 
'''Community'''
 
'''Community'''
 
*[[GEOS-Chem_High_Performance_Working_Group|GCHP Working Group]]
 
*[[GEOS-Chem_High_Performance_Working_Group|GCHP Working Group]]
*[[GEOS-Chem_High_Performance_Working_Group#Slack|GCHP on Slack]]
+
*[https://github.com/geoschem/gchp/issues GCHP GitHub Issue Tracker]
 +
*[[GEOS-Chem_High_Performance_Working_Group#Slack|Slack Workspace]]
 +
'''Troubleshooting'''
 +
*[[Running_GCHP:_Basics#Verifying_a_Successful_Run|Verifying a Successful Run]]
 +
*[[GEOS-Chem_12#Outstanding_issues_not_yet_resolved_in_GEOS-Chem_12|GEOS-Chem 12 known issues and bugs]]
 
'''Help Resources and Quick Links'''
 
'''Help Resources and Quick Links'''
 +
*[http://acmg.seas.harvard.edu/presentations/IGC9/talks/MonD_Clinics_lundgren_lizzie_2_mac.pptx IGC9 GCHP Model Clinic]
 
*[[Getting_Started_With_GCHP#Frequently_Asked_Questions|Frequently Asked Questions]]
 
*[[Getting_Started_With_GCHP#Frequently_Asked_Questions|Frequently Asked Questions]]
*[[GCHP_Basic_Example_Run#Verifying_a_Successful_Run|Verifying a Successful Run]]
+
*[[GCHP_Output_Data#Data_Visualization_and_Post-Processing|Tools for Regridding and Data Analysis]]
*[[GEOS-Chem_HP_Output_Data#Data_Visualization_and_Post-Processing|Tools for Regridding and Post-processing Output]]
+
*[http://acmg.seas.harvard.edu/forum_files/git_presentation_GSF_26Apr2018.pdf Introduction to git presentation by Lizzie Lundgren]
*[http://acmg.seas.harvard.edu/forum_files/git_presentation_GSF_26Apr2018.pdf Introduction to git]
+
 
 +
== Featured News ==
 +
 
 +
'''Speed improvements in GCHP 12.5.0'''
 +
 
 +
GCHP 12.5.0 uses MAPL version tag "gchp/v1.0.0" released in early 2019. The MAPL library, which is developed by NASA, is a wrapper for the Earth System Modeling Framework (ESMF), and handles not only I/O but also passing of data between components. With this version update GCHP input speed executes roughly 3-4 times faster than previous versions! This speed-up is due to parallelization within input read across one node, and the MAPL updates in this version are just the beginning of improvements to GCHP I/O bottlenecks.
 +
 
 +
'''Important Notes about GCHP 12.5.0'''
 +
 
 +
While we are encouraged by the improvements to GCHP in 12.5.0, a few issues remain open until the next MAPL update later this year.
 +
 
 +
*GCHP 12.5.0 currently has a slow memory leak due to updates to MAPL ExtData. NASA is aware of the issue and will be developing a fix soon. The memory leak is more pronounced for long simulations and can result in a simulation crashing due to exceeding memory available. Breaking up long runs into smaller consecutive runs avoids the issue but we find that doing so changes the output. Lizzie Lundgren (GCST) is investigating differences in GCHP output that happen when restarting a simulation, which may or may not be a problem with GEOS-Chem Classic as well. We are documenting all single versus multi-run differences at [http://ftp.as.harvard.edu/gcgrid/geos-chem/validation/multi_vs_single_run/ http://ftp.as.harvard.edu/gcgrid/geos-chem/validation/multi_vs_single_run/]. Look there to assess if the scale of the differences would be a problem for you.
 +
*GCHP 12.5.0 uses online ESMF regridding weights rather than external tile files for regridding. Due to ESMF domain decomposition rules this can result in a MAPL error if an input grid is too coarse for a run’s configured core count. We have seen this problem for 4°x5° input files when using >600 cores. GMAO has fixed this problem in a more recent version of MAPL. All 4°x5° input files are replaced with higher resolution files for GCHP 12.5.0 to avoid this issue. However, users may still run into problems if running with thousands of cores.
 +
*There is a positive bias in surface ozone over the southern ocean in GCHP. It shows in comparisons with both GCC 12.5.0 and GCHP 12.4.0. The issue is resolution-dependent and occurs in FV3. An issue has been created on GEOS github for this: https://github.com/GEOS-ESM/GEOSgcm_GridComp/issues/48.
 +
 
 +
'''GCHP changes in the pipeline'''
 +
 
 +
Several major development updates to GCHP are slated for release in fall 2019. Updates include:
 +
#Update to ESMF v8 public release
 +
#Separation of ESMF from the GCHP repository and inclusion as an external module
 +
#Replacement of GNU make with CMake for compilation
 +
#Use of new GMAO public GitHub repositories as git submodules
 +
#Restructuring of the GCHP repository to be a wrapper around GEOS-Chem, with GEOS-Chem at a git submodule within GCHP
 +
 
 +
The GCHP changes are necessary to keep pace with changes in the GMAO libraries upon which GCHP depends. They will enable ongoing inclusion of new MAPL developments that will improve GCHP performance and extend capabilities over time. '''''The GCST is looking for volunteer beta testers to try out the new version on their systems and provide feedback.''''' If you are interested in becoming a beta tester please contact the GEOS-Chem Support Team.
 +
 
 +
'''Recent Changes section added to GCHP online manual'''
 +
 
 +
There is now a [[Getting_Started_with_GCHP#Recent_Changes|Recent Changes]] section on the [[Getting_Started_with_GCHP|GCHP online manual homepage]] to help experienced users stay up-to-date with logistical changes for compiling and running GCHP. The GCHP Quick Start Guide is updated for every GCHP version release but it can be difficult for existing users to know where to look for changes. Instead, users should check the Recent Changes section to see a summary of what is new and then use the Quick Start Guide to hone in on the details. Summaries per version release will be archived (location TBD).
  
 
== GCHP Cubed Sphere Grid Geometry ==
 
== GCHP Cubed Sphere Grid Geometry ==
Line 32: Line 73:
 
*[http://geos-chem.org/cubed_sphere/CubeSphere_step-by-step.html Step-by-step illustration on how to create a cubed sphere grid]
 
*[http://geos-chem.org/cubed_sphere/CubeSphere_step-by-step.html Step-by-step illustration on how to create a cubed sphere grid]
 
*[http://acmg.seas.harvard.edu/geos/cubed_sphere/CubeSphere_comparision.html Comparison of different cubed sphere map projections]
 
*[http://acmg.seas.harvard.edu/geos/cubed_sphere/CubeSphere_comparision.html Comparison of different cubed sphere map projections]
*[http://acmg.seas.harvard.edu/geos/cubed_sphere/CubeSphere_stretch.html Creating a "stretched" cubed sphere]
+
*[http://acmg.seas.harvard.edu/geos/cubed_sphere/CubeSphere_stretch.html Creating a "stretched" cubed sphere to get fine resolution in a selected region of the world ]
**This is a way of getting fine resolution in a selected region of the world (e.g. akin to GEOS-Chem "Classic" nested-grid simulations)
+
 
+
Using a cubed sphere grid allows for natural parallelization. At least one unique core is assigned to each face on the cubed sphere, resulting in a constraint of at least six cores to run GCHP. The same number of cores must be assigned to each face, resulting in another constraint of total number of cores being a multiple of six. Communication between the cores occurs only during transport processes.
+
 
+
While any number of cores is valid as long as it is a multiple of six, you will typically start to see negative effects due to excessive communication if a core is handling less than around one hundred grid cells or a cluster of grid cells that are not approximately square. You can determine how many grid cells are handled per core by analyzing your grid resolution and resource allocation. For example, if running at C24 with six cores each face is handled by one core (6 faces / 6 cores) and contains 576 cells (24x24). Each core therefore processes 576 cells. Since each core handles one face, each core communicates with four other cores (four surrounding faces).
+
 
+
You can configure approximately how the cores are assigned to grid cell geometry by using the NX and NY configuration variables in GCHP. Imagine lining up the six face grids adjacent to each other to get a single rectangular array. The rectangle will have N grid cells width (e.g. 24 if a C24 grid), and 6N grid cells height (since 6 faces). NX is the number of segments the width N is broken into for core distribution. NY is the number of segments the height 6N is broken into and must always be a multiple of six. NX * NY is always the total number of cores.
+
 
+
For the case of a six core run, NX is equal to 1 and NY is equal to 6. This is because the entire N grid cells width is handled by 1 core (NX) and the 6N grid cells height is handled by 6 cores (NY), or one per face. If you instead wanted each face to be handled by four cores, and further constrain each core to handle one face quadrant, you would set NX equal to two and NY equal to twelve. A simple way of thinking about this is that core distribution across each face is with geometry NX x NY/6. In this last example that would be equivalent to 2x2.
+
  
 
----------------------------
 
----------------------------
  
--[[User:Lizzie Lundgren|Lizzie Lundgren]] ([[User talk:Lizzie Lundgren|talk]]) 14:40, 28 June 2018 (UTC)
+
'''''[[Main_Page|GEOS-Chem Main Page]]'''''

Latest revision as of 19:37, 28 August 2019

GEOS-Chem Main Page

About High Performance GEOS-Chem

GEOS-Chem with the high performance option (GCHP) is the next generation of GEOS-Chem. It uses a distributed memory capability designed to enable efficient scaling across many cores, and thus enable finer resolution simulations. GCHP has the advantage of using a cubed sphere geometry that enables more accurate transport and eliminates the polar singularity inherent to lat-lon grids. With these advances you can still run the model on a single machine at coarse resolution if you choose to do so.

If you are working on a project using GCHP, please add your name and project description to the GCHP projects list. Also help the GCHP Working Group pool performance information across systems by contributing your GCHP run information on the GCHP Timing Tests page.

If you would like to stay informed of GCHP developments, please join the GEOS-Chem High Performance Working Group mailing list and subscribe to GCHP notifications on GitHub. You may also join the GCHP Slack workspace to interact with other users. If you would like receive help or participate in conversations about GCHP, please use the GCHP GitHub issue tracker. You may also contact the GEOS-Chem Support Team if you have questions or need assistance, or reach out to the GCHP Working Group Co-Chairs.

GCHP Wiki Guide

Getting Started

Version Information

Validation and Performance

Community

Troubleshooting

Help Resources and Quick Links

Featured News

Speed improvements in GCHP 12.5.0

GCHP 12.5.0 uses MAPL version tag "gchp/v1.0.0" released in early 2019. The MAPL library, which is developed by NASA, is a wrapper for the Earth System Modeling Framework (ESMF), and handles not only I/O but also passing of data between components. With this version update GCHP input speed executes roughly 3-4 times faster than previous versions! This speed-up is due to parallelization within input read across one node, and the MAPL updates in this version are just the beginning of improvements to GCHP I/O bottlenecks.

Important Notes about GCHP 12.5.0

While we are encouraged by the improvements to GCHP in 12.5.0, a few issues remain open until the next MAPL update later this year.

  • GCHP 12.5.0 currently has a slow memory leak due to updates to MAPL ExtData. NASA is aware of the issue and will be developing a fix soon. The memory leak is more pronounced for long simulations and can result in a simulation crashing due to exceeding memory available. Breaking up long runs into smaller consecutive runs avoids the issue but we find that doing so changes the output. Lizzie Lundgren (GCST) is investigating differences in GCHP output that happen when restarting a simulation, which may or may not be a problem with GEOS-Chem Classic as well. We are documenting all single versus multi-run differences at http://ftp.as.harvard.edu/gcgrid/geos-chem/validation/multi_vs_single_run/. Look there to assess if the scale of the differences would be a problem for you.
  • GCHP 12.5.0 uses online ESMF regridding weights rather than external tile files for regridding. Due to ESMF domain decomposition rules this can result in a MAPL error if an input grid is too coarse for a run’s configured core count. We have seen this problem for 4°x5° input files when using >600 cores. GMAO has fixed this problem in a more recent version of MAPL. All 4°x5° input files are replaced with higher resolution files for GCHP 12.5.0 to avoid this issue. However, users may still run into problems if running with thousands of cores.
  • There is a positive bias in surface ozone over the southern ocean in GCHP. It shows in comparisons with both GCC 12.5.0 and GCHP 12.4.0. The issue is resolution-dependent and occurs in FV3. An issue has been created on GEOS github for this: https://github.com/GEOS-ESM/GEOSgcm_GridComp/issues/48.

GCHP changes in the pipeline

Several major development updates to GCHP are slated for release in fall 2019. Updates include:

  1. Update to ESMF v8 public release
  2. Separation of ESMF from the GCHP repository and inclusion as an external module
  3. Replacement of GNU make with CMake for compilation
  4. Use of new GMAO public GitHub repositories as git submodules
  5. Restructuring of the GCHP repository to be a wrapper around GEOS-Chem, with GEOS-Chem at a git submodule within GCHP

The GCHP changes are necessary to keep pace with changes in the GMAO libraries upon which GCHP depends. They will enable ongoing inclusion of new MAPL developments that will improve GCHP performance and extend capabilities over time. The GCST is looking for volunteer beta testers to try out the new version on their systems and provide feedback. If you are interested in becoming a beta tester please contact the GEOS-Chem Support Team.

Recent Changes section added to GCHP online manual

There is now a Recent Changes section on the GCHP online manual homepage to help experienced users stay up-to-date with logistical changes for compiling and running GCHP. The GCHP Quick Start Guide is updated for every GCHP version release but it can be difficult for existing users to know where to look for changes. Instead, users should check the Recent Changes section to see a summary of what is new and then use the Quick Start Guide to hone in on the details. Summaries per version release will be archived (location TBD).

GCHP Cubed Sphere Grid Geometry

GCHP uses a cubed sphere grid rather than the traditional lat-lon grid used in GEOS-Chem Classic. While regular lat-lon grids are typically designated as ΔLat ⨉ ΔLon (e.g. 4⨉5), cubed sphere grids are designated by the side-length of the cube. In GCHP we specify this as CX (e.g. C24 or C180). The simple rule of thumb for determining the roughly equivalent lat/lon for a given cubed sphere resolution is to divide the side length by 90. Using this rule you can quickly match C24 with 4x5, C90 with 1 degree, C360 with quarter degree, and so on.

Harvard graduate student Jiawei Zhuang created the following interactive illustrations demonstrating key features of the GCHP cubed sphere geometry:


GEOS-Chem Main Page