ncar / fasteddy-model

An NSF NCAR developed, parallelized and GPU-resident, large-eddy simulation code for accelerated modeling of the atmospheric boundary layer.

FastEddy®

©2016 University Corporation for Atmospheric Research

Open-source License

The FastEddy® model is licensed under the Apache License, Version 2.0 (the "License"); you may not use any source code in this repository except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

Description

FastEddy® (FE) is a large-eddy simulation (LES) model developed by the Research Applications Laboratory (RAL) at the U.S. National Science Foundation National Center for Atmospheric Research (NSF NCAR) in Boulder, Colorado, USA. The fundamental premise of FastEddy model development is to leverage the accelerated and more power efficient computing capacity of graphics processing units (GPU)s to enable not only more widespread use of LES in research activities but also to pursue the adoption of microscale and multiscale, turbulence-resolving, atmospheric boundary layer modeling into local scale weather prediction or actionable science and engineering applications.

Contact

Please submit all comments, feedback, suggestions, or questions by email to the NSF NCAR FastEddy team at [email protected]. Further information about FastEddy applications and research is available via the RAL website.

Citation

FastEddy should be cited as follows:

Sauer, J., and D. Muñoz-Esparza. "The FastEddy resident-GPU accelerated large-eddy simulation framework: model formulation, dynamical-core validation and performance benchmarks". Journal of Advances in Modeling Earth Systems, vol. 12 (2020) https://doi.org/10.1029/2020MS002100

Documentation

FastEddy documentation for this version and previous versions are available through Read the Docs.

Tutorials

FastEddy tutorials for idealized cases are available in the Tutorials section of the documentation.

Publications

FastEddy publications are available in the Publications section of the documentation.

fasteddy-model's Issues

Dynamic temperature/moisture roughness length (z0t) following Zilitinkevich (1995) formulation

Describe the New Feature

Implement a parameterization for dynamic temperature/moisture roughness length (z0t) following Zilitinkevich (1995) formulation

Acceptance Testing

List input data types and sources.
Describe tests required for new functionality.

Time Estimate

~0.5 day

Relevant Deadlines

Part of v2.0 release (June 2024)

New Parameters

surflayer_z0tdyn; /* dynamic z0t calculation following Zilitinkevich (1995) approach: 0= off, 1= constant Zilitinkevich coeff, 2= variable Zilitinkevich coeff */
Parameter is optional entry in the params file and set to option 1 by default.

Technical References

Zilitinkevich, S. "Non-local turbulent transport: Pollution dispersion aspects of coherent structure of connective flows." WIT Transactions on Ecology and the Environment 9 (1995): 53-60.

Define the Metadata

Assignee

• Assign this Task to the appropriate person

Labels

• Select pillar
• Select priority
• Select requestor(s)

New Feature Checklist (DETAILS TBD - THE INFO BELOW IS NOT YET DECIDED UPON)

• Complete the issue definition above.
• Fork this repository or create a branch of develop.
  Branch name: feature_<Issue Number>_<Description>
• Complete the development and test your changes.
• Add/update log messages for easier debugging.
• Add/update unit tests.
• Add/update documentation.
• Push local changes to GitHub.
• Submit a pull request to merge into develop.
  Pull request: feature <Issue Number> <Description>
• Define the pull request metadata, as permissions allow.
• Iterate until the reviewer(s) accept and merge your changes.
• Delete your fork or branch.
• Close this issue.

Set up the FastEddy Tutorial documentation

Describe the Task

The Fast Eddy Tutorial documentation currently lives in the FastEddyTutorial GitHub repository. We'd like to bring the documentation into this repository to keep the documentation and the code together. The task is to:

• Set up the documentation structure in this repository
• Set up webhooks with Read the Docs
• Set up continuous integration to automatically update the generated documentation on Read the Docs when changes are made to various branches of the repository.

Time Estimate

1-3 days of work

Relevant Deadlines

None

Define the Metadata

Assignee

• Assign this Task to the appropriate person

Labels

• Select pillar
• Select priority
• Select requestor(s)

Auxiliary scalar transport and dispersion: Example/Tutorial/Notebook

Describe the New Feature

Auxiliary scalar transport and dispersion: Example/Tutorial/Notebook

Acceptance Testing

List input data types and sources.
Describe tests required for new functionality.

Time Estimate

~2-3 days of work.

Relevant Deadlines

July release.

New Parameters

Define any new parameters here or state NONE.

Technical References

List relevant technical references here or state NONE.

Define the Metadata

Assignee

• Assign this Task to the appropriate person

Labels

• Select pillar
• Select priority
• Select requestor(s)

New Feature Checklist (DETAILS TBD - THE INFO BELOW IS NOT YET DECIDED UPON)

• Complete the issue definition above.
• Fork this repository or create a branch of develop.
  Branch name: feature_<Issue Number>_<Description>
• Complete the development and test your changes.
• Add/update log messages for easier debugging.
• Add/update unit tests.
• Add/update documentation.
• Push local changes to GitHub.
• Submit a pull request to merge into develop.
  Pull request: feature <Issue Number> <Description>
• Define the pull request metadata, as permissions allow.
• Iterate until the reviewer(s) accept and merge your changes.
• Delete your fork or branch.
• Close this issue.

Canopy Module

Describe the New Feature

Add the 2-equation canopy model

Acceptance Testing

List input data types and sources.
Describe tests required for new functionality.

Time Estimate

~3-days

Relevant Deadlines

Release v2.0 (June 2024)

New Parameters

These are the default values of the new parms file entries:
canopySelector = 0 # canopy selector: 0= off, 1= on
canopySkinOpt = 0 # canopy selector to use additional skin friction effect on drag coefficient: 0=off, 1=on
canopy_cd = 0.15 # non-dimensional canopy drag coefficient when canopySelector > 0
canopy_lf = 0.1 # representative canopy element length scale when canopySelector > 0

Technical References

Shaw, Roger H., and Ulrich Schumann. "Large-eddy simulation of turbulent flow above and within a forest." Boundary-Layer Meteorology 61, no. 1 (1992): 47-64.
Shaw, Roger H., and Edward G. Patton. "Canopy element influences on resolved-and subgrid-scale energy within a large-eddy simulation." Agricultural and forest meteorology 115, no. 1-2 (2003): 5-17.

Define the Metadata

Assignee

• Assign this Task to the appropriate person

Labels

• Select pillar
• Select priority
• Select requestor(s)

New Feature Checklist (DETAILS TBD - THE INFO BELOW IS NOT YET DECIDED UPON)

• Complete the issue definition above.
• Fork this repository or create a branch of develop.
  Branch name: feature_<Issue Number>_<Description>
• Complete the development and test your changes.
• Add/update log messages for easier debugging.
• Add/update unit tests.
• Add/update documentation.
• Push local changes to GitHub.
• Submit a pull request to merge into develop.
  Pull request: feature <Issue Number> <Description>
• Define the pull request metadata, as permissions allow.
• Iterate until the reviewer(s) accept and merge your changes.
• Delete your fork or branch.
• Close this issue.

Create the FastEddy-2.0 release

Describe the Task

Create the FastEddy-2.0 release.

• Update docs/conf.py
• Update the Release Notes
• Update the Downloads page
• Add two new publications

Time Estimate

1 hour

Relevant Deadlines

Should be created on Friday 8/9/24

New Parameters

None

Technical References

Adding the following:

Application of the NCAR FastEddy® Microscale Model to a Lake Breeze Front
https://www.mdpi.com/2073-4433/15/7/809

Implementation and Validation of a Generalized Actuator Disk Parameterization for Wind Turbine Simulations Within the FastEddy Model
http://doi.org/10.1002/we.2941

Define the Metadata

Assignee

• Assign this Task to the appropriate person

Labels

• Select pillar
• Select priority
• Select requestor(s)

Documentation: Various Modifications and Additions

Describe the Task

Apply these changes to the main_v1.1 and develop branches (using a feature branch for each):

• (DONE) Indicate how users should cite FastEddy in the README.md
• (DONE) Indicate how users should cite FastEddy in the documentation
• (DONE) Remove the version number from the main page of the documentation
• (DONE) Add a note on the front page pointing to our open, public GitHub repository
• (DONE) Add information under the Bugfixes and Enhancement section in the Release Notes to be more specific about the work that was included in Modifications to build on Derecho and a small fix restart capability of FastEddy runs #6 @jsauer-NCAR and @domingom will send @jprestop specific information to add in a bullet point list.
• (DONE) Add a Publications page (section by year and then numerical list). Include the following publications and use the format of the publications listed here.
• (DONE) Fix the link on the Getting Started page and the lower level tutorial examples pages. Change the text from "Instructions on how to build and run FastEddy on NSF NCAR’s Casper architecture https://github.com/NCAR/FastEddy-model/blob/main/README.md." to something like "Instructions on how to build and run FastEddy on NSF NCAR’s Casper architecture can be found in the README.md file at https://github.com/NCAR/FastEddy-model." New page: Running FastEddy on NCAR HPCs (link to that page for each version)
• (DONE, but moved to docs only) In the README.md file make these commands stand out as commands: module -t list , module load [intel or gnu/openmpi/cuda], make, make clean
• (DONE, but moved to docs only) Remove all of the suggested directories in the README.md file and the PBS submission script allowing people to make their own choices. For example. "A bash-based PBS job submission script for running the model on NSF NCAR's Casper machine. This script assumes you have cloned this repository into a /glade/work/$USER/FastEddy directory you created." Update EXAMPLEDIR to modify it to export EXAMPLEDIR=$CODEDIR/tutorials/examples.
• (DONE) Fix typo on this page: "The contents of the archive incude" should be "The contents of the archive include"
• (DONE) Add note to BOMEX use case - This tutorial example will only run on a single GPU.

Time Estimate

1-3 days of work

Relevant Deadlines

Hope to be finished by June/July release

New Parameters

None

Technical References

None

Define the Metadata

Assignee

• Assign this Task to the appropriate person

Labels

• Select pillar
• Select priority
• Select requestor(s)

Check logic for #GPUs, tBx,y,z and domain dimensions

Describe the Enhancement

Add logic to check for #GPUs, tBx,y,z and domain dimensions to be set up correctly.

Time Estimate

~0.5 day of work

Relevant Deadlines

List relevant project deadlines here or state NONE.

New Parameters

Define any new parameters here or state NONE.

Technical References

List relevant technical references here or state NONE.

Define the Metadata

Assignee

• Assign this Task to the appropriate person

Labels

• Select pillar
• Select priority
• Select requestor(s)

Documentation: Add Contributing Authors

Describe the Task

Add a Contributing Authors section or page with the following authors:
Jeremy Sauer
Domingo Munoz-Esparza
Eric Hendricks
Julie Prestopnik

On the main page, create a "Citation" header for the citation section under the description and add a "Contributing Authors" header and section underneath.

Time Estimate

1 hour

Relevant Deadlines

None

New Parameters

None

Technical References

None

Define the Metadata

Assignee

• Assign this Task to the appropriate person

Labels

• Select pillar
• Select priority
• Select requestor(s)

Consolidate FastEddy-tutorials content into FastEddy-model

Describe the Task

Consolidate FastEddy-tutorials repository content into FastEddy-model.

• Create a Tutorials folder under FastEddy-models with examples and notebooks sub-folders.
• Move the contents from FastEddy-tutorials example and notebooks folders here.
• Finally create a Zenodo repository to archive the FastEddy-tutorials dataset folder and content and link the DOI badge in the FastEddy-model Tutorials documentation.

Time Estimate

Estimate the amount of work required here.
2-3 hours

Relevant Deadlines

None

Define the Metadata

Assignee

• Assign this Task to the appropriate person

Labels

• Select pillar
• Select priority
• Select requestor(s)

Create python tools for pre/post-processing

Describe the Enhancement

Provide python scripts developed for various pre/post-processing tasks

• Create a Tools directory in the repository
• Provide the binary-to-netcdf converter
• Provide ...
• Provide ...

Time Estimate

1-3 days

Relevant Deadlines

Should be a part of the v2.0 release (June 2024)

Define the Metadata

Assignee

• Assign this Task to the appropriate person

Labels

• Select pillar
• Select priority
• Select requestor(s)

Documentation: Add a table and trademark

Describe the Task

Add a table with links to the Releases and Release Notes. Something like the WRF-ARW Code Downloads on this page.
Add the trademark to title in the documentation in both main_v1.1 and develop

Time Estimate

<1 day of work

Relevant Deadlines

2.0 release

Define the Metadata

Assignee

• Assign this Task to the appropriate person

Labels

• Select pillar
• Select priority
• Select requestor(s)

Adjust FastEddy-tutorials BOMEX notebook & RTD Moist dynamics instructions for hosting datasets under new repo

Describe the Task

Adjust the FE_Postrocessing_Example04_BOMEX.ipynb notebook and the RTD instructions for the moist dynamics (BOMEX) case to accommodate the change in host location of the datasets for initial conditions and "other model" data.

Time Estimate

< 1 day

Relevant Deadlines

In time for Milestone 1.1 release

Define the Metadata

Assignee

• Assign this Task to the appropriate person

Labels

• Select pillar
• Select priority
• Select requestor(s)

Create the official FastEddy-1.1.0 release

Describe the Task

Create the official FastEddy-1.1.0 release. Remember to:

• Update references to NCAR to be NSF NCAR
• Update doi to zenodo references
• Create the release notes
• Update the version, release_year, and release_date variables for the documentation in docs/conf.py.
• Look into Zenodo for this release

Time Estimate

<1 day of work

Relevant Deadlines

None

Define the Metadata

Assignee

• Assign this Task to the appropriate person

Labels

• Select pillar
• Select priority
• Select requestor(s)

Offshore: Example/Tutorial/Notebook

Describe the New Feature

Offshore: Example/Tutorial/Notebook

Acceptance Testing

List input data types and sources.
Describe tests required for new functionality.

Time Estimate

~2-3 days of work.

Relevant Deadlines

July release

New Parameters

Define any new parameters here or state NONE.

Technical References

List relevant technical references here or state NONE.

Define the Metadata

Assignee

• Assign this Task to the appropriate person

Labels

• Select pillar
• Select priority
• Select requestor(s)

New Feature Checklist (DETAILS TBD - THE INFO BELOW IS NOT YET DECIDED UPON)

• Complete the issue definition above.
• Fork this repository or create a branch of develop.
  Branch name: feature_<Issue Number>_<Description>
• Complete the development and test your changes.
• Add/update log messages for easier debugging.
• Add/update unit tests.
• Add/update documentation.
• Push local changes to GitHub.
• Submit a pull request to merge into develop.
  Pull request: feature <Issue Number> <Description>
• Define the pull request metadata, as permissions allow.
• Iterate until the reviewer(s) accept and merge your changes.
• Delete your fork or branch.
• Close this issue.

Add documentation notes to the release to describe fixes/enhancements/extensions

Describe the Task

Add documentation notes to the release to describe fixes/enhancements/extensions

Time Estimate

~1 day of work.

Relevant Deadlines

July release.

New Parameters

Define any new parameters here or state NONE.

Technical References

List relevant technical references here or state NONE.

Define the Metadata

Assignee

• Assign this Task to the appropriate person

Labels

• Select pillar
• Select priority
• Select requestor(s)

Add new 'scripts' directory and create includes for PBS script in docs

Describe the Task

Add new 'scripts' directory and create includes for PBS script in docs

Time Estimate

<1 day

Relevant Deadlines

2.0 release

New Parameters

None

Technical References

None

Define the Metadata

Assignee

• Assign this Task to the appropriate person

Labels

• Select pillar
• Select priority
• Select requestor(s)

First set of new features to v2.0 FastEddy-model code

Describe the New Feature

• Use PPOPTION3 instead of PPOPTION4 for pressure bottom/top boundary conditions. domingom#2
• Add factor (1+ rv2) to the f_cond term for consistency with mixing ratio formulation and adjustments to the time scale use depending on the moistureCon. domingom#3
• Divergence damping & 6th-order horizontal diffusion explicit filter options. Fix for the filters to work for dry simulations. Apply explicit filters consistent with the other physics (i.e., dependent on physics_oneRKonly). Remove unnecesary commented out declaration. domingom#4
• Offshore roughness parameterizations. domingom#5

New mandatory input parameters
surflayer_offshore = 0

New optional input parameters
filter_6thdiff_vert = 0
filter_6thdiff_vert_coeff = 0.03
filter_6thdiff_hori = 0
filter_6thdiff_hori_coeff = 0.03
filter_divdamp = 0
surflayer_offshore_opt = 0
surflayer_offshore_dyn = 1
surflayer_offshore_hs = 0.0
surflayer_offshore_lp = 0.1
surflayer_offshore_cp = 0.1
surflayer_offshore_theta = 0.0
surflayer_offshore_visc = 1

"surflayer_offshore_opt", "offshore roughness parameterization: ==0 (Charnock), ==1 (Charnock with variable alpha), ==2 (Taylor & Yelland), ==3 (Donelan), ==4 (Drennan), ==5 (Porchetta)"

Acceptance Testing

List input data types and sources.
Describe tests required for new functionality.

Time Estimate

1 day

Relevant Deadlines

Part of v2.0 release (June 2024)

Define the Metadata

Assignee

• Assign this Task to the appropriate person

Labels

• Select pillar
• Select priority
• Select requestor(s)

New Feature Checklist (DETAILS TBD - THE INFO BELOW IS NOT YET DECIDED UPON)

• Complete the issue definition above.
• Fork this repository or create a branch of develop.
  Branch name: feature_#19_1stCommitV2p0
• Complete the development and test your changes.
• Add/update log messages for easier debugging.
• Add/update unit tests.
• Add/update documentation.
• Push local changes to GitHub.
• Submit a pull request to merge into develop.
  Pull request: feature <Issue Number> <Description>
• Define the pull request metadata, as permissions allow.
• Iterate until the reviewer(s) accept and merge your changes.
• Delete your fork or branch.
• Close this issue.

Add templates for Issues and Pull Requests

Describe the Task

• Add issue templates for four different kinds of issues: new features, enhancements, bugs, and tasks
• Add a pull request template
• Add a .gitignore file so that tilda files generated by emacs and temp files surrounded by # will automatically not be checked into git

Time Estimate

<2 hours

Relevant Deadlines

None

Define the Metadata

Assignee

• Assign this Task to the appropriate person

Labels

• Select pillar
• Select priority
• Select requestor(s)

Add logic fix for number of grid points, block size, and number of GPUs

Describe the Enhancement

Add logic fix for number of grid points, block size, and number of GPUs
Part of v2.0 release (June 2024)

Time Estimate

~0.5 day of work

Relevant Deadlines

List relevant project deadlines here or state NONE.

New Parameters

NONE

Technical References

NONE

Define the Metadata

Assignee

• Assign this Task to the appropriate person

Labels

• Select pillar
• Select priority
• Select requestor(s)

Update the Issue Templates

Describe the Task

Add a Technical Reference subsection and a subsection for New Parameters to the appropriate Issue Templates in both main_v1.1 and develop.

Time Estimate

<1 day of work

Relevant Deadlines

None

Define the Metadata

Assignee

• Assign this Task to the appropriate person

Labels

• Select pillar
• Select priority
• Select requestor(s)

Auxiliary scalar transport and dispersion

Describe the New Feature

Add Auxiliary scalar module

Acceptance Testing

List input data types and sources.
Describe tests required for new functionality.

Time Estimate

~3-days

Relevant Deadlines

Part of v2.0 release (June 2024)

Define the Metadata

Assignee

• Assign this Task to the appropriate person

Labels

• Select pillar
• Select priority
• Select requestor(s)

New Feature Checklist (DETAILS TBD - THE INFO BELOW IS NOT YET DECIDED UPON)

• Complete the issue definition above.
• Fork this repository or create a branch of develop.
  Branch name: feature_<Issue Number>_<Description>
• Complete the development and test your changes.
• Add/update log messages for easier debugging.
• Add/update unit tests.
• Add/update documentation.
• Push local changes to GitHub.
• Submit a pull request to merge into develop.
  Pull request: feature <Issue Number> <Description>
• Define the pull request metadata, as permissions allow.
• Iterate until the reviewer(s) accept and merge your changes.
• Delete your fork or branch.
• Close this issue.

Canopy tutorial case, postprocessing notebook, and readthedocs

Describe the New Feature

Canopy tutorial case, postprocessing notebook, and readthedocs

Acceptance Testing

List input data types and sources.
Describe tests required for new functionality.

Time Estimate

~2 days of work.*

Relevant Deadlines

Part of v2.0 release (June 2024)

New Parameters

NONE

Technical References

NONE

Define the Metadata

Assignee

• Assign this Task to the appropriate person

Labels

• Select pillar
• Select priority
• Select requestor(s)

New Feature Checklist (DETAILS TBD - THE INFO BELOW IS NOT YET DECIDED UPON)

• Complete the issue definition above.
• Fork this repository or create a branch of develop.
  Branch name: feature_<Issue Number>_<Description>
• Complete the development and test your changes.
• Add/update log messages for easier debugging.
• Add/update unit tests.
• Add/update documentation.
• Push local changes to GitHub.
• Submit a pull request to merge into develop.
  Pull request: feature <Issue Number> <Description>
• Define the pull request metadata, as permissions allow.
• Iterate until the reviewer(s) accept and merge your changes.
• Delete your fork or branch.
• Close this issue.