clima / climatemachine.jl

Climate Machine: an Earth System Model that automatically learns from data

ClimateMachine.jl

NOTE THAT THIS REPO IS NOT CURRENTLY BEING MAINTAINED. PLEASE SEE THE FOLLOWING REPOSITORIES:

• CliMA/ClimaCore.jl
• CliMA/ClimaAtmos.jl
• CliMA/Thermodynamics.jl
• CliMA/CloudMicrophysics.jl
• CliMA/SurfaceFluxes.jl
• CliMA/ClimaLSM.jl
• CliMA/Oceananigans.jl
• CliMA/ClimaCoupler.jl

The Climate Machine is a new Earth system model that leverages recent advances in the computational and data sciences to learn directly from a wealth of Earth observations from space and the ground. The Climate Machine will harness more data than ever before, providing a new level of accuracy to predictions of droughts, heat waves, and rainfall extremes.

Documentation
Docs Build
Unit tests
Code Coverage
Bors
DOI

For installation instructions and explanations on how to use the Climate Machine, please look at the Documentation.

climatemachine.jl's Issues

Utilities not precompiled on Caltech GPU

After adding Utitilies to the gpu environment on the Caltech GPU, the MoistThermodynamics module is not being found.

The problem arises when I call any of the functions of moistthermodynamics from Vanilla.
It does not seem tied to MoistThermodynamics, however, it only occurs when I make a call to a MoistThermodynamics function on GPU.

ERROR: LoadError: InvalidIRError: compiling knl_volumegrad!(Val{2}, Val{4}, Val{0}, Val{0}, CuDeviceArray{Float64,4,CUDAnative.AS.Global}, CuDeviceArray{Float64,4,CUDAnative.AS.Global}, CuDeviceArray{Float64,4,CUDAnative.AS.Global}, Float64, CuDeviceArray{Float64,2,CUDAnative.AS.Global}, Int64) resulted in invalid LLVM IR Reason: unsupported call to the Julia runtime (call to jl_get_binding_or_error) Stacktrace: [1] macro expansion at /home/smarras/.julia/packages/CUDAnative/nqSUm/src/device/intrinsics/memory_shared.jl:24 [2] knl_volumegrad! at /central/home/smarras/climate-machine/CLIMA/src/ClimaAtmos/Dycore/src/VanillaAtmosDiscretizations_cuda.jl:15 Reason: unsupported call to the Julia runtime (call to jl_undefined_var_error) ...

Rotation

Where do you set the rotation rate, i.e. Coriolis parameter?

MoistThermodynamics performance improvements

The MoistThermodynamics module will need to be revised to improve performance:

• Use something like Cassette.jl to swap out things like exp with CUDA-native versions on GPUs
• Make sure all functions are type generic
• The function phase_partitioning_eq!, which has two output arguments, is wrapped in an explicit loop to handle two arrays on output. Other functions with one output argument can be broadcast. Make more consistent?
• Improve initial conditions for saturation adjustment so that secant method for root finding converges reliably

Add environments for CLIMA

Add a gpu, cpu, and visualize environment so that CLIMA can be launched with

• julia --project=env/gpu
• julia --project=env/cpu (default)
• julia --project=env/viz

Add test (CPU & GPU) for AtmosDycore

To test code on the GPU we'll need to do something like this https://github.com/JuliaGPU/gitlab-ci

Encapsulate the Dycore mesh

Right now, it seems that the dycore mesh consists of many tuples of arrays that define relations between element neighbors, vertices etc.

I think it would be good to make a struct Mesh and make these mesh-related arrays properties of Mesh so that functions calls can have simpler interfaces. For example, in L2errornorm

L2errornorm(::Val{dim}, ::Val{N}, time, Q, vgeo, elems, Qexact)

would become

L2errornorm(::Val{dim}, ::Val{N}, time, Q, mesh::Mesh, Qexact)

This will also greatly simplify the mesh initialization, which should be something of the form:

mesh = Mesh(params)

[AtmosDycore] Upgrade Julia CUDA packages

`finalize_atexit` is not supported on Windows

I just tried running the rising_thermal_bubble.jl on my windows machine and I'm getting this error:

ERROR: LoadError: finalize_atexit is not supported on Windows

Maybe we can add a local conditional to handle this (if Sys.iswindows(); ...; end).

Fork not in sync with main repo

When I git pull or git fetch from within my fork of climate-machine/CLIMA
git returns up to date although I am not.
In other words, my fork is not in sync with the main code I am forked from.

The log of my fork is:
commit 5f1226ac5b054fee5dc4cee0e2ecfc5997fb171d (HEAD -> master, origin/master, origin/HEAD) Merge: db7235e 4adbe39 Author: Lucas C Wilcox <[email protected]> Date: Thu Feb 21 15:08:42 2019 -0800

whereas the log of the main repo is:

commit 5358a667c357b1b4ab8f273f3edd8e5f873e464a (origin/master, origin/HEAD) Merge: 5f1226a e23af30 Author: Charles Kawczynski <[email protected]> Date: Tue Feb 26 13:00:51 2019 -0800 Merge pull request #80 from jkozdon/windows [AtmosDycore] Make drivers windows friendly

can anyone please assist on this?
thanks

'Float32' issue in AtmosDycore

It's possible there is an issue when the AtmosDycore is run in Float32 mode. Results for rising_thermal_bubble.jl give weird results (visualize the velocity field)

edit: thinking there may be an issue with the background state being included in the solution. Plan is to try to subtract it off to see if that helps.

h/t @smarras79

CUDAdrv on OSX

Has any of you been successful in building CUDAdrv on a Mac OSX (10.14 Mojave)?
No matter what steps I take, the build fails with
Pkg.build("CUDAdrv") Building CUDAdrv → ~/.julia/packages/CUDAdrv/LC5XS/deps/build.log┌ Error: Error buildingCUDAdrv: │ ERROR: LoadError: Initializing CUDA driver failed with code 3.

(no forum seems to have it solved.)

I am using:
Julia Version 1.0.3 Commit 099e826241 (2018-12-18 01:34 UTC) Platform Info: OS: macOS (x86_64-apple-darwin14.5.0) CPU: Intel(R) Core(TM) i5-7360U CPU @ 2.30GHz WORD_SIZE: 64 LIBM: libopenlibm LLVM: libLLVM-6.0.0 (ORCJIT, skylake)
with CUDA Toolkit 10.1 installed.

p.s. For the future, should questions like this one be reported as an Issue?

Use parametric types instead of `using CurrentPrecision: DFloat`

Need to convert more of the DFloat types to generic types.

Add appveyor

We should add appveyor to test on Windows.

Fix `jldoctest` typo in documentation

Small typo in
https://github.com/climate-machine/CLIMA/blob/master/src/ClimaAtmos/Dycore/src/Topologies.jl#L61,

jldoctes brickmesh
Should read
jldoctest brickmesh

Small issue, but just wanted to note

Utilities module

There was a discussion to move the Parameters type to a general utilities module. Just recording that idea here.

Flatten AtmosTCC data structures

Currently, the AtmosTCC data structures are tree-like, and they need to be flattened to better integrate with calling routines.

CLIMA nightly travis build fails with non-registered packages (e.g., Canary)

I'm often experiencing issues with CLIMA depending on Canary. For example, right now Travis is complaining that

ERROR: Build path for Canary does not exist: /home/travis/.julia/packages/Canary/L6RGe

for the nightly build, but is passing on Julia 1.0.

As far as I can see, none of my commits touched Canary and I never removed Canary from the Clima root Manifest. Any help from @lcw or @jkozdon is greatly appreciated!

Tracers performance improvement

I see that the tracers are added with a whole additional set of loops over i,j,k after closing the i,j,k loops for the states.
Unless there is a specific reason for that choice, I think that the tracers should be moved within the loops where the states are calculated.

Add deployment key for Documenter.jl

For Travis to auto-deploy documentation updates, we need to add a deployment key.

[AtmosDycore] Replace all thermodynamic calculations (energies, speed of sound, pressure etc.) by calls to MoistThermodynamics

We need to make integrating moisture in the dynamical core easier. It will be essential to replace the explicit thermodynamic calculations that are in the dry core now by calls to the MoistThermodynamics module. This includes

• Replace calculations of speed of sound (sqrt(gamma_d * P / ρ)) by calls to a speed of sound function (to be written) that is accurate for moist atmospheres. This should calculate sqrt(gamma * R_m * T) where gamma = c_pm/c_vm (for moist air), and R_m = gas_constant_air(...).
• Pressure should be calculated from air_pressure (if it is needed at all--in the speed of sound, the calculation via the temperature from air_temperature is more direct)
• Note that internal energy in the moist case is defined relative to a reference temperature T_0. Several expressions in the dry core are currently not consistent with that.

Flatten directory structure

I have been wondering if we want to further flatten the directory structure and use the convention the directory names and module names. Further we may want to use CamelCase for directory names since this matches the Julia module naming convention.

For instance

• Planet-parameters would become PlanetParameters
• CLIMA-atmos/Dycore would become CLIMAAtmosDycore

This seems to be more inline with what some other Julia projects are doing and it simplifies finding where a module is defined.

/cc @lcw @tapios @vchuravy @peterahrens @charleskawczynski

isbits CPU/GPU abstractions.

Sorry if I'm asking what has already been asked, apparently I missed the part of the walk-through where this was discussed. I was hoping to ask about how you guys are designing your abstractions to be isbits so they can be passed to GPU/CUDA kernels.

I was wondering if you had any of this code implemented or if you guys are discussing these issues on GitHub.

I had a look through VanillaAtmosDiscretizations_cuda.jl but couldn't find any structs. Maybe we can learn from your abstraction designs.

This is an issue we hit a while back (CliMA/Oceananigans.jl#59) and one which I'm currently working on (CliMA/Oceananigans.jl#147). We wanted to keep the CPU and GPU kernels shared so we just stripped away the abstractions which resulted in super long argument lists. I'm now refactoring the kernels to use isbits structs so hopefully they'll come out looking much better. Our grid, equation of state, parameters, etc. and arbitrary forcing functions are isbits but not the Field structs as they hold Array and CuArray types, which might need to be converted to an isbitstype using something like Adapt.jl which CUDAnative.jl uses to convert CuArrays to something like CuDeviceArrays.

---

For anyone not familiar with this issue: structs (abstractions) can only be passed to a GPU/CUDA kernel if the struct is an isbitstype type.

help?> isbitstype

  isbitstype(T)

  Return true if type T is a "plain data" type, meaning it is
  immutable and contains no references to other values, only
  primitive types and other isbitstype types. Typical examples
  are numeric types such as UInt8, Float64, and
  Complex{Float64}. This category of types is significant since
  they are valid as type parameters, may not track isdefined /
  isassigned status, and have a defined layout that is
  compatible with C.

Verify correctness of Regula Falsi method in RootSolvers

It seems that the Regula Falsi method tests are brittle for the CUDA tests (which operate at lower precision by default). Recording this here in case we are to use this method.

CLIMA vs CLIMA.jl - name conflict with attobot (for registering)

We will want to register CLIMA eventually and, according to attobot "It will only operate on packages with the standard ".jl" suffix (all other packages will be ignored)." So, it seems that currently, only CLIMA.jl is register-able.

This is not urgent, but I just wanted to start a conversation about it.

Remove `getindex` and `setindex!`

Switch to using functions and not getindex and setindex! in the AtmosDycore

time(runner)
settime!(runner, time)

instead of

runner[:time]
runner[:time] = time

GPUifyLoops.jl

Update AtmosDycore to use GPUifyLoops.jl to merge GPU and CPU kernels

Fold Atmos TCC into master

Checkpointing/restart

We need a way to checkpoint the Dycore for restarts (likely requires wrapping MPI I/O stuff).

Need discrete operator functions

We may need to iterate on the form of this to-do item, and we've discussed this but I'd like to just record the need here and discuss some pseudo-code examples.

The EDMF scheme (and likely other parameterization schemes) will need discrete operators (grad, div, etc.), provided by the dycore. Here's a pseudo-code example that I'm imagining:

Adds the derivative of `state_vector[f_sym]` with respect to
the direction, denoted by `dim`, on mesh element `elem`, and
stores the result in `state_vector[∇_sym]`.

compute_∇_local!(state_vector::StateVector,
                  ∇_sym::Symbol,
                  f_sym::Symbol,
                  mesh::Mesh,
                  dim::Int,
                  ::DerivativeType,
                  elem::Element
                  ) where Element where DerivativeType

Example:

vars = (:u, :v, :w, :dudx)
state_vector = StateVector(mesh, vars)

for e in over_elems(mesh)

  compute_∇_local!(state_vector,
                    :dudx,
                    :u,
                    mesh,
                    1,
                    VolumeDerivative(), # may also need FluxDerivative() or something
                    e
                    )

end

Change string eval to type dispatch

I think we should change https://github.com/climate-machine/CLIMA/blob/master/src/Utilities/src/MoistThermodynamics.jl#L280-L287

to apply type dispatch instead of evaluating a string.

Move moist tracers

As of now, moist tracers are handled after the dynamics. This should be changed because the moist dynamics depends on R_m, which depends on the values of the moist quantities.

Data file formats

For Oceananigans.jl we settled on supporting NetCDF output early as it's the most ubiquitous and familiar data format (see CliMA/Oceananigans.jl#31).

We've been using NetCDF.jl but it seems like an inactive package so we're thinking of switching to the more active and thoroughly tested NCDatasets.jl. NetCDF.jl has some missing features and seems slow. Output writing and compression was so slow compared to time stepping on the GPU that we ended up introducing asynchronous output writing (CliMA/Oceananigans.jl#137).

Anyways, just curious if NetCDF output is something you guys have looked at yet. If not, might be worth it to work on a common solution with features we all need (fast NetCDF output, option for asynchronous GPU output, standardized names, etc.)?

MoistThermodynamics compilation

`ERROR: LoadError: LoadError: LoadError: ArgumentError: Package CLIMAAtmosDycore does not have MoistThermodynamics in its dependencies:

• If you have CLIMAAtmosDycore checked out for development and have
  added MoistThermodynamics as a dependency but haven't updated your primary
  environment's manifest file, try Pkg.resolve().
• Otherwise you may need to report an issue with CLIMAAtmosDycore`

The use of

]dev ../../Utilities

from src/ClimaAtmos/Dycore

solves it but creates the following:

ERROR: LoadError: UndefVarError: LSRKmethods not defined

as I execute

julia --project

julia> include("./drivers/rising_thermal_bubble.jl")

(As a reference, MoistThermodynamics works correctly when used from Canary.)

Julia cannot constant-prop values through keyword arguments :(

In response to the FIXME in rising_thermal_bubble.jl, if you want to be sure that dim is treated as constant (to get loop unrolling and stuff), you'll need to pass dim as a positional argument. There are technical limitations in the compiler that make it difficult to propagate constants through keyword arguments. (we use the words "constant propagation" to refer to the process by which function arguments are treated as constant if they are treated as constants in the caller's scope.)

Change variable name `step` to something else

In several places, (for example here), step is used as a variable name, but step is a Julia intrinsic function:

help?> step
search: step StepRange StepRangeLen CompositeException RoundNearestTiesUp InvalidStateException isconcretetype

  step(r)

  Get the step size of an AbstractRange object.

  Examples
  ≡≡≡≡≡≡≡≡≡≡

  julia> step(1:10)
  1

  julia> step(1:2:10)
  2

  julia> step(2.5:0.3:10.9)
  0.3

  julia> step(range(2.5, stop=10.9, length=85))
  0.1

So, we should probably not overwriting it to avoid confusion.

Refactor AtmosDycore Runner naming

Consider refactoring the way the AtmosDycore runners are set up to make it more inline with Julia conventions. For now likely a thin refactoring with possible future major refactoring.

Move from

runner = Runner(:SpaceStepper, (space_params...), :TimeStepper, (time_params..))

to

space_runnner = SpaceRunner(space_params...)
time_runner = TimeRunner(space_runner, time_params...)
step!/run!/advance!/whatever!(time_runner, time_end, callbacks) #or something like this

Thermodynamics additions/checks needed

• Thermodynamics.jl should be checked carefully

• Tests should be added. For example, we must have

air_pressure([1, 1], [1, 1], [1, 0], [0, 0]) = [R_v, R_d]
latent_heat_vapor(T_0) = L_v0
latent_heat_fusion(T_0) = L_f0
latent_heat_sublim(T_0) = L_s0
sat_vapor_press_liquid(T_triple) = sat_vapor_press_triple
sat_vapor_press_ice(T_triple) = sat_vapor_press_triple

• As the dycore evolves, we should make sure the data structure for moisture variables (e.g., q_t, q_l, q_i) in the thermodynamics module is consistent with how the thermodynamic functions are called from the dycore and parameterizations. We may need to adapt these.

Remove hyphens from directory tree

Using hyphens in a folder effectively disables us from turning that folder into a package and is not aligned with coding conventions.

Stacked elements

Need to implement stacking of elements for 1-D solves and physics.

Variable name convention list

I think that it would be a good idea to start making a master list for variable names and their definition.

I'd like to get something up on the wall so that everyone has a common reference. I'd definitely like to hear people weigh in. @simonbyrne, @lcw @jkozdon @ali-ramadhan

Technically, the design docs should address this issue, however, we've perhaps limited ourselves by only allowing lowercase variables in our CodingConventions. We've started violating these naming conventions (for example here). I've done this myself here.

Perhaps this should be two issues, but I think that we should

• Develop a master variable name list
• Update the coding conventions such that they can be followed

I don't mean to harp on this issue. I just think that if we don't put in the effort to maintain/improve our naming conventions as we go, it will most certainly get worse.

Locking down the CLIMA repo?

At the kick-off meeting we discussed using the github pull request feature exclusively for merging into master. I noticed that right now CLIMA has no protected branches, so anyone on the climate-machine team can write to master, create branches, etc..

I am happy to lock all this down but figured we should discuss this.

One of the major reasons for exclusively using the pull request feature was so that users downstream had a heads up changes were coming and all changes had the opportunity for discussion. Not really a big problem when we have only a few users, but could quickly become a problem as new folks start contributing.

/cc @lcw @charleskawczynski @christophernhill @tapios @adcroft @fxgiraldo

`SVector`, `StructOfArrays`, etc...

Look to using SVector, CuArray compatible StructOfArrays, etc.. to improve readability of kernels; see also esderiv.jl scratch file.

Add Docs for AtmosDycore

size(vgeo) returns 70 elements for a grid of 100 elements

In Grid.jl vgeo is defined as

vgeo = zeros(DFloat, Nq^dim, _nvgeo, nelem)

In the risiting_thermal_bubble case I am building a 2D grid Ne = (10,1,10) for a 2D grid of 100 elements.
Why does size(vgeo) return nelem = 70?
Is vgeo different from how it was built in Canary? I could use vgeo properly in Canary to extract the z coordinates I need.
thanks

Need more MoistThermo conversion functions

To get Bomex running, we need several more MoistThermodynamics conversion functions:

• A saturation adjustment for liquid potential temperature
• All of the functions needed for the liquid potential temperature saturation adjustment
• A function to compute density potential temperature

Move to single package with submodules

We discussed this on Slack a bit, but should we move to a single global Project file, and make the subpackages be just submodules?

The main advantage is that we won't have to spend as much time editing and updating Manifest and Project files manually. I suspect this will get more difficult as we continue to add more functionality.

One downside is that you won't be able ] test a single subpackage any longer: we could keep the current directory structure, so you could still manually include("submodule/test/runtests.jl") if you wanted.

For the GPU support, I was thinking we could have an env/gpu subdirectory, along with a simple script which copies the top-level Project.toml, adds the GPU dependencies, and resolves as necessary.

Vertical interpolation along each column

The reading of external sounding relies on a 1D column-like interpolation between the sounding levels and the DG grid. This is done using Charlie's function "function get_z_coordinate(mesh::T) where T".

The current implementation of this can only work for flat domains. To work with topography, it will need to be generalized by extracting a one-dimensional DG grid for every column. Because it is possible that the grid structure will change once topography is added, we leave this for later.

Possible error in Prandtl value

Before I change it and affect everybody else, I believe there is a mistake in the value of

@parameter prandtl 71//10 in VanillaAtmosDistrectizations.jl

which should be

@parameter prandtl 71//100

Spherical grids

Implement something like cube-sphere meshes

Diagonalize functions that act on primitive types

I think that it would be wise to diagonalize types in functions that act on primitive types (e.g., MoistThermodynamics.jl), to provide some protection against mixed types getting accidentally passed in. For example,

Convert the following function signature from

function internal_energy_sat(T, ρ, q_t)
...
end

to

function internal_energy_sat(T :: R, ρ :: R, q_t :: R) where R <: Real
...
end