llnl / serac

Serac is a high order nonlinear thermomechanical simulation code

Serac is a 3D implicit nonlinear thermal-structural simulation code. Its primary purpose is to investigate multiphysics abstraction strategies and implicit finite element-based algorithm development for emerging computing architectures. It also serves as a proxy-app for LLNL's Smith code and heavily leverages the MFEM finite element library.

This project is under heavy development and is currently a pre-alpha release. Functionality and interfaces may change rapidly as development progresses.

Documentation

Build, run, and design documentation can be found at readthedocs.

Source documentation can be found here.

Contributions

We welcome all kinds of contributions: new features, bug fixes, and documentation edits.

For more information, see the contributing guide.

License

Copyright (c) 2019-2024, Lawrence Livermore National Security, LLC. Produced at the Lawrence Livermore National Laboratory.

Copyrights and patents in the Serac project are retained by contributors. No copyright assignment is required to contribute to Serac.

See LICENSE for details.

Unlimited Open Source - BSD 3-clause Distribution
LLNL-CODE-805541

SPDX usage

Individual files contain SPDX tags instead of the full license text. This enables machine processing of license information based on the SPDX License Identifiers that are available here: https://spdx.org/licenses/

Files that are licensed as BSD 3-Clause contain the following text in the license header:

SPDX-License-Identifier: (BSD-3-Clause)

External Packages

Serac bundles some of its external dependencies in its repository. These packages are covered by various permissive licenses. A summary listing follows. See the license included with each package for full details.

PackageName: Axom
PackageHomePage: https://github.com/LLNL/axom
PackageLicenseDeclared: BSD-3-Clause

PackageName: BLT
PackageHomePage: https://github.com/LLNL/blt
PackageLicenseDeclared: BSD-3-Clause

PackageName: MFEM
PackageHomePage: https://github.com/mfem/mfem
PackageLicenseDeclared: BSD-3-Clause

PackageName: radiuss-spack-configs
PackageHomePage: https://github.com/LLNL/radiuss-spack-configs
PackageLicenseDeclared: MIT License

PackageName: uberenv
PackageHomePage: https://github.com/LLNL/uberenv
PackageLicenseDeclared: BSD-3-Clause

serac's Issues

Revisit turning off zlib variant of MFEM

This was done due to a conflict with tribol. The MFEM zlib variant was gzstream at the time. This may be unnecessary now or at least needs to be worked through.

197a066#diff-827e676eaf262e395a2cdcfdd5b597de

Add eigenvalue analysis option

Add an interface to an external numerics library (SLEPc?) to solve for eigenmodes of the deformed system.

Make "bank" overridable by environment variable

Most pipelines should be run with the project bank. But when triggering a pipeline from the outside (multi-project) or manually, the bank may not be accessible.

A solution would be to make the bank overridable by environment variable.

Be a real project and use the Gitflow workflow

https://www.atlassian.com/git/tutorials/comparing-workflows/gitflow-workflow

We shouldn't just have one branch. We should move to having most of our development be in a new "develop" branch and rename "master" to "main" (this seems to be the defacto new standard).

Add Sidre output option

In addition to VisIt and GLVis

Move to using a SidreDataCollection for state variable storage

Look at NamedFieldsMap for guidance on named lookups to the data collection.

*Solver objects unsafe

There's a lot of use of raw pointers in the BaseSolver and related classes. Aside from being ambiguous about ownership, it also means that something as simple as passing one of these objects to a function will segfault.

template < typename T >
T do_anything(T foo) {
  ...
}

DynamicSolver oper(...);
do_anything(oper); // $ Segmentation fault (core dumped)

My vote is to just avoid raw pointers as much as is practical (in favor or std::shared_ptr or reference variables), but we could also disable some copy constructors and make these objects move-only.

Add rigid body option

Include an option for individual parts to be considered as 6-dof rigid bodies.

Add thermo-mechanical coupled solver

Add both closely coupled and operator split methods for a coupled thermomechanical solver object.

Move away from mfem::Array types

When possible, I think it would be best to go to std::Vector or std::map instead of mfem::Array types. This reduces the dependency on MFEM from a user perspective.

Problem with building serac

I'm trying to build serac on my computer. I get the following Checksum error both in building developer's tools and dependencies.

Error: ChecksumError: sha256 checksum failed for /home/saman/serac/uberenv_libs/builds/spack-stage-rz4d952n/4075.patch

and follows by

    Expected 3387faf4a71efe81c0fa17410b270ca7d352081ac88d2322df3da9bb6a6a3f2d but got 42d8b2163a2f37a745800ec13a96c08a3a20d5e67af51031e51f63313d0dedd1

/home/saman/serac/uberenv_libs/spack/lib/spack/spack/package.py:1088, in do_fetch:
       1085
       1086        self.stage.cache_local()
       1087
  >>   1088        for patch in self.spec.patches:
       1089            patch.fetch(self.stage)
       1090            if patch.cache():
       1091                patch.cache().cache_local()

anyone knows how to resolve this? Thanks.

Settle on casing across project for filenames, variables, and classes

This should be discussed as a group or just go with something.

I don't care either way, I just want it to be consistent.

Pot Hole Case? Camel Case?

Rename variables used to emulate master workflow

BUILD_DEPS is the name of the variable that will trigger the master pipeline, useful to test and debug the CI itself.
But the variable name is not well chosen. Let’s name it "MASTER_WORKFLOW".

Add sensible defaults for linear and nonlinear solves

Create a flexible parsing framework

We need ONE PARSER TO RULE THEM ALL! 😄

SeracTypes.h needs to be in a namespace

For clarity of where things are coming from....

Add documentation for use of SLIC macros

A nice style guide for when to use each of the SLIC calls would be useful. E.g. when should someone call a warning vs. and error vs. an info statement.

Define style and audit casing convention for variables and classes

I believe I mostly inherited MFEM's style. However @white238 and I were talking about making it snake_case for variables, camelCase for functions, PascalCase for structs and classes, and ALL_CAPS for macros and constants. We should decide on one style and do an audit of the code to make sure it conforms.

Fix the linelastic and dynamic test failures

Create a user parameter for number of test MPI tasks

Where in the serac building workflow the MPI fix should be added?

I'm not really sure. I'll open a new ticket for this because it would be nice for it to be a user configurable parameter. @white238 would probably be the one who knows best. The one test that is still failing is a BLT one as well, so it might need to be a setting there.

Originally posted by @jamiebramwell in #43 (comment)

Enable clang on the wsl with vscode support

Define and audit doxgen style

I was a bad developer and started this project without clearly defined doxygen use guides. We should decide on a style and push it to the repo.

Add automatic memory checking in Azure

Clang sanitizers would also be nice.

Index the state vector bundle by name

For clarity for the user, I think the m_state variable should be a std::map or something like it.

Audit const correctness

I am sure in my rush to get initial functionality out, I missed some things which could (and should) be marked const.

Add better nonlinear solvers

We should at the minimum add a line-search variant of the newton solver and possibly other variants.

Improve material models

Add anisotropic generic hyperelastic materials to help with Seth Watts's project.

Add Sidre data store option

Use the MFEM-based sidre data store to output a realistic data file. See here.

issues building on WSL

I'm having some problems building the TPLs on a Windows subsystem for Linux (ubuntu 18.04).

sam@provolone:~/serac$ python3 scripts/uberenv/uberenv.py --spec=+debug

...
linalg/superlu.cpp: In destructor ‘virtual mfem::SuperLUSolver::~SuperLUSolver()’:
linalg/superlu.cpp:235:4: error: ‘ScalePermstruct_t’ was not declared in this scope
    ScalePermstruct_t * SPstruct     = (ScalePermstruct_t*)ScalePermstructPtr_;
    ^~~~~~~~~~~~~~~~~
linalg/superlu.cpp:235:4: note: suggested alternative: ‘dScalePermstruct_t’
    ScalePermstruct_t * SPstruct     = (ScalePermstruct_t*)ScalePermstructPtr_;
    ^~~~~~~~~~~~~~~~~
    dScalePermstruct_t
linalg/superlu.cpp:235:24: error: ‘SPstruct’ was not declared in this scope
    ScalePermstruct_t * SPstruct     = (ScalePermstruct_t*)ScalePermstructPtr_;
                        ^~~~~~~~
linalg/superlu.cpp:235:24: note: suggested alternative: ‘struct’
    ScalePermstruct_t * SPstruct     = (ScalePermstruct_t*)ScalePermstructPtr_;
                        ^~~~~~~~
                        struct
linalg/superlu.cpp:235:58: error: expected primary-expression before ‘)’ token
    ScalePermstruct_t * SPstruct     = (ScalePermstruct_t*)ScalePermstructPtr_;
                                                          ^
linalg/superlu.cpp:236:4: error: ‘LUstruct_t’ was not declared in this scope
    LUstruct_t        * LUstruct     = (LUstruct_t*)LUstructPtr_;
    ^~~~~~~~~~

Could one of the recent changes to the dependencies and build system in the last few days be responsible? I confess to approving those PRs without actually rebuilding the TPLs myself. That being said, it took more than 2 hours for python3 scripts/uberenv/uberenv.py --spec=+debug to produce the error above, which makes it challenging to test if a PR breaks something or not.

Explore runtime polymorphism at the solver interface level

Potential mishandling of boundary conditions

The most recent change to the boundary condition data structures now involves iterating over
each of the members of m_ess_bdr to do things like elimination of essential degrees of freedom:

void ThermalSolver::CompleteSetup(){
  ...
  // Eliminate the essential DOFs from the stiffness matrix
  for (auto &ess_bc_data : m_ess_bdr) {
    m_K_e_mat.reset(m_K_mat->EliminateRowsCols(ess_bc_data->true_dofs));
  }
  ...
}

void ThermalSolver::QuasiStaticSolve(){
  ...
  // Apply the boundary conditions
  *m_bc_rhs = *m_rhs;
  for (auto &ess_bc_data : m_ess_bdr) {
    ess_bc_data->scalar_coef->SetTime(m_time);
    temperature.gf->ProjectBdrCoefficient(*ess_bc_data->scalar_coef, ess_bc_data->bc_markers);
    temperature.gf->GetTrueDofs(temperature.true_vec);
    mfem::EliminateBC(*m_K_mat, *m_K_e_mat, ess_bc_data->true_dofs, temperature.true_vec, *m_bc_rhs);
  }
  ...
}

It seems like m_K_e_mat is just being overwritten on each loop iteration, so it's not obvious to me that this is really handling the separate entries of m_ess_bdr correctly.

Changing the thermal test from

TEST(thermal_solver, static_solve){
  ...
  // Initialize the temperature boundary condition
  std::vector<int> temp_bdr(pmesh->bdr_attributes.Max(), 1);
  auto u_0 = std::make_shared<StdFunctionCoefficient>([](const auto & x){
    return x.Norml2();
  });
  therm_solver.SetTemperatureBCs(temp_bdr, u_0);
  ...
}

to

TEST(thermal_solver, static_solve){
  ...
  // Initialize the temperature boundary condition
  std::vector<int> temp_bdr(pmesh->bdr_attributes.Max(), 1);
  auto u_0 = std::make_shared<StdFunctionCoefficient>([](const auto & x){
    return 2 * x.Norml2();
  });
  therm_solver.SetTemperatureBCs(temp_bdr, u_0);

  // whoops, that first function was wrong--  
  // let me try to correct it by setting the right values!
  u_0 = std::make_shared<StdFunctionCoefficient>([](const auto & x){
    return x.Norml2();
  });
  therm_solver.SetTemperatureBCs(temp_bdr, u_0);
  ...
}

makes the test fail (incorrect answer).

I don't really understand the implementation of boundary conditions in MFEM well enough to do much more, can anyone else comment on this? Do we need to have a separate m_K_e_mat for each BoundaryConditionData?

Add debug builds to shared library builds

Something like /path/to/shared/libs/<SYS_TYPE>/debug/<time_date>

Add simple testing integration

I think we're at a state now where we can add some simple tests using our ctest framework via either Azure pipelines or Travis. Any takers?

Document style guide

Write a minimal "style guide" that will probably be amendments to the Google C++ style guide located here.

Change from MFEM's command line arguments to CLI11

Axom has built in support for CLI11. We should switch to that and away from MFEM's. CLI11 is more industry standard and is C++ vs C (ie std::strings vs char*).

Suggestion: specifying non mandatory variants in packages.yaml

serac/scripts/uberenv/packages/serac/package.py

Line 91 in 553f454

depends_on("axom~openmp~fortran~raja~umpire")

@white238 I think it would be better to specify variants that are not mandatory in the packages.yaml (EDIT: not host-config files). This way it does not force the installation of a new axom if one is already present for example, and developer can still install the blessed configuration with uberenv.

Add the option to output the deformed mesh

For Lagrangian simulations, the mesh nodes need to be updated prior to output. This should be added as an option in the base solver class.

Add Doxygen documentation to the Sphinx page

We should probably link our Doxygen documentation to the readthedocs page. I found a good example of how to do this here. Here's another good example.

Add initial GPU implementation to thermal solver

As GPU-enabled AMG options and full element assembly come online in the MFEM library, we should test them out in the simple thermal solver.

Add real unit tests for the solver objects

I was a bad developer and really wrote integrated tests for all of the solver objects. We should add proper unit tests.

Audit the memory management strategy of the solver class

Now that a few solvers (including a composite solver) exist, a reexamination of the memory management strategies used for each object may be useful.

Add initial hooks for future contact library development

Move the nonlinear solid solver to the base solver framework

Rewrite the quasistatic and dynamic hyperelastic solvers to use the generic base solver interface.

Investigate C++ linting tools

Let's try to find a code check tool that can handle context-aware C++ style checking. cpplint is a possibility.

Improve documentation with regards to testing

Add detailed documentation (or references to external sites) about the ctest and google test frameworks. Include how to run the tests and where the executables are located.

Unify linear and nonlinear solver objects

Change the LGTM tests to run against the checked in host-configs

This will speed up the turn around and limit the amount we hog machines during the day. Then have the full TPL build only go at night. This will also guard against system changes.

Move to MFEM 4.1

Upgrade the MFEM dependency to the newly released v4.1. This should happen sooner rather than later as it greatly improves GPU support.

Add entity set manager for boundary conditions

Add a more sensible interface for boundary conditions based on lists of geometric entities (faceset, nodeset, edgeset, etc).

Switch Gitlab scripts to use build_tpls.py

We should limit the duplication of build scripts we have. This requires some copying of work that I did on Axom to allow build_tpls.py to allow "--spec=%somecompilerspec" on the command line. Then removing the part of build_and_test.sh that build the TPLs.