enrico-dev / enrico Goto Github PK

I(Ubuntu20.04) ran this command on the build directory:
CC=mpicc CXX=mpicxx FC=mpifort cmake -DNEK_DIST=nek5000 -DUSR_LOC=../tests/singlerod/short/nek5000 ..

Here is the error message:

-- Compiling with Nek5000
-- Could NOT find SCALE (missing: SCALE_DIR)
-- Nek5000 configured with 'rodcht.usr' found in /home/fuad/thermal/enrico/tests/singlerod/short/nek5000
CMake Error at /usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:146 (message):
Could NOT find BLAS (missing: BLAS_LIBRARIES)
Call Stack (most recent call first):
/usr/share/cmake-3.16/Modules/FindPackageHandleStandardArgs.cmake:393 (_FPHSA_FAILURE_MESSAGE)
/usr/share/cmake-3.16/Modules/FindBLAS.cmake:810 (find_package_handle_standard_args)
/usr/share/cmake-3.16/Modules/FindLAPACK.cmake:197 (find_package)
vendor/nek5000/CMakeLists.txt:453 (find_package)-- Configuring incomplete, errors occurred!
See also "/home/fuad/thermal/enrico/build/CMakeFiles/CMakeOutput.log".
See also "/home/fuad/thermal/enrico/build/CMakeFiles/CMakeError.log".

The error files are:
CMakeError.log
CMakeOutput.log

NB: I was able to compile and install NekRS and successfully run examples.

The subchannel solver currently neglects frictional pressure drop along the main flow direction. This can be added to obtain a more realistic solution. If desired, we can iterate on the mass flowrate in each channel until the pressure drop over each channel is approximately constant, or we can permit a pressure distribution on the inlet for simplicity.

Currently, ENRICO calls Driver::write_step at every Picard iteration. I would be interested in specifying only certain Picard iterations and/or timestep intervals for calling write_step. For example, I'd be especially interested in calling write_step at only the final Picard iteration of each timestep.

For OpenmcDriver, this seems pretty easy to implement, since OpenmcDriver::write_step simply calls the openmc_statepoint_write API function from OpenMC. In that case, we could conditionally call write_step in CoupledDriver::execute

For NekDriver, it will be more difficult. To write the step, we don't directly call an API function from Nek5000. We simply rely on Nek to write the statepoint during the nek_solve API function, which is called in NekDriver::solve_step. Hence, conditionally calling write_step won't change the behavior. I'm looking at ways to workaround this, and I think I have some leads.

For SHIFT, I can't see what would need to happen.

Moving issue 3 from gitlab originally submitted by @RonRahaman:

The functions in nek_interface.h aren't namespaced (as OpenMC's and STREAM's functions are). Maybe namespace nek's functions, too?

Implement disjoint communicators for TH and neutronics solvers

This causes conflict with pre-installed OpenMC.

Please use external OpenMC when it is present.

The intention of the surrogate drivers is to permit easier/faster debugging and to provide some acceleration capabilities in the future. It is not the intention of this project to explore the fidelity of one conduction surrogate to another, so we can reduce some physics duplication by combining the SurrogateHeatDriver, Multi_Pin_Conduction, and Multi_Pin_Subchannel classes together into a single T/H surrogate used by both Shift and OpenMC.

The SurrogateHeatDriver includes more physics considerations than Multi_Pin_Conduction, such as a pellet-clad gap heat transfer coefficient. On the other hand, SurrogateHeatDriver has no solution in the fluid phase at all.

This MR will therefore use the solid solution from SurrogatHeatDriver and add the subchannel capabilities from Multi_Pin_Subchannel to give a single T/H surrogate. This will involve several steps:

• Add fluid phase solution to SurrogateHeatDriver

• Update OpenmcHeatDriver to include updates to fluid temperature and density

• Update Multiphysics_Driver to use consolidated T/H surrogate with new definitions for data mappings

Moving issue 29 from gitlab originally submitted by @aprilnovak:

To be able to easily interchange OpenMC/Shift and Nek5000/Surrogate, we need a shared base class for executing coupled and single-physics drivers.

Right now the SurrogateHeatDriver class is hardwired to work for a single assembly. To extend support to multiple assemblies, we'll need to add new input options in enrico.xml for specifying the placement of assemblies and an assembly pitch. In terms of changes needed in the SurrogateHeatDriver class itself, @kkiesling and I talked over a few different options:

• One option would be to add an extra dimension on the various data members that corresponds to the different assemblies, e.g., pin_centers_ would go from being xt::xtensor<double, 2> to xt::xtensor<double, 3>. The loops over pins/channels would then just need to account for the extra dimension for assemblies
• Alternatively, we could keep the SurrogateHeatDriver exactly as is, but introduce a new class on top of it that keeps a vector of SurrogateHeatDriver instances, one for each assembly. The new class would be what actually interfaces with CoupledDriver.

Right now, when we specify <n_pins_x> and <n_pins_y> for the heat/fluids surrogate, it doesn't account for guide tubes, so we end up mapping and doing data transfers on regions that don't make sense.

We recently ran into an issue with a model (longrod) where the bulk fluid temperature hit saturation. With the IAPWS functions, it should be pretty easy to check for this (e.g., on cell-average temperatures) and print out a warning to help diagnose this type of problem in the future.

The surrogate solvers in the smrt/ directory currently rely on SCALE/Trilinos for a variety of utility functions. In order to use these solvers for non-Shift builds, we'll need to remove the dependence on SCALE/Trilinos.

Moving issue 21 from gitlab originally submitted by @aprilnovak:

Enrico currently uses standard Picard iteration, where a field in iteration i + 1 is computed as the result of some operator A acting on the previous iteration field: u(i + 1) = A(u(i), v(i)). When Doppler feedback is strong and/or the fluid flowrate is sufficiently low, oscillations might be observed between successive iterations in the heat source and/or heat transfer fields. Underrelaxation is a common approach for both reducing these oscillations and speeding up slow convergence.

Underrelaxation computes a field in iteration i + 1 as a linear combination of the previous iterate and the most recently-computed approximation as u(i + 1)=(1 - a) * u(i) + A(u(i), v(i)), where a is a constant parameter in the range (0, 1]. Because underrelaxation is so commonly-used, it would be nice to have the ability to use it in Enrico. Implementing underrelaxation involves several steps:

• Create example demonstrating oscillations (without relaxation)
• Create example demonstrating slow convergence (without relaxation)
• Implement underrelaxation capabilities. People most commonly apply underrelaxation to the heat source, so I will assume relaxation of that field, with possible future extensions to other options.
• Apply underrelaxation to the two examples to demonstrate both the correct implementation and advantages of the approach.

The above tasks will assume a simple constant-a relaxation. The next steps can be to add stochastic approximation methods, which vary a (and in some cases the number of particles simulated in each Picard iteration).

• Implement Robbins-Monro stochastic approximation (variable a, constant number of particles/iteration)
• Implement Dufek-Gudowski stochastic approximation (variable a and number of particles/iteration)
• Apply Robbins-Monro to two examples
• Apply Dufek-Gudowski to two examples

During compiling, I get an error message
In Constructor It shows error ‘enrico::OpenmcDriver::OpenmcDriver(MPI_Comm)’:
error: ‘FILL_MATERIAL’ is not a member of ‘openmc' Line>40
and In Member function It shows error
‘virtual xt::xtensor<double, 1> enrico::OpenmcDriver::heat_source(double) const’:
error: ‘RESULT_SUM’ is not a member of ‘openmc’ Line > 90
What is the solution for this error?

I'd like to free a Comm in a few places. I'd like for the comm member to be freed, as well as group, size, and rank to be set to null or 0.

Should I implement this in a destructor? Or should I have a free member function?

@paulromano

Should we implement a way to suppress diagnostic messages? Currently, this includes the sanity check for volume calculations. For the disjoint communicators, I also have some diagnostics about the rank layout that I'd like to keep. But these both get pretty verbose in large-ish runs.

I'm considering either:

• A -v runtime flag
• The NDEBUG macro. This is defined by CMake for release builds. It's also understood by assert() as specified by the C++ standard. However, it doesn't appear to affect the behavior of Ensures or Expects in GSL.

I'm tending towards a -v flag.

We should be using gsl::index consistently throughout enrico.

@RonRahaman

Moving issue 20 from gitlab originally submitted by @RonRahaman:

Open question: Should we ensure that pull requests follow our source code formatting standards?

If so, we could add a format verification as a last step in the CI tests. We could specify misformatting as a yellow mark (a warning or expected error) rather than a red x.

If not, we can just periodically submit an PR where we run clang-format.

It would be useful to read/write the localq values to/from a Nek fld file. That would let us to do several useful things, including:

• Visualize the localq on the Nek mesh
• Use the localq from a restart file in a standalone Nek simulation

Moving issue 2 from gitlab originally submitted by @RonRahaman:

Right now, the OpenMC targets (openmc, libopenmc, etc.) are built in reasonable directories (./lib/ and ./bin/). However, the Nek5000 and STREAM targets are not put in reasonable directories.

Post-build, the targets will be placed like this, relative to the build directory:

• ./bin/openmc
• ./lib/libfaddeeva.a
• ./lib/libopenmc.so
• ./lib/libpugixml.a
• ./lib/libpugixml_fortran.a
• ./src/nek5000/libnek5000.a
• ./src/nek5000/nek5000
• ./libstream.a
• ./stream

I suggest we build:

• libstream and libnek5000 in ./lib/
• nek5000 and stream in ./bin/

The convention in most of this project is to define coupled drivers that only iterate between two codes. i.e. openmc_nek_driver iterates between OpenMC and Nek. The Multiphysics_Driver class iterates between a surrogate T/H model and either Shift or a two-group diffusion surrogate. For consistency, this class should be split up into two classes, one that couples Shift and the surrogate T/H, and a second that couples surrogate diffusion with surrogate T/H.

But, there may not be any motivation for coupling two surrogate models, since the surrogates are intended for debugging or coupling acceleration in the future. So, I propose renaming Multiphysics_Driver to ShiftHeatFluidsDriver and removing the ability to couple two surrogate models.

An alternative would be to split Multiphysics_Driver into two classes, with a lot of source code redundancy, since the mappings for Shift and the surrogate neutronics models are very similar.

Thoughts @sphamil @paulromano @RonRahaman ?

Moving issue 26 from gitlab originall submitted by @RonRahaman:

Currently, OpenmcHeatDriver hardcodes the number of azimuthal segments. We should probably provide a way to specify or infer this at runtime.

OpenmcHeatDriver::init_temperatures has the same bug that was fixed for init_densities in #65. ring_index and axial_index are incorrectly incremented.

Moving issue 28 from gitlab originally submitted by @aprilnovak:

To make it clear what the requirements are of a single-physics heat-fluids driver, we should have a common base class that defines the interfaces to accessing temperature and density. This concept is similar to the base class we have for neutronics drivers.

Both the surrogate T/H model and the OpenMC model specify a radial discretization in the rods. If the surrogate discretization is coarser than the OpenMC model, two bugs arise that could lead to a failure to conserve power (I think). To illustrate this by example, consider just the pellet region, with an OpenMC model with four radial cells but a surrogate model with only three radial cells:

OpenMC: x ------- x ------- x ------- x ------- x
Surrogate: x ----------- x ---------- x ---------- x

1. The ring_to_cell_inst_ member will not include all OpenMC cell instances in the pellet - only three of the cells would be included because we assemble ring_to_cell_inst_ by sweeping through center coordinates of the surrogate model. So for the above, the third OpenMC cell would not contribute a heat source to the surrogate model (and we would never even establish a tally for it).

2. The cell_inst_to_ring_ member will not include the third OpenMC cell, so it's temperature would never be updated.

Basically, if the surrogate model is coarser than the OpenMC model, some OpenMC cells will I think never be updated in temperature and never contribute a heat source to the T/H model.

I propose adding an input file parameter to say how many rings to expect in the OpenMC model and throw an error if the surrogate is too coarse. We could make this an optional field (and perform no check if absent). I foresee the case where an original model was set up correctly, but maybe the user wants to run a mesh refinement study and varies the discretization(s) without realizing they're introducing a bad mapping.

A more elegant way would be to use the pin_centers and search over the OpenMC surfaces to find the number of SurfaceZCylinders with those center coordinates to back out how many rings are in the OpenMC model, but that sort of goes against the philosophy so-far of "user needs to be careful."

Does anyone have opinions on how to ensure a good mapping here?

In principle, now that we have a consolidated driver, one can use enrico to run a coupled OpenMC+surrogate calculation that doesn't require Nek. It might be nice to allow builds of enrico without Nek. I'm thinking that if USR_LOC is not specified or turns up nothing, the build would proceed without Nek enabled and the relevant sections of the code ifdef-ed out.

Moving issue 12 from gitlab originally submitted by @pshriwise:

@rahaman and I were discussing supporting some input parameters in the stream.xml file.

We'd like to propose (at some point) placing driver parameters under and nodes, using an attribute to indicate which driver is intended for the simulation.

For example in the openmc_nek5000 cases:

<heatfluids driver="nek5000">                                                                                                       
  <casename>rodcht</casename>                                                                                                         
</heatfluids>

And in the case of the surrogate heat sovler:

<heatfluids driver="surrogate">                                                                                                                                                                                                                                                             
    <pellet_radius>0.406</pellet_radius>                                                                                                                                                                                                                                                    
    <clad_inner_radius>0.414</clad_inner_radius>                                                                                                                                                                                                                                            
    <clad_outer_radius>0.475</clad_outer_radius>                                                                                                                                                                                                                                            
    <fuel_rings>20</fuel_rings>                                                                                                                                                                                                                                                             
    <clad_rings>5</clad_rings>                                                                                                                                                                                                                                                              
    <pin_centers>                                                                                                                                                                                                                                                                           
      0. 0.                                                                                                                                                                                                                                                                                 
    </pin_centers>                                                                                                                                                                                                                                                                          
    <z>0.0 1.0 2.0 3.0 4.0 5.0 6.0 7.0 8.0 9.0 10.0</z>                                                                                                                                                                                                                                     
    <tolerance>1e-4</tolerance>                                                                                                                                                                                                                                                             
</heatfluids>

The vendored repo catch (Catch2) currently sits at a version of roughly 2.9.2, last updated in 2019, where as the latest release is 3.1.0. The current version being used here is not compatible with the M1 chip in new Macs, but that appears to be fixed in newer versions of Catch2. However, catch.hpp no longer exists with these newer versions, so headers in a few test files here will also need to be updated (see release documentation).

Moving issue 23 from gitlab originally submitted by @aprilnovak:

It seems that the .travis.yml file doesn't run unit tests or other tests that check simulation results against saved (presumably correct) values. It seems like the current tests only check whether a simulation succeeded. Is this right, or am I missing something?

If this is correct, I think it would be very helpful to add unit testing capabilities.

Currently, the user specifies an absolute tolerance for convergence via the parameter in enrico.xml. @emerzari and @hapfang suggested that we allow the user to set relative tolerance, instead.

If the neutronics and T/H solvers are to be easily swappable, the solution fields known to the coupled drivers have to be represented on the same meshes used in the single-physics drivers that they come from. For example, in OpenmcNekDriver, the power is an array of length equal to the number of OpenMC cells. This allows the OpenmcDriver to compute this array without any information regarding what T/H solver is used; then, the coupled drivers maps this data computed on the neutronics mesh onto the T/H mesh, which is T/H solver-dependent.

The ShiftNekDriver currently stores d_powers on the T/H mesh, which means that ShiftDriver cannot provide this power, and normalize it, without knowing information from Nek. We should instead change d_powers to be of length equal to the number of Shift tally cells, and project that onto the Nek mesh within ShiftNekDriver to isolate the single-physics drivers from each other.

Why isn't GSL Lite a proper submodule?

I just want to use frame of ENRICO. Can I compile it without OpenMC?

Got the same error

[100%] Linking CXX executable enrico
/usr/bin/ld: libenrico.a(nekrs_driver.cpp.o): undefined reference to symbol 'dlclose@@GLIBC_2.2.5'
/usr/bin/ld: /lib/x86_64-linux-gnu/libdl.so.2: error adding symbols: DSO missing from command line
collect2: error: ld returned 1 exit status
make[3]: *** [CMakeFiles/enrico.dir/build.make:116: enrico] Error 1
make[2]: *** [CMakeFiles/Makefile2:913: CMakeFiles/enrico.dir/all] Error 2
make[1]: *** [CMakeFiles/Makefile2:920: CMakeFiles/enrico.dir/rule] Error 2
make: *** [Makefile:235: enrico] Error 2

with gcc-9.3 and gcc-10.2. The following error from travis CI with gcc-9 and distro: ubuntu-focal

[ 98%] Building CXX object vendor/nekRS/CMakeFiles/nekrs-bin.dir/src/main.cpp.o
/home/travis/build/AI-Pranto/enrico/vendor/nekRS/src/main.cpp: In function ‘cmdOptions* processCmdLineOptions(int, char**)’:
/home/travis/build/AI-Pranto/enrico/vendor/nekRS/src/main.cpp:252:36: warning: ignoring return value of ‘int chdir(const char*)’, declared with attribute warn_unused_result [-Wunused-result]
  252 |     if(casepath.length() > 0) chdir(casepath.c_str());
      |                               ~~~~~^~~~~~~~~~~~~~~~~~
[ 98%] Linking CXX executable nekrs
/usr/bin/mpicxx   -fopenmp -O3 -DNDEBUG    CMakeFiles/nekrs-bin.dir/src/main.cpp.o  -o nekrs  -Wl,-rpath,/home/travis/build/AI-Pranto/enrico/tests/singlerod/short/build/vendor/nekRS:/home/travis/build/AI-Pranto/enrico/tests/singlerod/short/build/lib:::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::::: libnekrs.so ../../lib/libocca.so 
[ 98%] Built target nekrs-bin
Scanning dependencies of target nek5000_deps
[ 98%] Creating directories for 'nek5000_deps'
[ 99%] No download step for 'nek5000_deps'
[ 99%] No patch step for 'nek5000_deps'
[ 99%] No update step for 'nek5000_deps'
[ 99%] No configure step for 'nek5000_deps'
[ 99%] Performing build step for 'nek5000_deps'

ERROR: Cannot find a supported compiler!

travis CI

Right now, ENRICO defines its own CellInstance type that provides a bit more functionality over OpenMC's CellInstance type. We might want to consider moving the extra functionality from ENRICO into OpenMC.

Somewhat orthogonal to #61

By default, GSL Lite defines gsl::index as size_t. This contradicts the contradicts the C++ Core Guidelines, which strongly recommends a signed type for indices. However, one of the intended features of GSL Lite is the -Dgsl_CONFIG_SPAN_INDEX_TYPE macro, which can be used to redefine gsl::index (link).

It gets resolved like this in gsl-lite.hpp:

#ifndef  gsl_CONFIG_SPAN_INDEX_TYPE
# define gsl_CONFIG_SPAN_INDEX_TYPE  size_t
#endif

and then:

// index type for all container indexes/subscripts/sizes
typedef gsl_CONFIG_SPAN_INDEX_TYPE index;   // p0122r3 uses std::ptrdiff_t

Can we set -Dgsl_CONFIG_SPAN_INDEX_TYPE=ptrdiff_t when we configure ENRICO?

The subchannel solver currently assumes zero thermal resistance between the cladding and the coolant, which will tend to underpredict solid temperatures. We can add a convective heat transfer correlation to better account for this resistance.

Moving issue 27 from gitlab originally submitted by @aprilnovak:

The CoupledDriver currently only has a notion of solutions at the current and previous Picard iterations. The heat-fluids solver is usually run as a transient, which performs timestepping based on some solution at the start of the timestep. This initial solution from which the T/H solver marches forward in time should be the same as the CoupledDriver::temperature_ at the start of each timestep if we ever want to run transients (for pseudo-transients where we simulate to a steady state instead of solving PDEs with no time derivatives, the initial condition is irrelevant).

At the start of each timestep in CoupledDriver, we should tell the T/H solver what temperature it should use as the initial condition for that timestep. We can add a temperature_prev_ member to the HeatDriver and NekDriver to represent the temperature at the last timestep.

Moving issue 31 from gitlab originally submitted by @aprilnovak:

If we're going to save densities between Picard iterations and possibly use them for computing norms, we need to save the initial value used in the neutronics solver (since it runs first). So, CoupledDriver should add densities_ and densities_prev_ members, where densities_prev for the first iteration is either:

• read from the OpenMC input file, or

• computed based on Nek temperatures from a restart file using IAPWS

We can indicate which of these two options we would like to use by adding another XML tag in enrico.xml - density_ic, similar to temperature_ic.

My system places sysctl.h under /usr/include/linux.

CC=mpicc CXX=mpicxx FC=mpifort cmake -DNEK_DIST=nekrs ../../../..

[ 62%] Building CXX object vendor/nekRS/home/pranto/github/enrico/tests/singlerod/short/build/_deps/occa_content-build/CMakeFiles/libocca.dir/src/tools/sys.cpp.o
/home/pranto/github/enrico/tests/singlerod/short/build/_deps/occa_content-src/src/tools/sys.cpp:16:12: fatal error: sys/sysctl.h: No such file or directory
   16 | #  include <sys/sysctl.h>
      |            ^~~~~~~~~~~~~~
compilation terminated.
make[2]: *** [vendor/nekRS/home/pranto/github/enrico/tests/singlerod/short/build/_deps/occa_content-build/CMakeFiles/libocca.dir/build.make:2932: vendor/nekRS/home/pranto/github/enrico/tests/singlerod/short/build/_deps/occa_content-build/CMakeFiles/libocca.dir/src/tools/sys.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:1187: vendor/nekRS/home/pranto/github/enrico/tests/singlerod/short/build/_deps/occa_content-build/CMakeFiles/libocca.dir/all] Error 2
make: *** [Makefile:149: all] Error 2

I didn't see this on ubuntu-20.04.

• OS: POP!_OS 20.10
• kernel: 5.8.0-7630-generic

Moving issue 30 from gitlab originally submitted by @RonRahaman:

Currently, variable density is calculated in OpenmcNekDriver. We'd like to move this correction into either NekDriver or Nek itself.

Refs #14

In a 2x2 square lattice after adding an outer universe lattice.outer for the space outside of the defined lattice, It throws the following error message,

[ENRICO]: Updating temperature
terminate called after throwing an instance of 'std::out_of_range'
  what():  vector::_M_range_check: __n (which is 0) >= this->size() (which is 0)

If I remove lattice.outer, it fixes the issue. I have attached my input file here

I found this error message first in hexagonal lattice, and later I was able to reproduce the same error message in RectLattice by adding an outer universe.

@RonRahaman @pshriwise @lebuller and I discussed on a call this morning that the + 1 that is needed when going from 0-based arrays in ENRICO to 1-based arrays in Nek5000 should be pushed one level down, i.e., calls to temperature_at() and friends should not include the + 1. Instead those should happen at the point Nek functions are called.

Now that we have a subchannel solver in SurrogateHeatDriver, it would be nice if the VTK files that are spit out included the temperature solution not only in the pins but also in the fluid.

I have some proposals for updating the format of enrico.xml. Some of them are cleanup, and some of them are relevant to the disjoint communicator scheme.

As an example, consider the current enrico.xml for the shortrod case:

<?xml version="1.0"?>
<enrico>
  <driver_transport>openmc</driver_transport>
  <driver_heatfluids>nek5000</driver_heatfluids>
  <nek5000>
    <casename>rod_l</casename>
  </nek5000>
  <communication>overlapping</communication>
  <power>16.4e3</power>
  <pressure_bc>12.7553</pressure_bc>
  <max_timesteps>1</max_timesteps>
  <max_picard_iter>4</max_picard_iter>
  <openmc_procs_per_node>1</openmc_procs_per_node>
  <epsilon>0.5</epsilon>
</enrico>

I propose changes that look like this:

<?xml version="1.0"?>
<enrico>
  <neutronics>
    <driver>openmc</driver>
    <procs_per_node>1</procs_per_node>
    <nodes>4</nodes>
  </neutronics>

  <heat_fluids>
    <driver>nek5000</driver>
    <nodes>4</nodes>
    <casename>rod_l</casename>
  </heat_fluids>

  <power>16.4e3</power>
  <pressure_bc>12.7553</pressure_bc>
  <max_timesteps>1</max_timesteps>
  <max_picard_iter>4</max_picard_iter>
  <epsilon>0.5</epsilon>
</enrico>

Here are the important bits:

• There are separate XML elements for each driver's options
• Each driver has tags for <nodes> and <procs_per_node> that will be used to setup the driver's communicator.
  • If <nodes> and/or <procs_per_node> are omitted (or has a value <= 0), the driver will use all available nodes and/or all available processes-per-node, respectively.
  • If <nodes> is specified, the neutronics solver will use the "left-hand" nodes (the lowermost node indices according to an arbitrary node-indexing scheme) and the heat-fluids solver will use the right-hand nodes. Note that this allows arbitrarily-overlapping communicators without having to specify a particular scheme: one special case is fully-overlapping and one special case is fully-disjoint.

The left-hand/right-hand node splitting scheme is being prototyped here and is working well so far. This XML format proposal will correspond nicely to this.

Moving issue 24 from gitlab originally submitted by @aprilnovak:

Convergence can be improved if the fission source in iteration i+1 is taken from iteration i. The CoupledDriver currently calls the following OpenMC functions, in this order:

1. openmc_simulation_init: initialization stuff to be performed once per multiphysics solve; openmc_reset(); initialize_source()
2. openmc_run: loop over batches
3. openmc_statepoint_write()
4. openmc_simulation_finalize(): finalization stuff to be performed once per multiphysics solve; write tally results.

From my understanding, this means that we're doing a lot of duplicate work for each Picard iteration that we could reduce by dividing openmc_simulation_init and openmc_simulation_finalize into smaller functions that perform tasks needed once per multiphysics solve (like initializing the source, if we want to re-use it between Picard iterations) and once per iteration (like resetting tallies).