GithubHelp home page GithubHelp logo

inducer / meshmode Goto Github PK

View Code? Open in Web Editor NEW
25.0 9.0 24.0 2.9 MB

High-order unstructured mesh representation and discrete function spaces

Home Page: https://documen.tician.de/meshmode/

Python 99.73% Shell 0.27%
python finite-element-methods finite-elements discretization mesh meshes discontinuous-galerkin opencl scientific-computing

meshmode's Introduction

meshmode: High-Order Meshes and Discontinuous Function Spaces

Gitlab Build Status Github Build Status Python Package Index Release Page

Meshmode provides the "boring bits" of high-order unstructured discretization, for simplices (triangles, tetrahedra) and tensor products (quads, hexahedra). Features:

  • 1/2/3D, line/surface/volume discretizations in each, curvilinear supported.
  • "Everything is a (separate) discretization." (mesh boundaries are, element surfaces are, refined versions of the same mesh are) "Connections" transfer information between discretizations.
  • Periodic connectivity.
  • Mesh partitioning (not just) for distributed execution (e.g. via MPI).
  • Interpolatory, quadrature (overintegration), and modal element-local discretizations.
  • Independent of execution environment (GPU/CPU, numpy, ...) via array contexts.
  • Simple mesh refinement (via bisection). Adjacency currently only maintained if uniform.
  • Input from Gmsh, Visualization to Vtk (both high-order curvilinear).
  • Easy data exchange with Firedrake.

Meshmode emerged as the shared discretization layer for pytential (layer potentials) and grudge (discontinous Galerkin).

Places on the web related to meshmode:

meshmode's People

Contributors

a-alveyblanc avatar alexfikl avatar cmikida2 avatar gaohao95 avatar hirish99 avatar inducer avatar isuruf avatar kaushikcfd avatar majosm avatar matthiasdiener avatar mattwala avatar mtcam avatar shivamgupta2 avatar thomasgibson avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

meshmode's Issues

Better support moving geometry

Some problems that currently crop up:

  • Recreating meshes is awkward. @alexfikl proposes Discretization.copy(mesh=...) (or .replace(mesh=...)). (which?) That seems easy and useful, but it requires remembering the group factory (probably OK).
  • Meshes can't be created efficiently. Their data structure involves numpy at the moment. Maybe meshes should start using array contexts, too?
  • Group factories need to recompute discretization node arrays. If the mesh already uses the right unit nodes, this can probably be avoided. (#135 )

cc @alexfikl

ArrayContext.np.sqrt fails for device scalars

Doing a norm-like operation

actx.np.sqrt(actx.np.sum(x**2))

doesn't work after the transformations from #236. It seems like actx.np.sqrt doesn't like getting a scalar from sum and fails in this check

if not isinstance(assignee, Subscript):
raise ValueError("assignees in "
"ElementwiseMapKernelTag-tagged kernels must be "
"subscripts")

The same probably holds for any other elementwise operation in actx.np.

Accelerate resample_by_picking through better ordering

resample_by_picking routinely shows up at the top of our GPU profiles. Here's an example from a run (mirgecom wave-eager, nel_1d = 24, 3D, order 3):

 GPU activities:   15.95%  1.72898s     28160  61.398us  3.4240us  133.22us  resample_by_picking
                   14.57%  1.57934s     26499  59.599us  4.4480us  117.47us  multiply
                   14.41%  1.56223s      9698  161.09us  16.000us  543.23us  diff
                   12.79%  1.38640s     21601  64.182us  4.4150us  118.56us  axpbyz
                   11.66%  1.26342s      5283  239.15us  120.54us  529.85us  grudge_assign_0
                    8.81%  954.48ms     23428  40.741us  1.6640us  81.888us  axpb
                    7.93%  859.20ms     10560  81.363us  60.831us  135.04us  resample_by_mat
                    7.84%  849.96ms      1760  482.93us  481.79us  541.57us  face_mass
                    2.16%  233.96ms        62  3.7735ms  1.3440us  11.178ms  [CUDA memcpy DtoH]
                    1.58%  171.51ms      2235  76.738us  1.6640us  87.391us  [CUDA memcpy DtoD]
                    1.19%  128.99ms      3523  36.612us  19.328us  37.952us  [CUDA memset]
                    0.49%  53.375ms       440  121.31us  120.67us  136.42us  grudge_assign_1
                    0.49%  53.364ms       440  121.28us  120.67us  136.19us  grudge_assign_2
                    0.07%  7.2597ms       127  57.162us  1.1520us  847.90us  [CUDA memcpy HtoD]
                    0.03%  3.5939ms        12  299.49us  13.696us  529.66us  nodes_0
                    0.01%  1.4677ms         6  244.62us  14.112us  528.73us  actx_special_sqrt
                    0.00%  359.81us         6  59.967us  5.0880us  115.14us  divide
                    0.00%  136.06us         1  136.06us  136.06us  136.06us  actx_special_exp

It's especially striking that it's at the top of the list because it touches lower-dimensional data. (surface vs volume) multiply has a similar number of calls, but it touches volume data, and it completes more quickly.

I think there are two opportunities here that we could try:

  • Currently, the kernel has an indirection on the read and the write end (see the (very simple) source). For surjective/onto connections, we can do away with the indirection on write by appropriately sorting the source indices.
  • Even for non-surjective connections, it's likely that we would benefit by sorting by the write index, to try to keep the writes as coalesced as possible.

IMO it's likely that this will have a benefit (but I obviously can't guarantee it). I think it's worth trying.

cc @lukeolson

Updating how we flatten DOF arrays in anticipation for lazy evaluation

This issue is specifically addressing how to update the flatten routine for DOFArrays, found here:

def _flatten_dof_array(ary: Any, strict: bool = True):
if not isinstance(ary, DOFArray):
if strict:
raise TypeError(f"non-DOFArray type '{type(ary).__name__}' cannot "
"be flattened; use 'strict=False' to allow other types")
else:
return ary
actx = ary.array_context
if actx is None:
raise ValueError("cannot flatten frozen DOFArrays")
@memoize_in(actx, (_flatten_dof_array, "flatten_prg"))
def prg():
return make_loopy_program(
"{[iel,idof]: 0<=iel<nelements and 0<=idof<ndofs_per_element}",
"""result[grp_start + iel*ndofs_per_element + idof] \
= grp_ary[iel, idof]""",
name="flatten")
group_sizes = [grp_ary.shape[0] * grp_ary.shape[1] for grp_ary in ary]
group_starts = np.cumsum([0] + group_sizes)
result = actx.empty(group_starts[-1], dtype=ary.entry_dtype)
for grp_start, grp_ary in zip(group_starts, ary):
actx.call_loopy(prg(),
grp_ary=grp_ary,
result=result,
grp_start=grp_start)
return result

Basically, the summary of the problem is that we need to remove stateful updates of the array since this will break lazy-evaluation. The intent of #192 is to hunt down and update all instances of stateful array updates in meshmode. Specifically for the flattening routines, this bit of code is the main culprit we need to get ride of:

result = actx.empty(group_starts[-1], dtype=ary.entry_dtype)
for grp_start, grp_ary in zip(group_starts, ary):
actx.call_loopy(prg(),
grp_ary=grp_ary,
result=result,
grp_start=grp_start)

What was originally proposed in #192 (which I really liked 😞) was:

def _flatten_dof_array(ary: Any, strict: bool = True):
    if not isinstance(ary, DOFArray):
        if strict:
            raise TypeError(f"non-DOFArray type '{type(ary).__name__}' cannot "
                    "be flattened; use 'strict=False' to allow other types")
        else:
            return ary

    actx = ary.array_context
    if actx is None:
        raise ValueError("cannot flatten frozen DOFArrays")

    return actx.np.concatenate([actx.np.reshape(grp_ary, -1)
                                for grp_ary in ary])

But, turns out this isn't what we want to do since we want to maintain flexibility with regards to the memory layout of the array (x-ref: #192 (comment)).

I would appreciate some guidance on how to proceed forward now that #196 is in. cc @alexfikl, @inducer

Multiple boundary tags in meshmode export

.. warning::
Currently, no custom boundary tags are exported along with the mesh.
:mod:`firedrake` seems to only allow one marker on each facet, whereas
:mod:`meshmode` allows many.

FWIW, since firedrakeproject/firedrake#2007 entities can be marked with multiple tags and it all works. After building the plex, create a "Face Sets" label (or if you don't like that literal, a firedrake.cython.dmcommon.FACE_SETS_LABEL) (or CELL_SETS_LABEL for cells) and mark the points.

Two `resample_by_picking` in simple-dg TranslationUnit

I was just playing with (and being amazed by!) https://github.com/inducer/meshmode/pull/216.(https://github.com/inducer/meshmode/pull/216/commits/1de7f8f8fc3518ead4cd0191adf2238078636543 specifically, and inducer/pytato@585576a)

If I run python simple.dg --lazy and set a break point at https://github.com/inducer/arraycontext/blob/82b03d20849d9aac0f54cb4e9c00ba0f5908911c/arraycontext/impl/pytato/compile.py#L306, and then go

print(self.pytato_program.program)

the resulting TranslationUnit comes out with two copies of resample_by_picking that aren't visibly different:

>>> print(self.pytato_program.program)
---------------------------------------------------------------------------
KERNEL: resample_by_picking
---------------------------------------------------------------------------
ARGUMENTS:
ary: type: np:dtype('float64'), shape: (nelements_vec, n_from_nodes), dim_tags: (N1:stride:n_from_nodes, N0:stride:1), offset:  aspace: global
from_element_indices: type: np:dtype('int64'), shape: (nelements), dim_tags: (N0:stride:1), offset:  aspace: global
n_from_nodes: ValueArg, type: np:dtype('int32')
n_to_nodes: ValueArg, type: np:dtype('int64')
nelements: ValueArg, type: np:dtype('int64')
nelements_vec: ValueArg, type: np:dtype('int32')
pick_list: type: np:dtype('int32'), shape: (n_to_nodes), dim_tags: (N0:stride:1), offset:  aspace: global
result: type: np:dtype('float64'), shape: (nelements, n_to_nodes), dim_tags: (N1:stride:n_to_nodes, N0:stride:1), offset:  aspace: global
---------------------------------------------------------------------------
DOMAINS:
[nelements] -> { [iel] : 0 <= iel < nelements }
[n_to_nodes] -> { [idof] : 0 <= idof < n_to_nodes }
---------------------------------------------------------------------------
INAME IMPLEMENTATION TAGS:
idof: ConcurrentDOFInameTag()
iel: ConcurrentElementInameTag()
---------------------------------------------------------------------------
INSTRUCTIONS:
for idof, iel
    result[iel, idof] = ary[from_element_indices[iel], pick_list[idof]] if from_element_indices[iel] != -1 else 0  {id=insn}
end idof, iel
---------------------------------------------------------------------------
---------------------------------------------------------------------------
KERNEL: resample_by_picking_0
---------------------------------------------------------------------------
ARGUMENTS:
ary: type: np:dtype('float64'), shape: (nelements_vec, n_from_nodes), dim_tags: (N1:stride:n_from_nodes, N0:stride:1), offset:  aspace: global
from_element_indices: type: np:dtype('int32'), shape: (nelements), dim_tags: (N0:stride:1), offset:  aspace: global
n_from_nodes: ValueArg, type: np:dtype('int32')
n_to_nodes: ValueArg, type: np:dtype('int64')
nelements: ValueArg, type: np:dtype('int64')
nelements_vec: ValueArg, type: np:dtype('int32')
pick_list: type: np:dtype('int32'), shape: (n_to_nodes), dim_tags: (N0:stride:1), offset:  aspace: global
result: type: np:dtype('float64'), shape: (nelements, n_to_nodes), dim_tags: (N1:stride:n_to_nodes, N0:stride:1), offset:  aspace: global
---------------------------------------------------------------------------
DOMAINS:
[nelements] -> { [iel] : 0 <= iel < nelements }
[n_to_nodes] -> { [idof] : 0 <= idof < n_to_nodes }
---------------------------------------------------------------------------
INAME IMPLEMENTATION TAGS:
idof: ConcurrentDOFInameTag()
iel: ConcurrentElementInameTag()
---------------------------------------------------------------------------
INSTRUCTIONS:
for idof, iel
    result[iel, idof] = ary[from_element_indices[iel], pick_list[idof]] if from_element_indices[iel] != -1 else 0  {id=insn}
end idof, iel
---------------------------------------------------------------------------
(...SNIP...)

Any thoughts on why that might be?

cc @kaushikcfd @matthiasdiener

Adapt Firedrake interop to Firedrake complex branch

# All firedrake functions are the same dtype
dtype = self.firedrake_fspace().mesh().coordinates.dat.data.dtype
self._validate_field(mm_field, "mm_field", dtype=dtype)

seems to require all fields to agree with the geometry dtype, which seems overly restrictive given that, as of the complex merge, Firedrake supports complex data. Florian Bruckner reported this on Slack, with a backtrace like the following:

doublelayer_sphere.py:4: DeprecationWarning: meshmode.array_context is deprecated. Import this functionality from the arraycontext top-level package instead. This shim will remain working until 2022.
  from meshmode.array_context import PyOpenCLArrayContext
/firedrake/src/pytential/pytential/symbolic/primitives.py:1584: UserWarning: specified the name of the direction vector
  return DirectionalSourceDerivative(
/firedrake/src/sumpy/sumpy/kernel.py:1246: UserWarning: specified the name of the direction vector
  return type(kernel)(
/firedrake/src/firedrake/firedrake/external_operators/potential_evaluation/potentials.py:196: UserWarning: Functions in continuous function space will be projected to/from a 'Discontinuous Lagrange' space. Make sure any operators evaluated are continuous. Pass warn_if_cg=False to suppress this warning.
  warn("Functions in continuous function space will be projected "
/firedrake/src/loopy/loopy/target/execution.py:185: ParameterFinderWarning: Unable to generate code to automatically find 'nunit_dofs' from the shape of 'dst':
division with remainder in linear solve for 'nunit_dofs'
  warn("Unable to generate code to automatically "

Traceback (most recent call last):
  File "doublelayer_sphere.py", line 20, in <module>
    K = DoubleLayerPotential(u1, LaplaceKernel(dim=3), places, actx=actx, function_space=V, op_kwargs={'qbx_forced_limit': None})
  File "/firedrake/src/firedrake/firedrake/external_operators/abstract_external_operators.py", line 133, in evaluate
    return self._evaluate_action(x, *args, **kwargs)
  File "/firedrake/src/firedrake/firedrake/external_operators/potential_evaluation/potentials.py", line 144, in _evaluate_action
    return self._evaluate()
  File "/firedrake/src/firedrake/firedrake/external_operators/potential_evaluation/potentials.py", line 105, in _evaluate
    return self._eval_potential_operator(self.density, out=self)
  File "/firedrake/src/firedrake/firedrake/external_operators/potential_evaluation/potentials.py", line 99, in _eval_potential_operator
    return self.connection.to_firedrake(potential, out=out)
  File "/firedrake/src/firedrake/firedrake/external_operators/potential_evaluation/pytential.py", line 190, in to_firedrake
    self.target_to_meshmode_connection.from_meshmode(evaluated_potential)
  File "/firedrake/src/meshmode/meshmode/interop/firedrake/connection.py", line 515, in from_meshmode
    self._validate_field(mm_field, "mm_field", dtype=dtype)
  File "/firedrake/src/meshmode/meshmode/interop/firedrake/connection.py", line 356, in _validate_field
    check_dof_array(field, field_name)
  File "/firedrake/src/meshmode/meshmode/interop/firedrake/connection.py", line 349, in check_dof_array
    raise ValueError(f"'{arr_name}.entry_dtype' must be {dtype},"
ValueError: 'mm_field.entry_dtype' must be float64, not 'complex128'

cc @benSepanski

Grid generation fails with dimensionally disparate npoints

It would be good if we could make rectangular grids with uniform (or arbitrary) spacing. The example here is a rectangle [0, 10] x [0 , 1] wherein uniform spacing is attempted, i.e. npts = (101, 11). The grid generator appears to fail to produce this grid.

x0 = 0.0
y0 = 0.0
x1 = 10.0
y1 = 1.0
nx = 101
ny = 11

npts = (nx, ny)
a = (x0, y0)
b = (x1, y1)

from meshmode.mesh.generation import generate_regular_rect_mesh
mesh = generate_regular_rect_mesh(a=a, b=b, n=npts)

make_face_restriction/DG face mass matrix: Handling elements with different-shaped faces (e.g. pyramids)

Way out there, I know, but @thomasgibson, @lukeolson and I were thinking about it today. Currently, the DG face mass matrix code in grudge assumes that for each volume element, one element per face is generated, winding up with nfaces * nelements elements, and that the elements for each face are numbered contiguously. This is not feasible for a pyramid, or any other element type that has more than one shape of face. (Because element groups are numbered contiguously, and the different-shaped face element must be in a different group.) With time, we should probably remove this assumption, and instead move to "face-major" numbering (from "volume-element-major" numbering currently). Unfortunately, this has an efficiency impact in DG, as the face mass matrix must then find its face data from (for e.g. tetrahedra) four different places in memory rather than one contiguous one.

Relevant source:

def make_face_restriction(actx, discr, group_factory, boundary_tag,

Modal / nodal tracking

Right now, it's possible to feed modal data to things that expect nodal data and vice versa. Everything is just a bunch of numbers in a DOFArray. Especially if @thomasgibson's entropy-stable work manages to stick, this will be a persistent source of user error. We should perhaps have the DOFArray remember the discretization_key for each group and check those on input to make sure they're as expected.

Tolerance too strict in _test_node_vertex_consistency_resampling

We have seen lots of errors like these:

"/Users/mdiener/Work/efuse/meshmode/meshmode/mesh/__init__.py", line 1207, in _test_node_vertex_consistency
    assert _test_node_vertex_consistency_resampling(mesh, mgrp, tol)
  File "/Users/mdiener/Work/efuse/meshmode/meshmode/mesh/__init__.py", line 1195, in _test_node_vertex_consistency_resampling
    assert max_el_vertex_error < tol*size, max_el_vertex_error
AssertionError: 1.144170793882154e-15

This is the code:

def _test_node_vertex_consistency_resampling(mesh, mgrp, tol):

Firedrake connection: upcoming changes to the way we handle orientations

Just FYI, we have a (longish, perhaps by October) plan to change the way we handle orientation of meshes on the Firedrake side. Rather than globally orienting the mesh such that every physical element maps to the same reference element, each physical element will be mapped onto the reference element through some member of the relevant dihedral group. This will then entail changes to the way one grabs dofs in canonical reference element order.

We haven't worked out all the details yet. The rationale for this is (in the short term somewhat prosaically) wanting to support round-tripping checkpoint files from P to Q processes. Longer term it will enable mixed cell types and even just unstructured hex meshes (where the globally consistent orientation tricks we pull are not always possible).

Sampling at a point in space

@dshtey2 is looking for a way to evaluate DOFArray-based fields at given spatial positions. This issue is intended for discussion of that functionality: what code might already exist for this, what needs to be added, etc.

cc @inducer

L2ProjectionInverseDiscretizationConnection should allow modal output

This could avoid the forced (and possibly un-needed) modal-to-nodal conversion at the end of the connection:

return DOFArray(
actx,
data=tuple(
actx.einsum("ij,ej->ei",
vandermonde_matrix(grp),
c_i,
arg_names=("vdm", "coeffs"),
tagged=(FirstAxisIsElementsTag(),))
for grp, c_i in zip(self.to_discr.groups, coefficients)
)
)

I think this is easy to do since it receives a connection as input. It just needs to check whether the output side of the connection is modal, and if so, skip the extra conversion to nodal.

I noticed this while reviewing #192.

cc @alexfikl @thomasgibson

Should node finding use Newton or Gauss-Newton?

Upsides of Gauss-Newton:

  • More robust?
  • Gives us "the closest" if the exact point can't be found

Downside:

  • Conditioning penalty via the normal equations

def _find_src_unit_nodes_via_gauss_newton(
tgt_bdry_nodes,
src_bdry_nodes,
src_grp, src_mesh_grp,
tgt_bdry_discr, src_bdry_discr,
tol):
dim = src_grp.dim
_, nelements, ntgt_unit_nodes = tgt_bdry_nodes.shape
initial_guess = np.mean(src_mesh_grp.vertex_unit_coordinates(), axis=0)
src_unit_nodes = np.empty((dim, nelements, ntgt_unit_nodes))
src_unit_nodes[:] = initial_guess.reshape(-1, 1, 1)
import modepy as mp
src_grp_basis_fcts = src_grp.basis_obj().functions
vdm = mp.vandermonde(src_grp_basis_fcts, src_grp.unit_nodes)
inv_t_vdm = la.inv(vdm.T)
nsrc_funcs = len(src_grp_basis_fcts)

cc @xywei @alexfikl @majosm

Missing transforms on compose_index_maps in DirectDiscretizationConnection

See this FIXME:

# FIXME: Current arraycontext (2021-06-17, 9e5fb5d) does not map
# the iel_init iname to a GPU axis, leading this kernel to likely
# be very slow. Fortunately, this should only run during
# problem setup.
#
# cf. https://github.com/inducer/arraycontext/pull/29
# for a strategy that could/should be used instead.

Left over from #220.

x-ref: inducer/arraycontext#29, 03d883a

cc @matthiasdiener @thomasgibson

Spurious, (seemingly) very occasional NaN in

Last seen in https://github.com/inducer/meshmode/pull/213/checks?check_run_id=2770076958:

=================================== FAILURES ===================================
_ test_flatten_unflatten[<array context factory for <pyopencl.Device 'pthread-Intel(R) Xeon(R) CPU E5-2673 v4 @ 2.30GHz' on 'Portable Computing Language'>] _
[gw0] linux -- Python 3.9.4 /home/runner/work/meshmode/meshmode/.miniforge3/envs/testing/bin/python3
Traceback (most recent call last):
  File "/home/runner/work/meshmode/meshmode/test/test_array.py", line 93, in test_flatten_unflatten
    assert flat_norm(c - c_round_trip) < 1.0e-8
AssertionError: assert nan < 1e-08
 +  where nan = flat_norm((MyContainer(name='flatten', mass=DOFArray((cl.Array([[0.00000000e+000],\n       [            nan],\n       [4.65971693e-...  1.38196601e-01,\n               -1.11022302e-16]]),))                                          ],\n      dtype=object)) - MyContainer(name='flatten', mass=DOFArray((cl.Array([[0.00000000e+000],\n       [            nan],\n       [4.65971693e-...  1.38196601e-01,\n               -1.11022302e-16]]),))                                          ],\n      dtype=object))))
=============================== warnings summary ===============================

See also #210.

cc @alexfikl

Surface refinement node-vertex consistency error on Icelake-client

Continued from the end of #118.

test$  rm *vtu;pycl test_refinement.py 'test_refine_surfaces(_acf,  "icosphere", True)' 
Traceback (most recent call last):
  File "/home/andreas/src/meshmode/test/test_refinement.py", line 360, in <module>
    exec(sys.argv[1])
  File "<string>", line 1, in <module>
  File "/home/andreas/src/meshmode/test/test_refinement.py", line 349, in test_refine_surfaces
    refined_mesh = refine_uniformly(mesh, 1)
  File "/home/andreas/src/meshmode/meshmode/mesh/refinement/__init__.py", line 991, in refine_uniformly
    refiner.refine_uniformly()
  File "/home/andreas/src/meshmode/meshmode/mesh/refinement/no_adjacency.py", line 61, in refine_uniformly
    return self.refine(flags)
  File "/home/andreas/src/meshmode/meshmode/mesh/refinement/no_adjacency.py", line 227, in refine
    new_mesh = Mesh(new_vertices, new_el_groups, is_conforming=(
  File "/home/andreas/src/meshmode/meshmode/mesh/__init__.py", line 779, in __init__
    assert _test_node_vertex_consistency(
  File "/home/andreas/src/meshmode/meshmode/mesh/__init__.py", line 983, in _test_node_vertex_consistency
    assert _test_node_vertex_consistency_resampling(mesh, mgrp, tol)
  File "/home/andreas/src/meshmode/meshmode/mesh/__init__.py", line 971, in _test_node_vertex_consistency_resampling
    assert max_el_vertex_error < tol*size, max_el_vertex_error
AssertionError: 9.901371953343942

Quite likely a numpy behavior change.

  • numpy 1.20 wheel: fine
  • numpy 1.20.1 wheel: fine
  • pip install --no-binary ":all:" numpy cython: (resulting in 1.20.1) error (only on my laptop, not on dunkel)

cc @alexfikl because it feels less lonely that way. :)

Mesh distribution fails when number of ranks >= 29

When distributing a mesh generated by generate_regular_rect_mesh, I'm encountering the following error when running on 29 or more MPI ranks:

(dg) bash-4.2$ srun -n 29 python -m mpi4py example.py
Traceback (most recent call last):
  File "<removed>/.miniconda3/envs/dg/lib/python3.8/runpy.py", line 193, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "<removed>/.miniconda3/envs/dg/lib/python3.8/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "<removed>/.miniconda3/envs/dg/lib/python3.8/site-packages/mpi4py/__main__.py", line 7, in <module>
    main()
  File "<removed>/.miniconda3/envs/dg/lib/python3.8/site-packages/mpi4py/run.py", line 196, in main
    run_command_line(args)
  File "<removed>/.miniconda3/envs/dg/lib/python3.8/site-packages/mpi4py/run.py", line 47, in run_command_line
    run_path(sys.argv[0], run_name='__main__')
  File "<removed>/.miniconda3/envs/dg/lib/python3.8/runpy.py", line 263, in run_path
    return _run_module_code(code, init_globals, run_name,
  File "<removed>/.miniconda3/envs/dg/lib/python3.8/runpy.py", line 96, in _run_module_code
    _run_code(code, mod_globals, init_globals,
  File "<removed>/.miniconda3/envs/dg/lib/python3.8/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "example.py", line 21, in <module>
    local_mesh = mesh_dist.send_mesh_parts(mesh, part_per_element, num_parts)
  File "<removed>/code/meshmode/meshmode/distributed.py", line 85, in send_mesh_parts
    parts = [partition_mesh(mesh, part_per_element, i)[0]
  File "<removed>/code/meshmode/meshmode/distributed.py", line 85, in <listcomp>
    parts = [partition_mesh(mesh, part_per_element, i)[0]
  File "<removed>/code/meshmode/meshmode/mesh/processing.py", line 163, in partition_mesh
    part_mesh = Mesh(
  File "<removed>/code/meshmode/meshmode/mesh/__init__.py", line 718, in __init__
    raise ValueError("too few bits in element_id_dtype to represent all "
ValueError: too few bits in element_id_dtype to represent all boundary tags

Below is a simplified version of the code that can be used to reproduce the error:

from mpi4py import MPI


comm = MPI.COMM_WORLD

num_parts = comm.Get_size()

from meshmode.distributed import MPIMeshDistributor, get_partition_by_pymetis
mesh_dist = MPIMeshDistributor(comm)

dim = 2

if mesh_dist.is_mananger_rank():
    from meshmode.mesh.generation import generate_regular_rect_mesh
    mesh = generate_regular_rect_mesh(a=(-0.5,)*dim,
                                      b=(0.5,)*dim,
                                      n=(32,)*dim)

    part_per_element = get_partition_by_pymetis(mesh, num_parts)

    local_mesh = mesh_dist.send_mesh_parts(mesh, part_per_element, num_parts)
else:
    local_mesh = mesh_dist.receive_mesh_part()

I also have the serialized mesh output from an attempted run with 29 ranks (make_mesh.py).

Making `quadrature_rule` a public method on relevant element groups

This is based on a discussion in inducer/grudge#62.

Currently NodalElementGroupBase has a public user-facing method weights. This is used internally in grudge when integrating on these elements, using the unit_nodes as quadrature points.

Should we move _quadrature_rule (found in here and similar places) out and add quadrature_rule as a public method to NodalElementGroupBase. This would return a modepy.Quadrature object that can be used for reasoning about quadrature accuracy (and integration obviously 😄 ).

I am in favor of this. However, we have a problem that we need to address.

Problem:
As briefly described in this issue, some element groups do not have a concrete subclass of modepy.Quadrature (e.g. JacobiGaussQuadrature, LegendreGaussQuadrature, etc.) As a concrete example, consider subclasses of _MassMatrixQuadratureElementGroup. Grudge uses the weights method via the mass matrix and the interpolation nodes (unit_nodes) to compute using "mass matrix quadrature" (using @inducer's terminology here). It's easy enough to create a modepy.Quadrature object with the relevant weights/nodes, BUT when we have code that wants to check the order of accuracy via quad.exact_to, we run into problems.

Quick summary:
Creating a generic Quadrature object as a bag of nodes and weights lacks sufficient information to determine what exact_to needs to be. Currently, you can create a generic quadrature rule, but you cannot ask quad.exact_to (because it's NOT defined!)

If we want quadrature_rule to be a public-facing method, we should resolve the problem.

Refiner.get_current_mesh() raises TypeError when called

Eg.

>>> r = Refiner(m)
>>> r.refine(np.ones(m.nelements))
32
>>> r.get_current_mesh()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/matt/src/meshmode/meshmode/mesh/refinement/__init__.py", line 239, in get_current_mesh
    len(self.last_mesh.vertices[0])),
TypeError: generate_nodal_adjacency() missing 1 required positional argument: 'groups'

Distributed setup calls _make_cross_face_batches once per face

Here. It definitely shouldn't. As the name possibly suggests, _make_cross_face_batches is supposed to be a batched routine. (called once per rank pair) @MTCam spotted this when doing some parallel runs.

This is actually pretty bad, since it creates lots of small interpolation batches rather than one big one.

Should store per-interpolation-batch surjectivity information

In the (relatively common) surjective case, we could skip emitting these global barriers:

result[iel_init, idof_init] = 0 {id=init}
... gbarrier {id=barrier, dep=init}
)

result[iel_init, idof_init] = 0 {id=init}
... gbarrier {id=barrier, dep=init}

I noticed this while reviewing #192.

cc @thomasgibson

Make refiner work in 1D

@guptashvm: The refiner should also work in 1D, i.e. just some line segments strung together. Fortunately, the connectivity computation there should be effectively trivial. @mattwala is currently trying to make that happen. Any comments on gotchas and things that may not be straightforward?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.