ex-mech / pymech Goto Github PK

• Investigate upstream use of FrozenDict and __slots__ in store class.
• Main bottleneck: Improve / find alternative to combine_by_coords function to combine elements

The sdist and wheel can be installed but they may not work. A bug fix release is required.

In some cases, loading a nek file with readnek() and writing it back with writenek() gives an output that looks messed up on visit. Slicing through it only displays some elements. Some points are projected far from where they should be. I have reproduced this with both single and double precision files, writing back in both single and double precision. The files look fine when read with readnek(). I don't know if it's a pymech bug or a visit bug. I haven't tested whether Nek reads the file properly yet.

I wonder if this is related to the element map found in the beginning of files? I have never seen this documented or mentioned on the Nek mailing list. After the header and endianness check, the files start with what appears to be a permutation table for the elements. Why is it there? Why is it not identity when it comes from Nek? readnek() reorders the elements when reading, is there a reason for this?

I found some mistakes in the path when importing pymech modules in simsonsuite.py, exadata.py, neksuite.py

• clever enough

docs/simsonsuite.rst
19:readdns() is clever enough to figure out the rest.
32:readplane() is clever enough to figure out the rest.

docs/neksuite.rst
19:readnek() is clever enough to figure out the rest.
48:readrea() is clever enough to figure out the rest.

Meshtools needs documentation with examples. Some unit tests would be good too.

To get the time corresponding to a file, I know that one can use pymech.readnek(path).time. However, if I understand correctly, it reads the whole file just to get the time. There should be an API (pymech.readtime ?) to get the time without reading the other data in the file.

Not sure what exactly that is for, but merge the branches if it is ready :)

Hi,
some time ago I wrote some very basic Python scripts to post-process the fields from my simulations in Nek5000, and some of them rely on the package pymech. One of these scripts is used to extract slices at each GLL point in the streamwise direction (cross-sections basically). I noticed that when I use the new version of pymech the script takes a huge amount of time to be executed, whereas with the older version it took few minutes. Is someone aware of some tasks that are performed in the new versions and are much heavier than before?

I attach the script as a .md since .py are not supported here
extract_slices.md

Hello,

I have experienced an error when loading a Nek5000 output file with a mesh created from GMSH. A similar case with a mesh from genbox works flowlessly.

when running the following command :
open_dataset('channel3D_0.f00001')
I get the following error :
conflicting sizes for dimension 'y': length 8 on 'xmesh' and length 9 on 'y'

Could you help me resolve this issue ?

(Visualizing it with paraview works fine in both cases, as a check)

Modules under pymech.neksuite.:

• __init__ for backwards compat
• mesh: for {read,write}{rea,re2}
• map for readma2
• field for {read,write}{nek}

There aren't many functionalities implemented in meshtools right now. A few easy functions would be potentially useful:

• Generating a box. This is trivial and hasn't been done before because the native Nek5000 tool genbox does it, but having it in pymech would allow using it as a building blocks for more complex meshes.
• A function to map geometries, for example to map a [0, 1]×[0, 1] box to an arbitrary shape with four corners. This is also trivial and would allow for generating more complex meshes.
• Coons patches? (trivial using the two functions above)
• A function to generate the outside of a circle.
• Functions to add, delete or change boundary conditions for velocity, temperature and passive scalars.

More complicated and speculative:

• A function to generate a cylinder-box junction.
• Integrating my airfoils script in some way. Perhaps by generating a boundary-layer mesh and returning some kind of analytic representation of the outer boudary. Then, combined with the functions above it might be possible to write a tutorial for generating a complete airfoil mesh with a boundary layer and a few free-stream blocks.
• Getting inspired by Vitor's extrude_refine function to perform some kind of mesh refinement.

Feel free to add anything to this list!

I managed to bring down readnek time from 4.35 seconds to 0.2 seconds with the changes: ed6ec2d...215c3c4. Notable changes are the use of np.frombuffer and .reshape functions. I think writenek could be cleaned up similarly.

In vtksuite.py, we have a function called writevtk(). keep getting thrown an error which corresponds to line 195 of writevtk() - that we cannot add a Posix path with a string. Given that writevtk is still experimental, do we correct the code before line 195 converting 'fname' from PosixPath to str() ? Would this be a bug fix?

Sent by A. Vinodh

See:

• Implementation: pydata/xarray#4577
• Example: corteva/rioxarray#281

This should make it possible to use xarray directly as:

import xarray as xr
xr.open_dataset(..., engine="pymech")

• Archive in Zenodo Sandbox as test
• Check with @eX-Mech developers for corrections
• Archive in Zenodo

This is not a good pattern to have. @guillaumechauvat had mentioned this before.

logger.error(...)
return 1

Rather than use C / Fortran style integer return, we should raise an error to avoid confusing error messages. This however might break some compatibility in user code, because someone might be not catching the errors as they should.

We can also have a PymechError custom exception if it helps.

Ideally @guillaumechauvat could do it, but it seems that we would have to try to work out some basic description from the test suite.

I use the neksuite.readnek routine to read some nek fields which are in single precision, but once I read it, in python I have arrays of float64 (looking at pos, pres, vel attributes of each element). For memory issues, I need to store my data in single precision, is there a way to directly set it in pymech?

- Originally asked by @danielemassaro

For now, meshtools assumes that there is only a velocity field. Temperature or passive scalar fields are not supported.

I encountered another error for simulations done on 20 cores related to the dataset when using pymech.dataset.open_dataset("cbox0.f00030").

File ~/useful/project/20CONVECTION/miniconda3/lib/python3.9/site-packages/pymech/dataset.py:36, in open_dataset(path, **kwargs)
     33 else:
     34     raise NotImplementedError(f"Filetype: {path.suffix} is not supported.")
---> 36 return _open(path, **kwargs)

File ~/useful/project/20CONVECTION/miniconda3/lib/python3.9/site-packages/pymech/dataset.py:152, in _open_nek_dataset(path)
    150 elements = field.elem
    151 elem_stores = [_NekDataStore(elem) for elem in elements]
--> 152 elem_dsets = [
    153     xr.Dataset.load_store(store).set_coords(store.axes) for store in elem_stores
    154 ]
    156 # See: https://github.com/MITgcm/xmitgcm/pull/200
    157 if xr.__version__ < "0.15.2":

File ~/useful/project/20CONVECTION/miniconda3/lib/python3.9/site-packages/pymech/dataset.py:153, in <listcomp>(.0)
    150 elements = field.elem
    151 elem_stores = [_NekDataStore(elem) for elem in elements]
    152 elem_dsets = [
--> 153     xr.Dataset.load_store(store).set_coords(store.axes) for store in elem_stores
    154 ]
    156 # See: https://github.com/MITgcm/xmitgcm/pull/200
    157 if xr.__version__ < "0.15.2":

File ~/useful/project/20CONVECTION/miniconda3/lib/python3.9/site-packages/xarray/core/dataset.py:770, in Dataset.load_store(cls, store, decoder)
    768 if decoder:
    769     variables, attributes = decoder(variables, attributes)
--> 770 obj = cls(variables, attrs=attributes)
    771 obj.set_close(store.close)
    772 return obj

File ~/useful/project/20CONVECTION/miniconda3/lib/python3.9/site-packages/xarray/core/dataset.py:750, in Dataset.__init__(self, data_vars, coords, attrs)
    747 if isinstance(coords, Dataset):
    748     coords = coords.variables
--> 750 variables, coord_names, dims, indexes, _ = merge_data_and_coords(
    751     data_vars, coords, compat="broadcast_equals"
    752 )
    754 self._attrs = dict(attrs) if attrs is not None else None
    755 self._close = None

File ~/useful/project/20CONVECTION/miniconda3/lib/python3.9/site-packages/xarray/core/merge.py:483, in merge_data_and_coords(data, coords, compat, join)
    481 explicit_coords = coords.keys()
    482 indexes = dict(_extract_indexes_from_coords(coords))
--> 483 return merge_core(
    484     objects, compat, join, explicit_coords=explicit_coords, indexes=indexes
    485 )

File ~/useful/project/20CONVECTION/miniconda3/lib/python3.9/site-packages/xarray/core/merge.py:640, in merge_core(objects, compat, join, combine_attrs, priority_arg, explicit_coords, indexes, fill_value)
    635 variables, out_indexes = merge_collected(
    636     collected, prioritized, compat=compat, combine_attrs=combine_attrs
    637 )
    638 assert_unique_multiindex_level_names(variables)
--> 640 dims = calculate_dimensions(variables)
    642 coord_names, noncoord_names = determine_coords(coerced)
    643 if explicit_coords is not None:

File ~/useful/project/20CONVECTION/miniconda3/lib/python3.9/site-packages/xarray/core/dataset.py:204, in calculate_dimensions(variables)
    202             last_used[dim] = k
    203         elif dims[dim] != size:
--> 204             raise ValueError(
    205                 f"conflicting sizes for dimension {dim!r}: "
    206                 f"length {size} on {k!r} and length {dims[dim]} on {last_used!r}"
    207             )
    208 return dims

ValueError: conflicting sizes for dimension 'y': length 10 on 'xmesh' and length 1 on {'x': 'x', 'y': 'y', 'z': 'z'}

Originally posted by @akhoubani in #48 (comment)

The "PSTAT" module of the KTH framework requires some specialized output, specifically positionsof the interpolated mesh. These are now some scripts adapted from pymech functions. Those can be generalized and added to pymech:

• https://github.com/KTH-Nek5000/KTH_Examples/blob/99dda707764a770b75191470d489aa07b818a7fe/phill_PSTAT3D/writer_int_pos.py
• https://github.com/KTH-Nek5000/KTH_Examples/blob/99dda707764a770b75191470d489aa07b818a7fe/phill_PSTAT3D/reader_int_fld.py

There are also 2D functions in phill_PSTAT2D/pp_Python/modules

Hi,

I have already setup the PR #106 for a dangerous section in the code. Here the coordinates could then be read as double precision floating numbers (which is originally rounded off by 8 digits). Please have a check. Thank you very much!

Hey guys,

(probably this is not the best to post this)

I have a 3d field and I need to plot the evolution of the boundary layer thickness along x.
All examples refer to .mean. I can't even obtain a slice of an instantaneous quantity.

Is there a reference to functions like mean or slice? any chance to add some more usage examples?

Thanks for your time,
R

It would be useful to be able to read only one or few variables with readnek (for example only the temperature, vx and vy). It seems to me that it would not be very complicated?

We don't need all the functionalities of pydantic. It might be good enough and safer to use attrs with explicit converters.

With a file "Snakemake" containing

import pymech
print(pymech)

rule say_hello:
    shell:
        'echo "hello"'

The logs of snakemake -c1 are repeated twice.

I guess it is similar to snek5000/snek5000#182

For most cases #31 seems to be the reason, but it appears to happen (rarely) for certain runs even with box simulations as seen in #56.

In v1.5.0 we will at the very least provide a meaningful error message.

Originally posted by @ashwinvis in #31 (comment)

Options

In the order of increasing complexity...

• Reader for ma2 files?
• Third party library networkx / trimesh, ... ?
• Native genmap reader / cffi interface?

ma2 files structure

See subroutine dmp_mapfile (pmap,nel,depth,cell,nv,nrnk,npts,noutflow) in Nek5000/tools/genmap/genmap.f and the header of the ma2 file corresponding to:

$ strings abl.ma2
#v001         392         490           8         256        3136         490           0       
                                    
$ xxd -b abl.ma2 | less
00000000: 00100011 01110110 00110000 00110000 00110001 00100000  #v001 
00000006: 00100000 00100000 00100000 00100000 00100000 00100000        
0000000c: 00100000 00100000 00110011 00111001 00110010 00100000    392 
00000012: 00100000 00100000 00100000 00100000 00100000 00100000        
00000018: 00100000 00100000 00110100 00111001 00110000 00100000    490 
0000001e: 00100000 00100000 00100000 00100000 00100000 00100000        
00000024: 00100000 00100000 00100000 00100000 00111000 00100000      8 
0000002a: 00100000 00100000 00100000 00100000 00100000 00100000        
00000030: 00100000 00100000 00110010 00110101 00110110 00100000    256 
00000036: 00100000 00100000 00100000 00100000 00100000 00100000        
0000003c: 00100000 00110011 00110001 00110011 00110110 00100000   3136 
00000042: 00100000 00100000 00100000 00100000 00100000 00100000        
00000048: 00100000 00100000 00110100 00111001 00110000 00100000    490 
0000004e: 00100000 00100000 00100000 00100000 00100000 00100000        
00000054: 00100000 00100000 00100000 00100000 00110000 00100000      0 
0000005a: 00100000 00100000 00100000 00100000 00100000 00100000        
00000060: 00100000 00100000 00100000 00100000 00100000 00100000        
00000066: 00100000 00100000 00100000 00100000 00100000 00100000        
0000006c: 00100000 00100000 00100000 00100000 00100000 00100000        
00000072: 00100000 00100000 00100000 00100000 00100000 00100000        
00000078: 00100000 00100000 00100000 00100000 00100000 00100000        
0000007e: 00100000 00100000 00100000 00100000 00100000 00100000        
00000084: 11111010 01100001 11010001 01000000 11000101 00000000  .a.@..
0000008a: 00000000 00000000 10001000 00000001 00000000 00000000  ......
00000090: 00110101 00000001 00000000 00000000 10000111 00000001  5.....
00000096: 00000000 00000000 00110100 00000001 00000000 00000000  ..4...
0000009c: 10000110 00000001 00000000 00000000 00110011 00000001  ....3.
000000a2: 00000000 00000000 10000101 00000001 00000000 00000000  ......
000000a8: 00110001 00000001 00000000 00000000 11010111 00000000  1.....
000000ae: 00000000 00000000 00110101 00000001 00000000 00000000  ..5...
000000b4: 00100101 00000001 00000000 00000000 00110100 00000001  %...4.
000000ba: 00000000 00000000 00100100 00000001 00000000 00000000  ..$...
000000c0: 00110011 00000001 00000000 00000000 00110010 00000001  3...2.
000000c6: 00000000 00000000 00110001 00000001 00000000 00000000  ..1...
000000cc: 00110000 00000001 00000000 00000000 11010111 00000000  0.....
000000d2: 00000000 00000000 00100101 00000001 00000000 00000000  ..%...
000000d8: 01101100 00000001 00000000 00000000 00100100 00000001  l...$.
000000de: 00000000 00000000 01101010 00000001 00000000 00000000  ..j...
000000e4: 00110010 00000001 00000000 00000000 01101000 00000001  2...h.
000000ea: 00000000 00000000 00110000 00000001 00000000 00000000  ..0...
000000f0: 01100110 00000001 00000000 00000000 11111011 00000000  f.....
000000f6: 00000000 00000000 01101100 00000001 00000000 00000000  ..l...
000000fc: 01101011 00000001 00000000 00000000 01101010 00000001  k...j.
00000102: 00000000 00000000 01101001 00000001 00000000 00000000  ..i...
00000108: 01101000 00000001 00000000 00000000 01100111 00000001  h...g.
0000010e: 00000000 00000000 01100110 00000001 00000000 00000000  ..f...
00000114: 01100101 00000001 00000000 00000000 11110011 00000000  e.....
0000011a: 00000000 00000000 01101011 00000001 00000000 00000000  ..k...
00000120: 00111110 00000001 00000000 00000000 01101001 00000001  >...i.
00000126: 00000000 00000000 01000110 00000001 00000000 00000000  ..F...
0000012c: 01100111 00000001 00000000 00000000 01000100 00000001  g...D.
00000132: 00000000 00000000 01100101 00000001 00000000 00000000  ..e...

A possibility would be to be able to choose between native (default), little or big endian. This can be achieved easily using ndarray.byteswap()

edit: or better, ndarray.newbyteorder.

This would make things obvious. Of course, we can keep an alias module exadata for compatibility.

Thanks to @pschlatt1 for the suggestion.

See if it is more efficient than current approach.

See https://github.com/RonRahaman/fld_pyutils/blob/master/fld_data_memmap.py

We encountered an IndexError: list index out of range in geometry reading for loading nek5000 field files. readnek works fine for some dumped field files, while fails for some other files with the same geometry. I share two field files, one working and one broken.

This field file works fine.
cbox0.f00298.txt

While the following gives the error.
cbox0.f00299.txt

home/Dev/miniconda3/lib/python3.8/site-packages/pymech/neksuite.py in readnek(fname)
    139     # read geometry
    140     for iel in elmap:
--> 141         el = data.elem[iel - 1]
    142         for idim in range(var[0]):  # if var[0] == 0, geometry is not read
    143             read_file_into_data(el.pos, idim)

IndexError: list index out of range

https://cibuildwheel.readthedocs.io/en/stable/deliver-to-pypi/
https://scikit-hep.org/developer/gha_wheels

We only need one OS and one build, since this is a pure-python project.