Comments (5)
I went over these messages and would tend to agree that .npy looks like a good match. Reading its description it looks like our use case and it has the bindings one might need to use it directly. I like the idea of a tar with a yaml header plus the grid as npy.
from eko.
Incidentally we speeded up quite a lot the IO operations @Zaharid @felixhekhorn.
Most likely since the yaml
parser is quite slow (and maybe same thing for the writer), so we get quite a lot from stripping the bulk data from the .yaml
.
from eko.
The full proposal is then to make the output:
- a
.tar
archive of a folder containing- a
.yaml
file with metadata - a
.npy.lz4
rank 5 tensor
- a
from eko.
Out of curiosity, how big are these files typically?
from eko.
It really depends on how many Q2 you ask for, and which interpolation grid you use (on input and output, there is a third one used internally that does not affect the output shape, and it's used to increase accuracy even when the input and output are rough; by default they are all the same, but maybe there won't be a default at all).
I tested with current version, and for a single Q2, with the same xgrid in input and output of 49 points, you get:
908k
compressed11M
uncompressed
For the uncompressed is essentially (whatever block size) x input_xgrid x output_xgrid x #Q2
, for the compressed one might be less trivial (even more considering that we'll compress the rank 5 tensor, so all Q2 together).
from eko.
Related Issues (20)
- Typo in docs HOT 2
- `evolve_pdfs` doesn't produce correct LHAPDF grids
- Is `matching_order` correctly determined? HOT 8
- On the backwards-compatibility or future-proofing of ekos HOT 15
- Remove photon and top from LHAPDF grid if they are not active HOT 5
- seemingly infinite error loop when running evolve_pdfs
- infinite error loop running evolve_pdfs HOT 5
- Packaging for rust
- Release for py 3.12 HOT 1
- Update `eko.io.dictlike` HOT 2
- Add a_Qg^3
- Version being saved as 0.0.0 HOT 3
- Couplings method is called with array instead of float
- Publish Rust crates HOT 5
- Update N3LO P_gq with newer results. HOT 1
- Define MSRV HOT 2
- Segmentation fault when running benchmark after rustification HOT 10
- Clarify `genpdf` usage HOT 2
- Oxidation roadmap
- Fix maturin deployment to PyPI HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from eko.