Comments (3)
For context, the blocker here is some sort of internal support for variable chunking in Cubed? And presumably some rechunking to regular chunking will be needed at the end?
from flox.
For context, the blocker here is some sort of internal support for variable chunking in Cubed? And presumably some rechunking to regular chunking will be needed at the end?
Yes, that what I had been thinking. However, now I think it should be possible to choose the rechunk boundaries when resampling so that each output chunk has the same number of groups. For the example shown in https://flox.readthedocs.io/en/latest/implementation.html#method-blockwise, the output would have two groups per chunk, rather than (2, 2, 3, 1) groups in each chunk. (It's OK if the last chunk has fewer groups.) There is slightly more data transferred this way, but it avoids a final rechunk, which avoids a whole dataset copy, so I think it's worth a try.
from flox.
The other way to think of this then is that you want cohorts
with equal sized cohorts (except for the last one)
from flox.
Related Issues (20)
- Optimize `split_every` HOT 4
- Error when data variables have different dimensions HOT 2
- Flox seems much slower in some cases? HOT 2
- possible support for sparse arrays HOT 2
- Examples in docs can be hard to read in dark mode HOT 1
- Support scipy < 1.11 HOT 3
- Add docs for `method` guessing
- optimize nanquantile
- ⚠️ Nightly upstream-dev CI failed ⚠️
- Unable to assign flox method and quantile method in xarray_reduce HOT 2
- ⚠️ Nightly upstream-dev CI failed ⚠️
- `xarray_reduce` is incompatible with `DataArray.pipe` due to mandatory `func` kwarg. HOT 1
- Handling sparse matrices HOT 11
- TypeError: no implementation found for 'numpy.asarray' on types that implement __array_function__: [<class 'pint.Quantity'>] HOT 2
- Support grouping by multiple variables with Cubed map-reduce
- Consider `preferred_method="blockwise"` if `by` is sorted HOT 1
- why is using flox slower than not using flox (on a laptop) // one example from flox docs HOT 8
- First execution of groupby on Xarray with Flox takes a lot of time HOT 4
- make cubed support more prominent in docs
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from flox.