Comments (2)
I recall @seberg working on some ufunc methods for sparse arrays at a sprint, but don't recall if it ended up working out
That mostly worked, but was a bit slow. The point being that the approach was to extract the result sparsity pattern. Then extract the data and apply the normal ufunc to it. But at least for memory bound ufuncs like add, it was much slower (maybe 3-4x, but don't recall). Now, of course that could probably be optimized a bit by specialization e.g. of binary ufuncs. But in the end, the way scipy-sparse works, it seemed potentially useful as a fallback to implement any ufunc for any dtype NumPy supports, even if scipy sparse has never heard of them. But probably not as a replacement for most operations.
from flox.
Thanks for opening the issue!
I think the workflows we'd like to see are pretty straight forward. We'd like:
- the array being computed on to be sparse (e.g. the first argument), but producing a dense result
- The distributed array being computed on to have sparse chunks
- Probably some additional reduction methods that ignore zeros. At the very least a
count_nonzero
/nzcount
(btw, would love suggestions on naming here) method
I've thought a little bit about implementation.
- I don't know if a
ufunc
based approach will work. I recall @seberg working on some ufunc methods for sparse arrays at a sprint, but don't recall if it ended up working out - I think only the in-memory array computing layer would need any work, as combining stats across chunks should be the same
- I haven't figured out how this fits in with the existing engines. Aside from just implementing sparse support in
numpy_groupies
- A graphblas engine would probably be very fast here
- I only really care about 2d sparse arrays at the moment
from flox.
Related Issues (20)
- Optimize `split_every` HOT 4
- Error when data variables have different dimensions HOT 2
- Flox seems much slower in some cases? HOT 2
- Examples in docs can be hard to read in dark mode HOT 1
- Support scipy < 1.11 HOT 3
- Add docs for `method` guessing
- optimize nanquantile
- ⚠️ Nightly upstream-dev CI failed ⚠️
- Unable to assign flox method and quantile method in xarray_reduce HOT 2
- ⚠️ Nightly upstream-dev CI failed ⚠️
- `xarray_reduce` is incompatible with `DataArray.pipe` due to mandatory `func` kwarg. HOT 1
- Handling sparse matrices HOT 11
- TypeError: no implementation found for 'numpy.asarray' on types that implement __array_function__: [<class 'pint.Quantity'>] HOT 2
- Support grouping by multiple variables with Cubed map-reduce
- Implement `method='blockwise'` for Cubed HOT 3
- Consider `preferred_method="blockwise"` if `by` is sorted HOT 1
- why is using flox slower than not using flox (on a laptop) // one example from flox docs HOT 8
- First execution of groupby on Xarray with Flox takes a lot of time HOT 4
- make cubed support more prominent in docs
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from flox.