Comments (10)
That's definitely weird, it should be zero copy. Can you confirm that calling Enum.map(df.names, fn name -> df[name] end)
does not cause a big impact on memory? And then what about Enum.map(df.names, fn name -> Series.to_tensor(df[name]) end)
? Those should all be light weight references.
from explorer.
@josevalim Yeah both are pretty instant.
That's definitely weird, it should be zero copy.
In this case I do expect some churn. Polars is a column store, and Nx.stack
with axis: 1
is effectively doing a transpose.
from explorer.
Which backend are you using? EXLA or BinaryBackend?
from explorer.
Also, can you try Nx.concatenate instead of stack? I just want to rule something out quickly.
from explorer.
BinaryBackend, unfortunately :(
My intention is to pass this to EXGBoost.train
, but EXGBoost
has some issues with non-binary backends.
I've tried using the EXLA backend -> serializing -> deserializing but that didn't work. (There were many steps to the process so I might've messed something up.)
from explorer.
Oh sorry missed your earlier message.
Also, can you try Nx.concatenate instead of stack? I just want to rule something out quickly.
Will do. I'm already trying this in the background to see if it's Nx or Explorer:
transposed = Nx.stack(df) # worked pretty much instantly
Nx.transpose(transposed) # currently working hard...
Will try your thing once it's done.
from explorer.
In any case, I think we should make Nx.stack a default callback, so when invoked for EXLA, it does not perform several copies. I will open up a separate issue.
from explorer.
Nx.transpose(transposed)
OOM'd.
Nx.concatenate(df)
succeeded pretty quickly (though it has shape {1_909_672_222}
). Did you want me to pass any options to Nx.concatenate/2
?
from explorer.
I guess we can close this. We have optimized concatenate in Nx for the default backend and there is an open issue to optimize stack too.
from explorer.
Agreed!
For posterity, these additions to Nx made stack(..., axis: 1)
no longer OOM and take 3.5 mins on my VM:
My work around was taking ~11.5 min, so this is a great improvement.
Further improvements should come from addressing this issue:
from explorer.
Related Issues (20)
- Error using is_finite and is_infinite within mutate HOT 1
- Explorer NIF broken on FreeBSD HOT 12
- Support Elixir built in Duration struct HOT 1
- Bug: Rounding Error in Tests HOT 1
- exposing the `fold` expressions from Polars HOT 7
- :nif_panicked "Chunk require all its arrays to have an equal number of rows" HOT 1
- Sorting an empty DataFrame results in a runtime Polars error HOT 1
- Performance of `DataFrame.new/2` on dataframes containing list columns HOT 7
- `Series.filter` should work inside `DataFrame.summarise` HOT 5
- Large memory usage when using `Explorer.Dataframe.concat_columns` on 30k (small) data frames. Memory leak? HOT 4
- [Not Issue] - Are the plans to use duckdb as an alternative backend? HOT 2
- Support streaming: true on collect HOT 1
- Select from different dataframe inside mutate HOT 1
- Config to make table view default like in Polars itself HOT 1
- Duckdb select statement only works on non first line in the cell HOT 4
- Explorer.DataFrame.from_query/4 with :snowflake adapter returns dtype error on any numeric field HOT 3
- Orders of names after CSV loading seems buggy since 9.0 HOT 5
- Join on columns of type `:list` HOT 2
- Discrepancy between typespec for Series.cast/2 (parameter dtype) and implementation HOT 1
- LazyFrame not being able to cast dtypes HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from explorer.