Comments (7)
Hi @hoetmaaiers π
I believe you can't change the type of the series in the transformation.
Since you are trying to execute Elixir code for this, how about using Enum.map/2
for the transformation?
Something like this:
alias Explorer.{DataFrame, Series}
df = DataFrame.from_map(%{a: ["00:30:00", "01:00:00", "05:30:00"]})
transform_duration = fn duration ->
time = Time.from_iso8601!(duration)
time.hour + (time.minute / 60) + (time.second / 60*60)
end
new_a_series = df["a"] |> Series.to_list() |> Enum.map(transform_duration) |> Series.from_list()
DataFrame.mutate(df, a: new_a_series)
WDYT?
PS: you don't need that |> Series.from_list()
part for this to work.
from explorer.
This is a bug! We shouldn't assume you're not changing dtypes when using Series.transform/2
. I'll fix it. π
from explorer.
@hoetmaaiers the solution suggested by @philss is basically what Explorer is doing under the hood anyway. The fix I pushed today means your original code will work now -- it was a bug I introduced when implementing Series.transform/2
. I accidentally had it so there was an assumption that transform would retain the original dtype and that's not every useful π. So now we check the dtype of the new list before creating a series from it.
from explorer.
Thank you @philss , thinking outside Explorer with regular Elixir, why didn't I think of this myself...
@cigrainger, does this mean @philss isn't the only way of doing this?
from explorer.
The combination of DataFrame.mutate and Series.transform isn't working for me. Probably it is my bad, but the documentation seems to miss this combination. What am I doing wrong?
df = DataFrame.from_map(%{a: ["00:30:00", "01:00:00", "05:30:00"]})
transform_duration = fn duration ->
time = Time.from_iso8601!(duration)
time.hour + (time.minute / 60) + (time.second / 60*60)
end
DataFrame.mutate(df, a: &Series.transform(&transform_duration.(&1["a"])))
Returns me this compile error: ** (CompileError) dataframe.exs:36: nested captures via & are not allowed: &transform_duration.(&1["a"])
from explorer.
You are not allowed to use & inside &, thatβs what the compile error is telling you. Could the error message have been clearer in this case?
from explorer.
No the error message is clear, no doubt. I'm struggling with the proper combination of transform and mutate and tried several approaches.
Maybe the documentation for this use case can be clearer? I based myself on the notebook example in this repo.
from explorer.
Related Issues (20)
- Seeing `:nif_not_loaded` error for `Series.split/2` when mutating a dataframe HOT 1
- [Feature request] Add support for read_database in Polars backend. HOT 1
- Using `sort_by` with a grouped data frame doesn't respect `nils:` option HOT 1
- `{:datetime, :second}` dtype support HOT 2
- Add :streaming option to DataFrame.to_csv/3 HOT 1
- Exporting to CSV with a duration column returns an error
- Regression in `DataFrame.concat_rows/2` in v0.8.2 HOT 1
- Filter throwing undefined variable error HOT 1
- Error using is_finite and is_infinite within mutate HOT 1
- Explorer NIF broken on FreeBSD HOT 12
- Support Elixir built in Duration struct HOT 1
- Bug: Rounding Error in Tests HOT 1
- exposing the `fold` expressions from Polars HOT 7
- :nif_panicked "Chunk require all its arrays to have an equal number of rows" HOT 1
- Sorting an empty DataFrame results in a runtime Polars error HOT 1
- Performance of `DataFrame.new/2` on dataframes containing list columns HOT 7
- `Series.filter` should work inside `DataFrame.summarise` HOT 5
- Large memory usage when using `Explorer.Dataframe.concat_columns` on 30k (small) data frames. Memory leak? HOT 4
- [Not Issue] - Are the plans to use duckdb as an alternative backend? HOT 2
- Support streaming: true on collect HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from explorer.