Filters not being pushed down to pyarrow dataset about polars

[adriangb]

Looking at the source code it seems like this should be handled:

polars/crates/polars-plan/src/logical_plan/pyarrow.rs

Line 87 in d7d720a

AnyValue::Datetime(v, tu, tz) => Some(to_py_datetime(v, &tu, tz.as_ref())),

. I'm not familiar enough with the polars codebase to diagnose further.

[ion-elgreco]

What do you see when you do df.explain() on the polars plan

[adriangb]

FILTER [(col("a").cast(Datetime(Microseconds, Some("UTC")))) == (2024-01-01 00:00:00.dt.replace_time_zone([String(earliest)]))] FROM

  PYTHON SCAN 
  PROJECT */1 COLUMNS

(by the way it's a self contained example, no external data, you can run it and play with it)

[ion-elgreco]

Maybe the feature flags are just missing, I can't see them in the Python cargo.toml. Neither in the polars crate cargo.toml can I see polars-plan/dtype-datetime

[adriangb]

Isn’t this it?

polars/py-polars/Cargo.toml

Line 54 in 467b885

"dtype-full",

[ion-elgreco]

I only see it through here:

polars/crates/polars-lazy/Cargo.toml

Lines 60 to 61 in 467b885

	"polars-plan/temporal",
	"polars-expr/temporal",

[adriangb]

I guess the easiest thing to do is to compile with a panic in there and see if it's even being hit.