Speed comparison of package compared to Pandas / Pyarrow_ops about datafusion-python

[jorgecarleitao]

fyi @andygrove @alamb, as IMO this is another good way of benchmarking. Note that datafusion-python is just building and running the query.

Note also that the result is wrong.

[jorgecarleitao]

Hi @TomScheffers , for some reason I did not get this notification!! :(

Thanks a lot for raising this. I agree with you that it is not acceptable; the issue is valid 👍

I tried this before and IMO it is DataFusion, specifically the groupBy operation. It has a performance issue: even spark is faster than DataFusion for the groupBy. If you try an aggregation or a filter you will see that the performance is relatively good.

We are tracking this on the DataFusion side, where the penalty is observed. I think that others are observing the same in DataFusion.

[TomScheffers]

@jorgecarleitao Thank you for your reply. Obviously, the best place to solve this issue is at its source, so good luck with that. I love the work you guys are doing on datafusion as an in-memory arrow interface.

For the python users among us: I am working on a datafusion like package in python (wombat_db) to interface with arrow in a dataframe like fashion. I would like some feedback on the project, so please check it out!