[NodeJS]: readIPC from buffer fails with 'Arrow file does not contain correct header', while it works in ArrowJS about nodejs-polars

[ritchie46]

Ah.. Polars doesn't have that distinction no. So the IPC is the stream and the .arrow is the feather file as the IPC data + additional headers?

Then we must add this.

[0xgeert]

• .arrow file that works correctly
• .ipc file that doesn't work

[ritchie46]

The IPC readers are implemented upstream. Could you make this issue here? https://github.com/jorgecarleitao/arrow2

[jorgecarleitao]

I am a bit surprised about pl.readIPC(fromArrow) and pl.readIPC(fromIPC): shouldn't these be two different signatures? One thing is to read a stream (.ipc), the other is a file (.arrow). I think that we are just missing a readIPCStream in Polars' API that can read arrow streams (as opposed to arrow files).

[joshuataylor]

Hi!

I'm keen to get this into polars, as Snowflake uses this for their response format and would be awesome to get it in for reading data straight from SF into Polars.

Here is a quick primer about the streaming files from Arrow: https://arrow.apache.org/docs/python/ipc.html
And the guide here from arrow2 about reading the stream: https://jorgecarleitao.github.io/arrow2/io/ipc_stream_read.html

IMHO, supporting files initially is fine, later can do other streaming support.

I've started looking into this, and the major blocker I can see is projections.

In arrow2, projections are not supported here: https://github.com/jorgecarleitao/arrow2/blob/main/src/io/ipc/read/stream.rs#L185

So we will need to build the projection from the chunks.

Thoughts?

[stinodego]

Transfering this to the NodeJS repo as I have no way to reproduce this using Python/Rust. Not sure if this is still relevant.

[Bidek56]

@stinodego
Python Polars 0.19.2 throws the same error on this file: exceptions.ArrowErrorException: OutOfSpec("InvalidHeader")

df = pl.read_ipc('https://paste.c-net.org/ViperMoronic')

It seems that .ipc file needs to start and end with ARROW1 for Polars to work