I'd love to be able to read from s3, like in Apache Arrow about explorer

[philss]

Yeah, I changed my mind. I think it worth to add this feature 👍

[kimjoaoun]

You mean reading a dataframe using something like this example below?

Explorer.DataFrame.from_bucket("arn:aws:s3:::bucket_name/key_name")

[isaacsanders]

Or via URL, s3://bucket-name/data/date=2022-03-01/

[isaacsanders]

Arrow defines “file system” types that enable this behavior.

[philss]

I like the idea but I would suggest to not include such feature in the project right now because of the dependencies we would need to require.
Maybe a good approach would be to create such functionality in a separated project first. WDYT?

[isaacsanders]

It’s a first class activity in Apache Arrow, so I didn’t think it would be unusual to include here. It might be the case that implementing a package with Apache Arrow bindings and using the from_arrow function could work.

[isaacsanders]

This whole issue seems like it isn’t far off of what #91 is proposing, right?

[cigrainger]

I agree. I think the example here should give us a good start. I'll look into this.

[josevalim]

This will be closed as part of #190.

[isaacsanders]

As written, I don't see #190 as actually closing this issue.

[josevalim]

The idea is that you would read from S3, using something like Req, and then call DF.from_parquet(...) or whatever format it is on S3.