Concurrent usage about parquet

[cswank]

I haven't tried that, but I'm certain that concurrent writing would not be safe.

One thing you might want to consider is writing your data to multiple parquet files instead. At parsyl we build sensor devices and each device has its own parquet file, so we have 1000s of them.

You would just have to determine if there is a natural way to partition your data. We parttion by device. Maybe partitioning by date (one parquet file per month) or something like that might work for you too.

[aminmir326]

Thank you for the reply. I figured writing to multiple files would be the way to go.