strategicblue / parquet-floor

A lightweight Java library that facilitates reading and writing Apache Parquet files without Hadoop dependencies

parquet-floor

A lightweight Java library that facilitates reading and writing Apache Parquet files without Hadoop dependencies

Distribution

This library is distributed via Maven Central.

<dependency>
    <groupId>blue.strategic.parquet</groupId>
    <artifactId>parquet-floor</artifactId>
    <version>1.36</version>
</dependency>

Mission

Reading Parquet files in Java ought to be easy, but you can't seem to avoid pulling in most of Hadoop as dependencies. There are quite a few people complaining about this (e.g. https://stackoverflow.com/questions/59939309/read-local-parquet-file-without-hadoop-path-api, https://stackoverflow.com/questions/29279865/parquet-without-hadoop and https://issues.apache.org/jira/browse/PARQUET-1126), but there are no simple solutions out there.

This library is put together using the fewest possible dependencies. In order to avoid pulling in the Hadoop dependency tree, it deliberately re-implements certain classes in the org.apache.hadoop package. Code has been lifted from the Apache Hadoop project (particularly https://github.com/apache/hadoop/tree/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs).

parquet-floor's Issues

Add factory method to ParquetWriter for OutputStream

It'd be nice to have a flexibility to provide own OutputStream or OutputFile.
ParquetWriter.writeOutputFile() is currently private and therefore user has no means of providing anything but file.

Do you want repeated value support?

Hi, I forked the parquet-floor repository for use in Marginalia Search, and hacked in support for repeated values (both java.util.List and Trove Lists, but the second is due to my particular needs for transporting lists of primitive values).

Give me a holler if you want this functionality in the mainline repo and I'll see about cleaning up the implementation making a pull request for backporting it here.

can support java8

Unable to open generated parquet files in python

I'm getting following error when opening in python3 with latest pyarrow, fastparquet:

pyarrow.lib.ArrowInvalid: Could not open Parquet input source '<Buffer>': Parquet magic bytes not found in footer. Either the file is corrupted or this is not a parquet file.