00mjk / connectors

Connectors for Delta Lake

Connectors

We are building connectors to bring Delta Lake to popular big-data engines outside Apache Spark (e.g., Apache Hive, Presto, Apache Flink) and also to common reporting tools like Microsoft Power BI.

Introduction

This is the repository for Delta Lake Connectors. It includes

• Delta Standalone: a native library for reading and writing Delta Lake metadata.
• Connectors to popular big-data engines (e.g., Apache Hive, Presto, Apache Flink) and to common reporting tools like Microsoft Power BI.

Please refer to the main Delta Lake repository if you want to learn more about the Delta Lake project.

Delta Standalone

Delta Standalone, formerly known as the Delta Standalone Reader (DSR), is a JVM library to read and write Delta tables. Unlike https://github.com/delta-io/delta, this project doesn't use Spark to read or write tables and it has only a few transitive dependencies. It can be used by any application that cannot use a Spark cluster.

• To compile the project, run build/sbt standalone/compile
• To test the project, run build/sbt standalone/test
• To publish the JAR, run build/sbt standaloneCosmetic/publishM2

See Delta Standalone for detailed documentation.

Connectors

Hive Connector

Read Delta tables directly from Apache Hive using the Hive Connector. See the dedicated README.md for more details.

Flink/Delta Connector

Use the Flink/Delta Connector to read and write Delta tables from Apache Flink applications. The connector includes a sink for writing to Delta tables from Apache Flink, and a source for reading Delta tables using Apache Flink (still in progress.) See the dedicated README.md for more details.

sql-delta-import

sql-delta-import allows for importing data from a JDBC source into a Delta table.

Power BI connector

The connector for Microsoft Power BI is basically just a custom Power Query function that allows you to read a Delta table from any file-based data source supported by Microsoft Power BI. Details can be found in the dedicated README.md.

Reporting issues

We use GitHub Issues to track community reported issues. You can also contact the community for getting answers.

Contributing

We welcome contributions to Delta Lake Connectors repository. We use GitHub Pull Requests for accepting changes.

Community

There are two mediums of communication within the Delta Lake community.

• Public Slack Channel

  • Register here
  • Login here

• Public Mailing list

Local Development & Testing

• Before local debugging of standalone tests in IntelliJ, run all standalone tests using SBT. This helps IntelliJ recognize the golden tables as class resources.