windpiger / connectors

Connectors for Delta Lake

Connectors

We are building connectors to bring Delta Lake to popular big-data engines outside Apache Spark (e.g., Apache Hive, Presto).

Introduction

This is the repository for Delta Lake Connectors. It includes a library for querying Delta Lake metadata and connectors to popular big-data engines (e.g., Apache Hive, Presto). Please refer to the main Delta Lake repository if you want to learn more about the Delta Lake project.

Building

The project is compiled using SBT. It has the following subprojects.

1. Delta uber jar

This project generates a single uber jar containing Delta Lake and all it transitive dependencies (except Hadoop and its dependencies).

• Most of the dependencies are shaded to avoid version conflicts. See the file build.sbt for details on what are not shaded.
• Hadoop and its dependencies is not included in the jar because they are expected to be present in the deployment environment.
• To generate the uber jar, run build/sbt core/compile
• To test the uber jar, run build/sbt coreTest/test

2. Hive connector (WIP)

This project contains all the code needed to make Hive read Delta Lake tables.

• To compile the project, run build/sbt hive/compile
• To test the project, run build/sbt hive/test
• To generate the connector jar run bulld/sbt hive/package

Refer to SBT docs for more commands.

Reporting issues

We use GitHub Issues to track community reported issues. You can also contact the community for getting answers.

Contributing

We welcome contributions to Delta Lake Connectors repository. We use GitHub Pull Requests for accepting changes.

Community

There are two mediums of communication within the Delta Lake community.

• Public Slack Channel

  • Register here
  • Login here

• Public Mailing list