balek / liquibase-impala

Liquibase extension to add Impala Database support

Table of contents

1. About liquibase-impala
2. Notes on compatibility
3. How to install
4. How to use
   • with a Maven plugin
   • with a standalone liquibase binary
   • Liquibase-impala specific configuration
5. How to test locally

About liquibase-impala

Liquibase-impala is a Liquibase extension, which adds support for Impala and Hive.

Notes on compatibility

As of version 1.1.x the plugin was tested and should work with the following versions of external dependencies:

Dependency	Versions
Liquibase	3.5.2, 3.5.3
Impala JDBC driver	Cloudera Impala JDBC 2.5.32
Hive JDBC driver	Cloudera Impala Hive 2.5.18

Version 1.2.x

Dependency	Versions
Liquibase	3.5.2
Impala JDBC driver	Cloudera Impala JDBC 2.6.4
Hive JDBC driver	Cloudera Impala Hive 2.6.2

Other configurations are likely to work too so you are encouraged to test with your versions. Let us know when you do!

How to install

version 1.1.x

As of version 1.1.x liquibase-impala depends on proprietary Cloudera connectors for Impala and Hive. These are not present in any public Maven repositories. Therefore, to build and install the plugin, you must do the following:

1. Download Impala JDBC driver and its dependencies from http://www.cloudera.com/downloads/connectors/impala/jdbc/2-5-32.html
2. Download Hive JDBC driver from http://www.cloudera.com/downloads/connectors/hive/jdbc/2-5-18.html
3. Unpack and install the following dependencies in your local Maven repository, using standard Maven command: mvn install:install-file -Dfile=${file} -DgroupId=${groupId} -DartifactId=${artifactId} -Dversion=${version} -Dpackaging=jar

file	groupId	artifactId	version
ql.jar	com.cloudera.impala.jdbc	ql	2.5.32
hive_metastore.jar	com.cloudera.impala.jdbc	hive_metastore	2.5.32
hive_service.jar	com.cloudera.impala.jdbc	hive_metastore	2.5.32
ImpalaJDBC41.jar	com.cloudera.impala.jdbc	ImpalaJDBC41.jar	2.5.32
TCLIServiceClient.jar	com.cloudera.impala.jdbc	TCLIServiceClient.jar	2.5.32
HiveJDBC41.jar	com.cloudera.hive.jdbc	HiveJDBC41.jar	2.5.18

4. (optional, but recommended) Deploy the above artifacts to an internal, private Maven repository such as Nexus or Artifactory, for subsequent use.
5. Build liquibase-impala by executing mvn clean install. This will install liquibase-impala in your local Maven repo and create a liquibase-impala.jar fat-jar in the target/ directory.
6. (optional, but recommended) Deploy liquibase-impala to your internal, private Maven repository.

version 1.2.x

1. Download Impala JDBC driver and its dependencies from https://www.cloudera.com/downloads/connectors/impala/jdbc/2-6-4.html
2. Download Hive JDBC driver from https://www.cloudera.com/downloads/connectors/hive/jdbc/2-6-2.html
3. Unpack and install the following dependencies in your local Maven repository, using standard Maven command: mvn install:install-file -Dfile=${file} -DgroupId=${groupId} -DartifactId=${artifactId} -Dversion=${version} -Dpackaging=jar

file	groupId	artifactId	version
ImpalaJDBC41.jar	com.cloudera.impala.jdbc	ImpalaJDBC41	2.6.4
HiveJDBC41.jar	com.cloudera.hive.jdbc	HiveJDBC41	2.6.2

4. (optional, but recommended) Deploy the above artifacts to an internal, private Maven repository such as Nexus or Artifactory, for subsequent use.
5. Build liquibase-impala by executing mvn clean install. This will install liquibase-impala in your local Maven repo and create a liquibase-impala.jar fat-jar in the target/ directory.
6. (optional, but recommended) Deploy liquibase-impala to your internal, private Maven repository.

How to use

There are two distinct ways liquibase-impala can be used to manage your Impala or Hive database.

with a Maven plugin

To use liquibase-impala in concert with liquibase-maven-plugin:

1. Make sure liquibase-impala is present in your local or remote (internal) Maven repo.
2. Add the following to your pom.xml file:

<build>
  <plugins>
    <!-- (...) -->
    <plugin>
      <groupId>org.liquibase</groupId>
      <artifactId>liquibase-maven-plugin</artifactId>
      <version>${liquibase.version}</version>
      <dependencies>
        <!-- (...) -->
        <dependency>
          <groupId>org.liquibase.ext.impala</groupId>
          <artifactId>liquibase-impala</artifactId>
          <version>${liquibase.impala.version}</version>
        </dependency>
      </dependencies>
    </plugin>
  </plugins>
</build>

3. Run Liquibase as you normally would using Maven plugin, for example:

mvn liquibase:update \
  -Dliquibase.changeLogFile=changelog/changelog.xml \
  -Dliquibase.driver=com.cloudera.hive.jdbc41.HS2Driver \
  -Dliquibase.username=<user>
  -Dliquibase.password=<password>
  -Dliquibase.url=jdbc:hive2://<host>:<port>/<database>;UID=<user>;UseNativeQuery=1

with a standalone liquibase binary

1. Make sure that liquibase is on your $PATH
2. Modify liquibase.properties according to your Impala/Hive endpoint
3. Put liquibase-impala fat-jar on your classpath, f.e. under the ${LIQUIBASE_HOME}/lib
4. Start migration, f.e.: liquibase update

Liquibase-impala specific configuration

Liquibase-impala provides a number of additional configuration parameters that can be used to influence its behaviour:

parameter	values	description
liquibase.lock	true (default), false	enables/disables locking facility for a given job
liquibase.syncDDL	true (default), false	wraps every statement with SYNC_DDL

How to test locally

Script examples/run.sh performs basic integration testing of Impala and Hive, which includes:

• update execution
• tag execution
• rollback execution

The script can be executed with the command ./run.sh <both|hive|impala> PATH_TO_LIQUIBASE_HOME