expediagroup / apiary-extensions

Extensions available for use in Apiary

Overview

A collection of useful extensions to Apiary. These optional extensions can be seamlessly integrated with Apiary components.

More details about each extension can be found in their respective README files.

Deployment

Easy to deploy terraform modules for these extensions are available in Apiary-extensions-terraform.

For more information about Apiary, please refer to the main Apiary project page.

Contact

Mailing List

If you would like to ask any questions about or discuss Apiary please join our mailing list at

https://groups.google.com/forum/#!forum/apiary-user

Legal

This project is available under the Apache 2.0 License.

Copyright 2018-2021 Expedia, Inc.

apiary-extensions's Issues

com.expedia.apiary.extensions.metastore.metrics.CodahaleMetricsTest failing

All tests in this class are failing with the following error

java.lang.NoClassDefFoundError: org/apache/hadoop/mapred/JobConf

Add support for onInsert to the apiary-metastore-listener

As a user of the Apiary Metastore event listener
I'd like to add support for events being sent on the onInsert method event
so that my downstream processes can react to these events.

Acceptance Criteria

• onInsert event is handled and an SNS message is sent for it.

Use standard AWS mechanism for configuring a client

Currently in the Apiary Metastore Listener we have our own environment variable for specifying the AWS region to use. We should instead use https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/java-dg-region-selection.html to be more standard with how this is done in AWS.

Add a version field to SNS messages

We expect the JSON messages sent by the Apiary SNS listener to change over time. For this reason it would be useful to have a version field that downstream processes can use to possibly alter their behaviour based on version. This is similar to what we do in https://github.com/HotelsDotCom/circus-train/tree/master/circus-train-aws-sns#json-messages - see the protocolVersion field.

move Apiary RangerAuth from apiary-metastore-docker repo

create cloudwatch codahale metastore metrics reporter

As sysops,
I want to send metastore metrics to cloudwatch,
so that we have visibility into metastore internals.

move Apiary GlueSync from apiary-metastore-docker repo

so that we can setup jenkins pipeline and publish gluesync jar to maven repos

Remove transitive dependency on JDK tools jar

The transitive dependency on

      <groupId>org.apache.hbase</groupId>
      <artifactId>hbase-client</artifactId>

in turn is attempting to pull in the JDK's tools.jar for Java 1.6 and 1.7 which causes Maven builds to fail if these versions of the JDK aren't available. Since we don't use the hbase-client at all we can just remove this dependency.