Kafka Connect Attunity Conversion

This project is a POC taking the JSON generated by Attunity and converting it to the proper Kafka connect data types. This is not production code and not a replacement for a proper Kafka Connect Connector for Attunity.

Current limitations

Limited types are supported.

All of the data that is coming from Attunity is currently formatted as strings and requires proper conversion to the strongly typed Kafka Connect data type. The only implemented types are below.

Type	Notes	Implementation
Decimal		Finished
String		Finished
Timestamp	Implementation needs the data from that comes from Attunity	Stubbed
Date	Implementation needs the data from that comes from Attunity	Stubbed

Here is an example of the data that is returned from Attunity.

{
  "task_name": "oracle2kafka",
  "table": "CUSTOMER",
  "schema": "NAVDEMO",
  "op": "F",
  "data": [
    {
      "C_CUSTKEY": "26"
    },
    {
      "C_NAME": "Carol Palmer"
    },
    {
      "C_ADDRESS": "26 Rodeo Drive"
    },
    {
      "C_NATIONKEY": "22"
    },
    {
      "C_PHONE": "32-363-455-4837"
    },
    {
      "C_ACCTBAL": "11371.0500000000"
    },
    {
      "C_MKTSEGMENT": "AUTOMOBILE"
    },
    {
      "C_COMMENT": "NULL"
    }
  ]
}

Schema has the be defined in configuration

The schema currently has to be defined in the configuration file. In the example below input.0 is the first input for the system. The schema and table settings are used to look at the kafka topic and select which record will get processed. These settings match the schema and table fields in the json above.

input.0.schema=NAVDEMO
input.0.table=CUSTOMER
input.0.kafka.topic=connect_navdemo_customer
input.0.keys=C_CUSTKEY
input.0.fields.C_CUSTKEY.type=decimal
input.0.fields.C_CUSTKEY.precision=10
input.0.fields.C_CUSTKEY.scale=0
input.0.fields.C_NAME.type=string
input.0.fields.C_ADDRESS.type=string
input.0.fields.C_NATIONKEY.type=decimal
input.0.fields.C_NATIONKEY.precision=10
input.0.fields.C_NATIONKEY.scale=0
input.0.fields.C_PHONE.type=string
input.0.fields.C_ACCTBAL.type=decimal
input.0.fields.C_ACCTBAL.precision=38
input.0.fields.C_ACCTBAL.scale=10
input.0.fields.C_MKTSEGMENT.type=string
input.0.fields.C_COMMENT.type=string

Name	Notes
schema	Schema name from the Attunity JSON.
table	Table name from the Attunity JSON.
kafka.topic	The output Kafka topic the connector will write the normalized data to.
keys	The primary keys for the table. These columns will be used for the primary key to Kafka.

Running in development

mvn clean package
export CLASSPATH="$(find target/ -type f -name '*.jar'| grep '\-package' | tr '\n' ':')"
$CONFLUENT_HOME/bin/connect-standalone $CONFLUENT_HOME/etc/schema-registry/connect-avro-standalone.properties config/MySourceConnector.properties

ptzagk / kafka-connect-attunity-conversion Goto Github PK