uejb / kafka-connect-solr

Kafka Connect connector for writing to Solr.

Introduction

Documentation

Confluent Hub

The SOLR connector is a high speed mechanism for writing data to Apache Solr.

Tip

If you are seeing error messages such as Invalid version (expected 2, but 60) or the data in not in 'javabin' format compare the version of the Solr Server against the version of solrj the connector is compiled with. This error message is most likely due to a version mismatch between the server and solrj. To address this try replacing the solr-solrj-*.jar packaged with the connector with the version that matches the Solr server you are connecting to.

Sink Connectors

Cloud Solr

This connector is used to connect to SolrCloud using the Zookeeper based configuration.

Tip

The target collection for this connector is selected by the topic name. Transformations like the RegexRouter transformation can be used to change the topic name before it is sent to Solr.

Configuration

Authentication

solr.password

The password to use for basic authentication.

Importance: High

Type: Password

Default Value: [hidden]

solr.username

The username to use for basic authentication.

Importance: High

Type: String

Connection

solr.zookeeper.hosts

Zookeeper hosts that are used to store solr configuration.

Importance: High

Type: List

solr.zookeeper.chroot

Chroot within solr for the zookeeper configuration.

Importance: High

Type: String

Indexing

solr.delete.documents.enabled

Flag to determine if the connector should delete documents. General practice in Kafka is to treat a record that contains a key with a null value as a delete.

Importance: Medium

Type: Boolean

Default Value: true

solr.commit.within

Configures Solr UpdaterRequest for a commit within the requested number of milliseconds. -1 disables the commit within setting and relies on the standard Solr commit setting.

Importance: Low

Type: Int

Default Value: -1

Examples

Drop Topic prefix

In this example each record has an incoming topic name prefixed with solr-. Assuming that our topic is solr-customer the following example will strip the prefix of solr- allowing us to write to the collection named customer. This is accomplished by using the RegexRouter transformation that is bundled with Apache Kafka.

Select one of the following configuration methods based on how you have deployed Kafka Connect. Distributed Mode will the the JSON / REST examples. Standalone mode will use the properties based example.

Distributed Mode Json

{
  "name" : "cloudSolrSinkConnector1",
  "config" : {
    "connector.class" : "com.github.jcustenborder.kafka.connect.solr.CloudSolrSinkConnector",
    "tasks.max" : "1",
    "topics" : "topic1,topic2,topic3",
    "solr.zookeeper.hosts" : "zookeeper.example.com:2181",
    "solr.username" : "freddy",
    "solr.password" : "password12345",
    "transforms" : "dropPrefix",
    "transforms.dropPrefix.type" : "org.apache.kafka.connect.transforms.RegexRouter",
    "transforms.dropPrefix.regex" : "solr-(.*)",
    "transforms.dropPrefix.replacement" : "$1"
  }
}

Standalone Mode Properties

connector.class=com.github.jcustenborder.kafka.connect.solr.CloudSolrSinkConnector
tasks.max=1
topics=topic1,topic2,topic3
solr.zookeeper.hosts=zookeeper.example.com:2181
solr.username=freddy
solr.password=password12345
transforms=dropPrefix
transforms.dropPrefix.type=org.apache.kafka.connect.transforms.RegexRouter
transforms.dropPrefix.regex=solr-(.*)
transforms.dropPrefix.replacement=$1

Standard

This example will connect to a Solr Cloud cluster without authentication.

Select one of the following configuration methods based on how you have deployed Kafka Connect. Distributed Mode will the the JSON / REST examples. Standalone mode will use the properties based example.

Distributed Mode Json

{
  "name" : "cloudSolrSinkConnector1",
  "config" : {
    "connector.class" : "com.github.jcustenborder.kafka.connect.solr.CloudSolrSinkConnector",
    "tasks.max" : "1",
    "topics" : "topic1,topic2,topic3",
    "solr.zookeeper.hosts" : "zookeeper.example.com:2181"
  }
}

Standalone Mode Properties

connector.class=com.github.jcustenborder.kafka.connect.solr.CloudSolrSinkConnector
tasks.max=1
topics=topic1,topic2,topic3
solr.zookeeper.hosts=zookeeper.example.com:2181

Basic Authentication

This example will connect to a Solr Cloud cluster using basic authentication.

Select one of the following configuration methods based on how you have deployed Kafka Connect. Distributed Mode will the the JSON / REST examples. Standalone mode will use the properties based example.

Distributed Mode Json

{
  "name" : "cloudSolrSinkConnector1",
  "config" : {
    "connector.class" : "com.github.jcustenborder.kafka.connect.solr.CloudSolrSinkConnector",
    "tasks.max" : "1",
    "topics" : "topic1,topic2,topic3",
    "solr.zookeeper.hosts" : "zookeeper.example.com:2181",
    "solr.username" : "freddy",
    "solr.password" : "password12345"
  }
}

Standalone Mode Properties

connector.class=com.github.jcustenborder.kafka.connect.solr.CloudSolrSinkConnector
tasks.max=1
topics=topic1,topic2,topic3
solr.zookeeper.hosts=zookeeper.example.com:2181
solr.username=freddy
solr.password=password12345

Force collection name

In this example we do not care about the incoming topic name. We want to force all topics to a specific collection. This is accomplished by using the RegexRouter transformation that is bundled with Apache Kafka.

Select one of the following configuration methods based on how you have deployed Kafka Connect. Distributed Mode will the the JSON / REST examples. Standalone mode will use the properties based example.

Distributed Mode Json

{
  "name" : "cloudSolrSinkConnector1",
  "config" : {
    "connector.class" : "com.github.jcustenborder.kafka.connect.solr.CloudSolrSinkConnector",
    "tasks.max" : "1",
    "topics" : "topic1,topic2,topic3",
    "solr.zookeeper.hosts" : "zookeeper.example.com:2181",
    "solr.username" : "freddy",
    "solr.password" : "password12345",
    "transforms" : "dropPrefix",
    "transforms.dropPrefix.type" : "org.apache.kafka.connect.transforms.RegexRouter",
    "transforms.dropPrefix.regex" : ".*",
    "transforms.dropPrefix.replacement" : "forced-collection"
  }
}

Standalone Mode Properties

connector.class=com.github.jcustenborder.kafka.connect.solr.CloudSolrSinkConnector
tasks.max=1
topics=topic1,topic2,topic3
solr.zookeeper.hosts=zookeeper.example.com:2181
solr.username=freddy
solr.password=password12345
transforms=dropPrefix
transforms.dropPrefix.type=org.apache.kafka.connect.transforms.RegexRouter
transforms.dropPrefix.regex=.*
transforms.dropPrefix.replacement=forced-collection

Standard Solr

This connector is used to connect to write directly to a Solr core.

Tip

The target collection for this connector is selected by the topic name. Transformations like the RegexRouter transformation can be used to change the topic name before it is sent to Solr.

Configuration

Authentication

solr.password

The password to use for basic authentication.

Importance: High

Type: Password

Default Value: [hidden]

solr.username

The username to use for basic authentication.

Importance: High

Type: String

Connection

solr.url

Url to connect to solr with.

Importance: High

Type: String

Indexing

solr.queue.size

The number of documents to batch together before sending to Solr. See ConcurrentUpdateSolrClient.Builder.withQueueSize(int)

Importance: High

Type: Int

Default Value: 100

solr.delete.documents.enabled

Flag to determine if the connector should delete documents. General practice in Kafka is to treat a record that contains a key with a null value as a delete.

Importance: Medium

Type: Boolean

Default Value: true

solr.thread.count

The number of threads used to empty ConcurrentUpdateSolrClients queue. See ConcurrentUpdateSolrClient.Builder.withThreadCount(int)

Importance: Medium

Type: Int

Default Value: 1

solr.commit.within

Configures Solr UpdaterRequest for a commit within the requested number of milliseconds. -1 disables the commit within setting and relies on the standard Solr commit setting.

Importance: Low

Type: Int

Default Value: -1

Examples

Standard

This example will connect to a Solr Cloud cluster without authentication.

Select one of the following configuration methods based on how you have deployed Kafka Connect. Distributed Mode will the the JSON / REST examples. Standalone mode will use the properties based example.

Distributed Mode Json

{
  "name" : "httpSolrSinkConnector1",
  "config" : {
    "connector.class" : "com.github.jcustenborder.kafka.connect.solr.HttpSolrSinkConnector",
    "tasks.max" : "1",
    "topics" : "topic1,topic2,topic3",
    "solr.url" : "http://solr.example.com:8993/"
  }
}

Standalone Mode Properties

connector.class=com.github.jcustenborder.kafka.connect.solr.HttpSolrSinkConnector
tasks.max=1
topics=topic1,topic2,topic3
solr.url=http://solr.example.com:8993/

Basic Authentication

This example will connect to a Solr Cloud cluster using basic authentication.

Select one of the following configuration methods based on how you have deployed Kafka Connect. Distributed Mode will the the JSON / REST examples. Standalone mode will use the properties based example.

Distributed Mode Json

{
  "name" : "httpSolrSinkConnector1",
  "config" : {
    "connector.class" : "com.github.jcustenborder.kafka.connect.solr.HttpSolrSinkConnector",
    "tasks.max" : "1",
    "topics" : "topic1,topic2,topic3",
    "solr.url" : "http://solr.example.com:8993/",
    "solr.username" : "freddy",
    "solr.password" : "password12345"
  }
}

Standalone Mode Properties

connector.class=com.github.jcustenborder.kafka.connect.solr.HttpSolrSinkConnector
tasks.max=1
topics=topic1,topic2,topic3
solr.url=http://solr.example.com:8993/
solr.username=freddy
solr.password=password12345