dev-ozan / go-dcp-couchbase

The Go implementation of the Couchbase to Couchbase with DCP.

Go Dcp Couchbase

Go Dcp Couchbase streams documents from Couchbase Database Change Protocol (DCP) and writes to Couchbase bucket in near real-time.

Features

• Less resource usage and higher throughput.
• Update multiple documents for a DCP event(see Example).
• Handling different DCP events such as expiration, deletion and mutation(see Example).
• Managing batch configurations such as maximum batch size, batch bytes, batch ticker durations.
• Scale up and down by custom membership algorithms(Couchbase, KubernetesHa, Kubernetes StatefulSet or Static, see examples).
• Easily manageable configurations.

Concepts

General Concept

Merge at target bucket

Example

Struct Config

package main

import (
	"github.com/Trendyol/go-dcp-couchbase"
	"time"

	"github.com/Trendyol/go-dcp-couchbase/config"
	dcpConfig "github.com/Trendyol/go-dcp/config"
	"github.com/Trendyol/go-dcp/logger"
)

func main() {
	c, err := dcpcouchbase.NewConnector(&config.Config{
		Dcp: dcpConfig.Dcp{
			Hosts:      []string{"localhost:8091"},
			Username:   "user",
			Password:   "password",
			BucketName: "dcp-test",
			Dcp: dcpConfig.ExternalDcp{
				Group: dcpConfig.DCPGroup{
					Name: "groupName",
					Membership: dcpConfig.DCPGroupMembership{
						RebalanceDelay: 3 * time.Second,
					},
				},
			},
			Metadata: dcpConfig.Metadata{
				Config: map[string]string{
					"bucket":     "dcp-test-meta",
					"scope":      "_default",
					"collection": "_default",
				},
				Type: "couchbase",
			},
			Debug: true,
		},
		Couchbase: config.Couchbase{
			Hosts:          []string{"localhost:8091"},
			Username:       "user",
			Password:       "password",
			BucketName:     "dcp-test-backup",
			BatchSizeLimit: 10,
			RequestTimeout: 10 * time.Second,
		},
	}, dcpcouchbase.DefaultMapper, logger.Log, logger.ErrorLog)
	if err != nil {
		panic(err)
	}

	defer c.Close()
	c.Start()
}

Configuration

Dcp Configuration

Check out on go-dcp

Couchbase Specific Configuration

Variable	Type	Required	Default	Description
Hosts	[]string	yes		Couchbase connection urls
BucketName	string	yes		Defines Couchbase bucket name
ScopeName	string	no	_default	Defines Couchbase scope name
CollectionName	string	no	_default	Defines Couchbase collection name
BatchSizeLimit	int	no	1000	Maximum message count for batch, if exceed flush will be triggered.
BatchTickerDuration	time.Duration	no	10s	Batch is being flushed automatically at specific time intervals for long waiting messages in batch.
BatchByteSizeLimit	int	no	10485760	Maximum size(byte) for batch, if exceed flush will be triggered.
RequestTimeout	int	no	3s	Maximum request waiting time. Value type milliseconds.

Exposed metrics

For DCP related metrics see also.

Contributing

Go Dcp Couchbase is always open for direct contributions. For more information please check our Contribution Guideline document.

License

Released under the MIT License.