tsdrm / redis-connect-dist

Real-Time Event Streaming & Change Data Capture

Redis Connect 🚀 is a distributed platform that enables real-time event streaming, transformation, and propagation of changed-data events from heterogeneous data platforms to multi-model Redis Enterprise and Redis Stack.

✅ No Code ✅ Seamless Integration ✅ Multi-Tenancy ✅ Linear-Scalability ✅ High-Availability ✅ Support

Table of Contents

• Core Capabilities
• Requirements
• Quick Start
• End-to-End Dockerized Demos
• Video Tutorials

Core Capabilities

Cloud-Native Architecture Redis Connect has a cloud-native and shared-nothing architecture which allows any node to operate stand-alone or as a cluster member. Its platform-agnostic and lightweight design requires minimal infrastructure and avoids complex dependencies on 3rd-party platforms. All you need is Redis Enterprise.
	Multi-Tenancy | Partitioning | Linear Scalability Redis Connect can manage multi-tenant (jobs) data replication pipelines end-to-end within a single cluster node. Jobs support a variety of different source databases which can be collocated without becoming noisy neighbors. Streaming and initial load jobs can be partitioned for linear scalability across a single or multiple cluster nodes.
High-Availability | Recovery Redis Connect jobs update their checkpoint upon each committed changed-data event within a transactional scope. In the occurrence of node failure, or network split, a job would failover to another node and seamlessly begin replication from the last committed checkpoint. Data would not be lost, and order would be maintained. Redis Connect is supported on Kubernetes environments including OpenShift.
	Custom Transformations Redis Connect Jobs support user-defined business logic simply by adding a JAR to the /extlib directory. Users can create custom workflows that include user-defined stages for proprietary business rules, custom transformations, de-tokenization, and more. Users can also extend the supported list of Target Sinks.
REST API | CLI | Swagger UI Redis Connect is entirely data-driven and relies on Redis Enterprise as its metadata store. Users can configure, start, stop, migrate, and restart jobs via its built-in REST API and/or interactive CLI. Redis Connect also exposes a Swagger UI to simplify endpoint discovery and operational experience.
	Enterprise-Grade Security Redis Connect jobs are stateless so changed-data events are always in-transit. Redis Connect benefits from Redis Enterprise’s enterprise-grade security capabilities including RBAC, TLS, and more. Credentials, secrets, and trust-store passwords are never stored within Redis Connect however can be dynamically rotated with minimal disruption to the replication pipeline. Vault integration is supported.

Requirements

Minimum Production Hardware Requirements

• 1GB of RAM
• 4 CPU cores
• 20GB of disk space
• 1G Network

Runtime Requirements

• JRE 11+ e.g. Azul OpenJDK

Quick Start

Download

Download latest release for Linux or Windows and unarchive redis-connect-<version>.<build>.[tar.gz|zip] archive
Docker image can be found at DockerHub

Linux:

tar vxf <tarfile name>

Windows:

unzip <zipfile name>

The following subdirectories will be extracted under /redis-connect -
bin – Startup scripts
lib – Dependencies
config – Credentials property files, jobmanager.properties, and job-config (JSON) examples
extlib – Custom/External dependencies e.g. custom stage, source-database drivers, etc.

Getting Started

Review options by running Redis Connect startup script
Linux:

redis-connect/bin$ ./redisconnect.sh    
-------------------------------
Redis Connect startup script.
*******************************
Please ensure that the value of REDISCONNECT_JOB_MANAGER_CONFIG_PATH points to the correct jobmanager.properties in /home/viragtripathi/qa/vm/redis-connect/bin/redisconnect.conf before executing any of the options below
Check the value of redis.connection.url and credentials.file.path in jobmanager.properties e.g.
redis.connection.url=redis://redis-19836.c9.us-east-1-2.ec2.cloud.redislabs.com:19836
credentials.file.path=/var/secrets/redis
*******************************
Usage: [-h|cli|start]
options:
-h: Print this help message and exit.
cli: init Redis Connect CLI
start: init Redis Connect Instance (Cluster Member)
-------------------------------

Windows:

redis-connect\bin> redisconnect.bat

Prerequisite Configuration ❗
Update credentials.file.path and redis.connection.url within /config/jobmanager.properties Example - jobmanager.properties

Start Redis Connect Instance
Linux:

redis-connect/bin$ ./redisconnect.sh start

Windows:

redis-connect\bin> redisconnect.bat start

Open browser to access Swagger UI - http://localhost:8282/swagger-ui/index.html
For quick start, use 'cdc_job' as jobName

Create Job Configuration - /connect/api/vi/job/config/{jobName}
For quick start, use a sample job configuration: PostgreSQL, Oracle, SQL Server, MongoDB, MySQL, DB2 and VERTICA

Prerequisite Configuration ❗
Update credentialsFilePath, databaseURL, database.dbname, database.hostname, database.port, schemaAndTableName, and columns within sample job configuration for source and target, where applicable

Start Job - /connect/api/vi/job/transition/start/{jobName}/{jobType}
For quick start, use 'stream' as jobType

Confirm Job Claim - /connect/api/vi/jobs/claim/{jobStatus}
For quick start, use 'all' as jobStatus

Insert some records to the source and confirm they have arrived in Redis. Enjoy!

End-to-End Dockerized Demos

Video Tutorials