Mirror Maker 2 [MM2] is an open-source tool which allows users to geo replicate streams from one region to another. Following use-cases can be met with MM2
- Active-Passive setup
- Active-Active setup
- Migrating from on-premise to OSS
- Multi cloud setup
This quick start will get you started with the Mirror Maker 2 on OCI.
-
Permission to
manage
the following types of resources in your Oracle Cloud Infrastructure tenancy:vcns
,internet-gateways
,route-tables
,network-security-groups
,subnets
,OSS
, andinstances
. -
Quota to create the following resources: 1 VCN, 1 subnets, 1 Internet Gateway, 1 NAT Gateway, 2 route rules, 2 OSS stream pools, 1 OSS Connect Harness and 1 compute instances.
If you don't have the required permissions and quota, contact your tenancy administrator. See Policy Reference, Service Limits, Compartment Quotas.
Now, you'll want a local copy of this repo. You can make that with the commands:
git clone https://github.com/oracle-quickstart/oci-oss-mirrormaker.git
cd oci-oss-mirrormaker
ls
-
Complete the prerequisites described here.
-
Create a
terraform.tfvars
file, and specify the following variables:
# Authentication
tenancy_ocid = "<tenancy_ocid>"
current_user_ocid = "<user_ocid>"
fingerprint = "<finger_print>"
private_key_path = "<pem_private_key_path>"
# Region
region = "<oci_region>"
targetRegion = "<target_region>"
# Compartment
compartment_ocid = "<compartment_ocid>"
generate_public_ssh_key = false
public_ssh_key = "<ssh-rsa AAAAB3NzaC .... rsa-key-20211034>"
private_ssh_key_path = "<path_to_pem_file>"
Run the following commands:
terraform init
terraform plan
terraform apply
When you no longer need the deployment, you can run this command to destroy the resources:
terraform destroy
Mirror Maker 2 makes use of the Kafka connect ecosystem to instantiate three connectors, which are:-
- MirrorSourceConnector
- MirrorCheckpointConnector
- MirrorHeartbeatConnector
We will be creating the following resources for this Quick start
- Stream pool in source and target regions
- Kafka Connect Harness in the target region
- VCN and instance in the public subnet of VCN.
- And finally we install and run MM2 on the instance.
NOTE: This setup is only for OSS capability demonstration purpose. In production one would use private endpoints for streams and regions would be connected by the DRG or a VPN/Fastconnect.
- OSS doesn’t support compacted topics yet. So creation of offsets, heartbeat and checkpointing topics by mirror maker will throw an exception. But when Auto Create Topics in enabled in the Stream Pool then at the time of publishing data these topics will be auto created although they will not be compacted.
- sync.group.offsets.enabled=true is not supported in OSS
- sync.topic.acls.enabled=true is not supported in OSS