Hybrid Analytical / Transactional Processing

Replication Proof Of Concept

Instructions

This project will build a two node cluster. One MariaDB/ColumnStore server and one MaxScale server. However, before you can begin, please go to the MariaDB website and get your enterprise token. You will need it to run this project.

https://customers.mariadb.com/downloads/token/

Prerequisites

Setup

git clone https://github.com/mariadb-corporation/mariadb-columnstore-htap.git
cd htap_poc
vagrant plugin install vagrant-vbguest
vagrant up
ansible-playbook provision.yml -e "mariadb_token=<YOUR_TOKEN_HERE>"

When provisioning is complete you should have two test schemas (innodb_schema and columnstore_schema) each containing a table called orders.

There will also be an orders.csv file located in the /tmp/ folder on the MariaDB node.

Accessing Virtual Machines Directly:

For MariaDB: vagrant ssh node1
For MaxScale: vagrant ssh node2

Connecting To The Database From External Client:

Host: 10.10.10.11
User: dba
Password: Demo_password1
Port: 3306

Testing Replication:

MariaDB [(none)]> LOAD DATA INFILE '/tmp/orders.csv' INTO TABLE innodb_schema.orders
FIELDS TERMINATED BY ',' OPTIONALLY ENCLOSED BY '"'
LINES TERMINATED BY '\n';

The data should be immediately replicated from innodb_schema.orders to columnstore_schema.orders.

Notes

The following items are already included in the playbook, they are just listed here for reference:

Additional my.cnf settings:

Replication Filters

For this version of HTAP, we will rely on the functionality of MaxScale's Binlog Filter in order to match specific table names and rewrite source to target schema.

InnoDB transactions are written to the MariaDB binary log.
MariaDB server becomes a slave of itself and connects through MaxScale.
MaxScale matches chosen tables, rewrites schema names, and sends the filtered information back to MariaDB for use in ColumnStore.

Sample Configuration:

[myreplicationfilter]
type=filter
module=binlogfilter
match=/[.]orders/
rewrite_src=innodb
rewrite_dest=columnstore

Note: This example uses a basic regex matching for demonstration purposes only. To learn about more complex regex matching please visit Regex 101.

Alter Replicated Tables

Changing replicated tables is done dynamically through the MaxScale filters and the REST-API.

Create A Filter

To create a filter, you would use the POST command with a request body something like this:

{
   "data":{
      "id":"foo",
      "type":"filters",
      "attributes":{
         "module":"binlogfilter",
         "parameters":{
            "match":"orders",
            "exclude":"test_orders"
         }
      }
   }
}

This can be done in one line or with a text file called 'body.txt' for example:

curl -X POST -d '{"data":{"id":"foo","type":"filters","attributes":{"module":"binlogfilter","parameters":{"match":"orders","exclude":"test_orders"}}}}' admin:mariadb@localhost:8989/v1/filters

curl -X POST -d @body.txt admin:mariadb@localhost:8989/v1/filters

Get Filter Info

curl -X GET admin:mariadb@localhost:8989/v1/filters/foo

Delete A Filter

curl -X DELETE admin:mariadb@localhost:8989/v1/filters/foo

isabella232 / mariadb-columnstore-htap Goto Github PK

mariadb-columnstore-htap's Introduction