tapdata Goto Github PK

1. Login tapdata platform

2. In the left navigation panel, click Connections

3. On the right side of the page, click Create

4. In the pop-up dialog, search and select MySQL

5. On the page that you are redirected to, follow the instructions below to fill in the connection information for MySQL

6. Click Test, make sure all test pass, then click Save

1. Create MySQL and MongoDB data source

2. In the left navigation panel, click Data Pipelines -> Data Replications

3. On the right side of the page, click Create

4. Drag and drop MySQL and MongoDB data sources onto the canvas

5. Drag a line from the MySQL data source to MongoDB

6. Configure the MySQL data source and select the data tables you want to synchronize

7. Click the Save button in the upper right corner, then click the Start button

8. Observe the indicators and events on the task page until data is in sync

1. Create MySQL and PostgreSQL data source

2. In the left navigation panel, click Data Pipelines -> Data Transformation

3. On the right side of the page, click Create

4. Drag and drop MySQL and PostgreSQL data sources onto the canvas

5. Drag a line from the MySQL data source to PostgreSQL

6. Click the plus sign on the connection line and select Field Rename

7. Click Field Rename node, change i_price to price, i_data to data in config form

8. Click the Save button in the upper right corner, then click the Start button

9. Observe the indicators and events on the task page until data is in sync

Materialized view is a special feature of tapdata, You can give full play to the characteristics of MongoDB document database and create the data model you need, try enjoy it !

In this example, I will make a view using 2 tables in MySQL: order and product, make product as a embedded document of order, here is the step:

1. Create MySQL and MongoDB data source

2. In the left navigation panel, click Data Pipelines -> Data Transformation

3. On the right side of the page, click Create

4. Click mysql data source in left up side, then drag and drop order table and product table onto the canvas

5. Drag and drop "Master-slave merge" node in left bottom side onto the canvas

6. Drag a line from the order table to Master-slave merge

7. Drag a line from the product table to Master-slave merge

8. Drag and drop MongoDB data source onto the canvas, and drag a line from the "Master-slave merge" node to MongoDB node

9. Click "Master-slave merge" node, then drag product table into order table in the right side in "Table Name"

10. Click "Master-slave merge" node, then click product table, config Data write model to "Match and Merge", Field write path to "product", Association Conditions to "order_id" => "order_id", then you can see Schema in bottom changed

11. Click MongoDB node, and config target table name as order_with_product, update condition field config as "order_id"

12. Click the Save button in the upper right corner, then click the Start button

13. Observe the indicators and events on the task page until data is in sync

14. Check collection order_with_product in MongoDB, and you will see the data model

Using the data verification feature, you can quickly check whether the synchronized data is consistent and accurate

1. In the left navigation panel, click Data Pipelines -> Data Validation

2. On the right side of the page, click Task Consistency Validation

3. Choose 1 task, and valid type choose "All Fields Validation", it means system will check all fields for all record

4. Click Save, then click Execute in the task list

5. Wait validation task finished, click Result in the task list, and check the validation result