hadoop2databricks's Introduction

How to use the scripts in this repository?

param_file.csv is generated by running hdi_dump_metadata.py in On-prem environment
source_details.csv is generated by running getsourcedump.py in On-prem environment

# Create RDBMS Source

source /opt/infoworks/bin/env.sh;python create_rdbms_source.py --source_connection_file_path source_details.csv --source_creation_template templates/create_source_template.json --host_name localhost --host_port 2999 --auth_token <> --cluster_template default_template

# Add source to cluster template

python add_source_to_clustertemplate.py --source_name AR_Test_TD --cluster_template default_template

# Script to configure the sources

source /opt/infoworks/bin/env.sh;python source_migration_v2.py --configuration_json_path source_AR_Test_TD.json --source_name AR_Test_TD --source_type rdbms --host_name localhost --host_port 2999 --auth_token <> --cluster_template default_template

# Script to run the historical data migration

source /opt/infoworks/bin/env.sh;python run_migration.py --host <> --token <> --cluster_id 0718-041317-gilt53 --param_file param_file.csv

Requirements

The folder structure in ADLS Gen2 should match the target hdfs directory structure of the on-prem sources
Source schema and table name should be same in HDI and Databricks

Supported Features

Supports migration of timestamp-based incremental tables and batch_id based incremental tables
Supports reading of parquet and orc files
Partitioning is supported
Column renames are supported

Unsupported Features

Datatypes over-rides are not supported

hadoop2databricks's People

Contributors

Watchers

hadoop2databricks's Issues

Convert CSV, XML and JSON files to Delta Format

Currently, the scripts convert only the ORC/Parquet to the Delta format. Need ability to convert CSV, JSON and XML files to Delta which helps in migration of Nifi and Sqoop loads

Recommend Projects