This tool wraps AWS DataSync to provide scriptable task creation and execution.
- DataSync Agent
- python 3.6+
# Install dependencies (preferably in a virtual environment)
pip3 install -r requirements.txt
# Execute run-task.py with the specified configuration.
python3 run-task.py --config-file config.py
# If source/destination locations change frequently and the task needs to run on a regular basis,
# create a cron job for run-task.py
0 23 * * * python3 run-task.py --config-file config.py
# Users may also call the task runner as a docker container
docker build -t run-task .
docker run \
-v $PWD/config.py:/config.py:ro \ # mount configuration file at /config.py
-v ~/.aws:/root/.aws:ro \ # mount .aws directory if needed
run-task -c /config.py # execute run-task with the provided configuration
A task has been created with fixed source and destination locations. When called with the configuration below, the task runner will copy yesterday's files (which are assumed to be in a folder called YYYY-MM-DD) to the destination. If the folder does not exist, it will not be copied to the destination.
from datetime import date, timedelta
yesterday = date.today() - timedelta(days=1)
folder = yesterday.strftime("%Y-%m-%d")
def configure_task():
return {
"TaskArn": "arn:aws:datasync:<region>:<account>:task/<task-id>",
"Includes": [{"FilterType": "SIMPLE_PATTERN", "Value": folder}],
}
No task or locations have been created yet. The task runner will:
- Create a source nfs location
- Create a destination s3 location
- Create and execute a task which syncs source files to the destination
from datetime import date, timedelta
from os import path
yesterday = date.today() - timedelta(days=1)
folder = yesterday.strftime("%Y-%m-%d")
def configure_task():
return {
"CloudWatchLogGroupArn": "arn:aws:logs:<region>:<account>:log-group:<log-group-name>:*",
"SourceLocation": {
"Type": "nfs",
"Config": {
{
"ServerHostname": "source-server-hostname",
"Subdirectory": f"/source/subdirectory",
"OnPremConfig": {
"AgentArns": [
"arn:aws:datasync:<region>:<account>:agent/<agent-id>"
]
},
}
},
},
"DestinationLocation": {
"Type": "s3",
"Config": {
"S3BucketArn": "arn:aws:s3:::<destination-bucket>",
"S3Config": {
"BucketAccessRoleArn": "arn:aws:iam::<account>:role/<access-role-name>"
},
"Subdirectory": f"/destination/subdirectory",
},
},
"Includes": [{"FilterType": "SIMPLE_PATTERN", "Value": folder}],
}
The task runner will execute a predefined task and invoke the following user-defined lifecycle hooks:
- Before task configuration
- Before task execution
- During task execution (at regular intervals)
- After task execution
Note: Hooks allow the user to inspect the current task and its execution status (enabling integration with other tools), and to cancel task creation or execution.
def configure_task():
return { "TaskArn": "arn:aws:datasync:<region>:<account>:task/<task-id>" }
def before_task_configuration():
""" Invoked before task configuration """
return True # False cancels task creation and execution
def before_task_execution(task_status):
""" Invoked before task execution """
print(task_status)
return True # False cancels task execution (eg: if we only wish to create/configure tasks and/or locations, or if execution conditions are not met)
def during_task_execution(task_status, task_execution_status):
""" Invoked during task execution at regular intervals """
print(task_status)
print(task_execution_status)
return True # False cancels task execution
def after_task_execution(task_status, task_execution_status):
""" Invoked after task execution has completed. """
print(task_status)
print(task_execution_status)
The run-task module exposes the following methods for interacting with DataSync.
- create_location - creates a DataSync location
- update_location - updates a DataSync location's metadata
- create_task - creates a DataSync task
- update_task - updates a DataSync task's metadata
- run_task - creates and/or executes a DataSync task given a configuration file