isabella232 / rex-flink

automate Flink tasks with Rex

NAME

Rex::Flink - automate Flink tasks with Rex

VERSION

version v0.5.0

SYNOPSIS

# Rexfile
use Rex::Flink;

# CLI
$ rex Flink:setup
$ rex Flink:start
$ rex Flink:stop

DESCRIPTION

This module automates Flink tasks with Rex.

Please note this is a work still heavily in progress, the interface might be subject to change before version 1.0.0.

TASKS

Flink:setup

Sets Flink up from binary tarball. The installation directory can be controlled from CMDB via install_path. Default is ~/flink.

Supported options:

--hadoop

Hadoop version (default: 27)

--scala

Scala version (default: 2.11)

--version

Flink version (default: 1.2.0)

Flink:configure

Configures Flink from a file, which is treated as a standard Rex template. The name of the file to be deployed can be controlled from CMDB via config_file. Default is templates/flink-conf.yaml.

Flink:start

Starts a Flink service.

Supported options:

--service

Service name to start (default: jobmanager)

--mode

Execution mode to start the service with (default: local)

Flink:stop

Stops a Flink service.

Supported options:

--service

Service name to stop (default: jobmanager)

Flink:Pipeline:upload

Uploads a compiled pipeline to the remote machine from the local one. By default it uploads into ~/flink/current/pipelines.

Supported options:

--file

Local filename of the pipeline to upload.

Flink:Pipeline:run

Runs a pipeline on the remote machine.

Supported options:

--file

Remote filename of the pipeline to run. Mandatory, without any default value.

--class

Class with the pipeline entry point.

--parallelism

Parallelism for the pipeline.

--options

Any extra options to be passed to the pipeline.

DIAGNOSTICS

• No pipeline file specified

No pipeline file is specified to run with Flink:Pipeline:run task.

CONFIGURATION AND ENVIRONMENT

This module can use configuration specified under the flink node in the CMDB. Example YAML structure:

flink:
    config_file: templates/flink-conf.yaml
    hadoop: 27
    install_path: ~/flink
    scala: 2.11
    version: 1.2.0

This module does not use any environment variables. See also TESTING.

DEPENDENCIES

This module depends on the modules detailed below.

CPAN modules:

• Const::Fast

• Rex

INCOMPATIBILITIES

There are no known incompatibilities with other modules.

BUGS AND LIMITATIONS

There are no known bugs. Make sure they are reported.

TESTING

This module is capable to do testing with the help of the Rex::Test module inside a VM via libvirt.

A URL pointing to a VM image to be used for the testing should be specified via the REX_FLINK_TEST_IMAGE environment variable. For example:

# CLI
$ REX_FLINK_TEST_IMAGE='file:///var/lib/livirt/images/image.qcow2' rex Test:run
$ REX_FLINK_TEST_IMAGE='https://domain.tld/image.qcow2' rex Test:run

AUTHOR

Ferenc Erki <[email protected]>

COPYRIGHT AND LICENSE

This software is copyright (c) 2017 by adjust GmbH.

This is free software; you can redistribute it and/or modify it under the same terms as the Perl 5 programming language system itself.