Drake
Drake is a simple-to-use, extensible, text-based data workflow tool that organizes command execution around data and its dependencies. Data processing steps are defined along with their inputs and outputs and Drake automatically resolves their dependencies and calculates:
- which commands to execute (based on file timestamps)
- in what order to execute the commands (based on dependencies)
Drake is similar to GNU Make, but designed especially for data workflow management. It has HDFS support, allows multiple inputs and outputs, and includes a host of features designed to help you bring sanity to your otherwise chaotic data processing workflows.
Drake walk-through
If you like screencasts, check out this Drake walk-through video recorded by Artem Boytsov, Drake's primary designer:
Installation
Drake has been tested under Linux, Mac OS X and Windows 8. We've not tested it on other operating systems.
Drake installs itself on the first run of the drake
shell script; there is no
separate install script. Follow these instructions to install drake manually:
- Make sure you have Java version 6 or later.
- Download the
drake
script from themaster
branch of this project. - Place the
drake
script on your$PATH
. (~/bin
is a good choice if it is on your path.) - Set it to be executable. (
chmod 755 ~/bin/drake
) - Run it (
drake
)
Homebrew
If you're on a Mac you can alternatively use Homebrew to install Drake:
brew install drake
Upgrade Drake
Starting with Drake version 1.0.0, once you have Drake installed you can easily upgrade your version of Drake by running drake --upgrade
. The latest version of Drake will be downloaded and installed for you.
Download or build the uberjar
You can build Drake from source or run from a prebuilt jar. Detailed instructions
Use Drake as a Clojure library
You can programmatically use Drake from your Clojure project by using Drake's Clojure front end. Your project.clj dependencies should include the latest Drake library, e.g.:
[factual/drake "1.0.1"]
Faster startup time
The JVM startup time can be a nuisance. To reduce startup time, we recommend using the way cool Drip. Please see the Drake with Drip wiki page.
Basic Usage
The wiki is the home for Drake's documentation, but here are simple notes on usage:
To build a specific target (and any out-of-date dependencies, if necessary):
$ drake mytarget
To build a target and everything that depends on it (a.k.a. "down-tree" mode):
$ drake ^mytarget
To build a specific target only, without any dependencies, up or down the tree:
$ drake =mytarget
To force build a target:
$ drake +mytarget
To force build a target and all its downtree dependencies:
$ drake +^mytarget
To force build the entire workflow:
$ drake +...
To exclude targets:
$ drake ... -sometarget -anothertarget
By default, Drake will look for ./Drakefile
. The simplest way to run your workflow is to name your workflow file Drakefile
, and make sure you're in the same directory. Then, simply:
$ drake
To specify the workflow file explicitly, use -w
or --workflow
. E.g.:
$ drake -w /myworkflow/my-workflow.drake
Use drake --help
for the full list of options.
Documentation, etc.
The wiki is the home for Drake's documentation.
A lot of work went into designing and specifying Drake. To prove it, here's the 60 page specification and user manual. It's stored in Google Docs, and we encourage everyone to use its superb commenting feature to provide feedback. Just select the text you want to comment on, and click Insert -> Comment (Ctrl + Alt + M on Windows, Cmd + Option + M on Mac). It can also be downloaded as a PDF.
There are annotated workflow examples in the demos directory.
There's a Google Group for Drake where you can ask questions. And if you found a bug or want to submit a feature request, go to Drake's GitHub issues page.
Visualize your workflow
See more detail
Asynchronous Execution of Steps
Please see the wiki page on async.
Plugins
Drake has a plugin mechanism, allowing developers to publish and use custom plugins that extend Drake. See the Plugin wiki page for details.
HDFS Compatibility
Drake provides HDFS support by allowing you to specify inputs and outputs like hdfs:/my/big_file.txt
.
If you plan to use Drake with HDFS, please see the wiki page on HDFS Compatibility.
Amazon S3 Compatibility
Thanks to Chris Howe, Drake now has basic compatibility with Amazon S3 by allowing you to specify
inputs and outputs like s3://bucket/path/to/object
.
If you plan to use Drake with S3, please see the wiki doc on S3 Compatibility.
Drake on the REPL
You can use Drake from your Clojure REPL, via drake.core/run-workflow
. Please see the Drake on the REPL wiki page for more details.
Stuff outside this repo
Thanks to Lars Yencken, we now have Vim syntax support for Drake:
Also thanks to Lars Yencken, utilities for making life easier in Python with Drake workflows.
Courtesy of @daguar, an alternative approach to installing Drake on Mac OS X.
Original blog post announcing Drake's open source release
An epic knock-down-drag-out set of threads on Hacker News discussing the design merits of Drake
License
Source Copyright © 2012-2015 Factual, Inc.
Distributed under the Eclipse Public License, the same as Clojure uses. See the file COPYING.