etsy / sahale

A Cascading Workflow Visualizer

Sahale

A tool to record and visualize metrics captured from Cascading (Scalding) workflows at runtime.

Designed to target the pain points of analysts and end users of Cascading, Sahale provides insight into a workflow's runtime resource usage and makes job debugging and locating relevant Hadoop logs easy. The tool reveals optimization opportunities by exposing inefficient MapReduce jobs in a larger workflow, and enables users to track the execution history of their workflows.

Sahale has been verified to work with several Cascading DSLs, but the example projects and several features are tailored to Twitter's Scalding DSL, which is the DSL we use at Etsy. Pull requests for better support for other flavors of Cascading are welcome!

Installation

There are two components to Sahale, a flowtracker JAR that contains the code that tracks running jobs and a Node.JS dashboard for visualizing the tracked jobs.

Flowtracker

The flowtracker JAR is published to Maven Central for easy inclusion in your projects:

<dependency>
  <groupId>com.etsy.sahale</groupId>
  <artifactId>flowtracker_2.11</artifactId>
  <version>2.0.0</version>
</dependency>

or, for the Google-Auth-enabled variant:

<dependency>
  <groupId>com.etsy.sahale</groupId>
  <artifactId>flowtracker-gcp_2.11</artifactId>
  <version>2.0.0</version>
  <exclusions>
    <exclusion>
      <groupId>com.google.api-client</groupId>
      <artifactId>google-api-client</artifactId>
    </exclusion>
  </exclusions>
</dependency>

Note that we recommend excluding the transitive dependency google-api-client when running on Google Compute Engine / Dataproc, because the Dataproc clusters already have versions of google-api-client preinstalled. We have run into errors when including our own copy of google-api-client, despite our attempts to match the packaged version with the preinstalled version.

Configuration

You will need add some configuration for the flow tracker. Create the file src/main/resources/flow-tracker.properties in your project with the following contents:

host=http://host.where.nodejs.server.is.running
port=5735

The host setting will be the hostname of the server where you install the dashboard (see below)

You can also add the setting sahale.flow.selected.configs in this file. This is a comma-delimited list of Hadoop configuration properties that will be tracked along with the other job information.

If this configuration step is not performed job tracking will fail.

Dashboard

Prerequistes

1. A MySQL instance that Sahale can use.

2. A server with Node.JS and npm installed.

Installation

1. Set up the MySQL database. Follow the instructions in create_db_tables.sql to do this.

2. Clone the Sahale repo onto the server where you would like to run the dashboard.

3. Modify db-config.json to point to your database.

4. Run npm install from the root directory of the Sahale repo checkout.

Now you can run node app to start the Sahale dashboard. It will be running on port 5735, so go to <hostname>:5735 in your browser to see the dashboard.

Enabling Google Authentication

Sahale FlowTracker can be configured to send Google-Auth bearer tokens along with its requests to the Sahale server. This is used when the Sahale server is running in Google Cloud Platform, for example under Google App Engine with authentication enabled. To use this feature, include flowtracker-gcp in your Cascading project, and use the com.etsy.sahale.GoogleAuthFlowTracker class instead of the ordinary com.etsy.sahale.FlowTracker. GoogleAuthFlowTracker works on Hadoop clusters running in Google Compute Engine or Dataproc without any configuration, generating tokens using the application default service account credentials. For clusters not running in GCE/Dataproc, it also accepts a service account JSON file as an optiona argument.

Sahale server does not yet support Google authentication directly; our approach has been to run Sahale server behind an authenticating proxy server, for which there are several open source options.

Upgrading

Sahale server (NodeJS app) and FlowTracker (Scala client jar) must always maintain parity between clients and server versions. An upgrade deployment must coordinate the distribution of the new client jar and restart of the updated server. In rare cases (tagged in the Git repo) Sahale will make breaking changes that will require addition steps. The two notable cases are listed below:

Upgrade to 2.0.1

In addition to some minor fixes, we upgrade to scalding 0.17.4. We are alo publishing scala 2.11 and 2.12 artifacts for the first time. We will drop scala 2.11 support in a future version.

Upgrade to 1.2.0

We added a new FlowTracker module, flowtracker-gcp, which adds a new FlowTracker subclass, GoogleAuthFlowTracker. This subclass adds Authorization headers to its requests to Sahale server, passing OAuth bearer tokens generated from Google service accounts. This class is meant to be used with a Sahale server running in an environment employing service-to-service authentication via, for example, Google App Engine.

There were no data model or other incompatibility changes in this release.

Upgrade to 1.1.0

There were no data model or other incompatibility changes, but Sahale jar is now built with Scala 2.11 and published as flowtracker_2.11.

Upgrade to 1.0.0

There were no data model or other incompatibility changes, but the flowtracker JAR is now being published to Maven Central. It is no longer required to manually build this JAR; see the above instructions for its use and configuration.

Upgrade from 0.7 to 0.8

The data model has changed. Please recreate your backing MySQL tables using the script in src/main/sql before using the new client jar or restarting the NodeJS app. The old tables and data will remain, the new tables will be suffixed with _new unless you opt to alias them. No additional change is required.

Upgrade from 0.5 to 0.6

Mark incompatible changes between older and newer versions of Scala/Scalding. If your org still uses older versions of Scala/Scalding, please see this commit. All other changes and feature upgrades in the 0.6 line will work as expected with this commit reverted and your own choice of versions applied to the pom.xml.

The Name

Sahale was handmade at Etsy.com and is named for Sahale Mountain, which is a wonderful vantage point from which to view the Cascades

sahale's Issues

Remove CDH dependencies and update hadoop deps

Targeting apache Hadoop 2.7.0

If the FlowTracker can't initialize a connection to Sahale, it should not attempt to push final stats.

Correct Progress Bar with Map-Only Stages

When a job has map-only stages the progress bar can be incorrect (e.g. converging to 50% for a job with only map tasks).

Some failed steps report bad running time

We check at the end of each step to capture the full step running times. We need to reject tasks minStartTimes that erroneously report 0 (or anything less than full workflow start time) for this to avoid ending up with a UNIX timestamp instead of a task duration in seconds.

Shutdown hook leads to instability

We have found the shutdown hook causes instability and should be removed. These operations are too complex to happen at jvm shutdown.

Support pluggable username linking

Internally Etsy has a great staff directory that we link to from the displayed usernames in Sahale. Make this pluggable so others can take advantage of this easily.

Autoselect Stage for Single-Stage Jobs

If a job has only a single stage it should be automatically selected in the graph when the page loads.

Include glue code and example jobs targeting other Cascading DSLs

Like the pure Java API in Cascading, Cascalog, etc.

Pull requests welcome. I'll take a swipe at this myself sooner or later but its low priority as Etsy's piece of the Cascading landscape really focuses on Scalding.

Upgrade scalding to 0.17.X

Set up end-to-end tests?

Hi there 👋I didn't find any end-to-end tests in Sahale. I maintain an open source library that creates Playwright/Jest tests and runs them in CI.

Would you like me to set up end-to-end tests on this project? If you'd rather I add tests to a different project that's fine too!

Release Tags

Greetings! I have a small request: When you cut a Sahale release, would you kindly tag it and push the tag to github? Currently, I only see a v1.1.0 and a v0.5 tag.

I'm automating a deployment of the node application, and to get v2.0.1 I have to poke through the commit history to find the commit where that version was set in pom.xml and use that commit's hash as the git ref in config management. I'd much rather just have a tag.

Thanks for your consideration!

Consider providing multiple Maven profiles

To deal with major version differences in Scala and the various Hadoop frameworks used in Sahale/FlowTracker. Not sure we'll go this route yet, but take a look at various ways it could break down and have a good think on it.

Include configurable mapping from RM/JT URLs to Hadoop Cluster names for display

It would be nice to add properties to the config file on the Sahale side (the db-config.json) that maps expected flow.jt_url props coming from tracked jobs to shorthand cluster names so they could be displayed in the main Running/Completed jobs tables.

Add full Hadoop Counter support

In progress

Factor out browser state from graph view

In progress

Upgrade scalatest to 3.X

Pull Out Links to Error Logs

When a job fails, pull out the links to error logs and display this information as another tab with the job links and step running times.

It should also change that pane to default to the error link tab instead of the running times.

Changing Scala version

After changing scala version and submitting fat jar to cluster throws error,
NoClassDefFoundError: scala/reflect/ClassManifest

I think this happens because there is some classes still compiled with lower scala versions.

I think, ideal would be to have cross compiled jar in maven (or other) repositories.

What do you think?

I would be glad to help if there is anything I can do.
Best

Pull Out Real Input/Output Taps

Filter the list of taps to include only real (i.e. not in the temp directory) directories and display this information as another tab with the job links and step running times.

Sahale should report bytes to/from GCS if that information is reported to it

Move Existing Log Links to New "Links" Tab

There are some log links in the UI that can be moved to the new "Links" tab. Some kind of link/button to open up that tab can replace the existing "View Hadoop Logs" link.

Providing host as parameter

Instead of hard coding the monitoring server in flow-tracker.properties file, would it be good idea to provide it as parameter to cascading/scalding jobs?

Similar to --track-job also provide --host & --port?

This way re-assemblying fat jar won't be needed if host server changes.

Flowtracker NPE

java.lang.NullPointerException
	at com.etsy.sahale.FlowTracker.pushReport(FlowTracker.scala:271)
	at com.etsy.sahale.FlowTracker.pushFlowReport(FlowTracker.scala:244)
	at com.etsy.sahale.FlowTracker.updateFlow(FlowTracker.scala:157)
	at com.etsy.sahale.FlowTracker.run(FlowTracker.scala:137)
	at java.lang.Thread.run(Thread.java:745)