mozilla / telemetry-analysis-service

Telemetry Analysis Service

atmo - The code for the Telemetry Analysis Service

The full documentation can be found on Read The Docs:

https://atmo.readthedocs.io/

telemetry-analysis-service's Issues

Timed out jobs should have a trace of it in the logs

See https://bugzilla.mozilla.org/show_bug.cgi?id=1316626

Automatically populate the "Further Reading:" section on cluster page

See https://bugzilla.mozilla.org/show_bug.cgi?id=1318842

Remove ability to specify env vars in .env files

It's a security risk

Add ability to extend cluster lifetime optionally

See https://bugzilla.mozilla.org/show_bug.cgi?id=1337809

Allow users to disable jobs for a time

See https://bugzilla.mozilla.org/show_bug.cgi?id=1310614

Add client side form validation

See https://bugzilla.mozilla.org/show_bug.cgi?id=1319077

Add EMR release 5.3.1 to atmo

See https://bugzilla.mozilla.org/show_bug.cgi?id=1336983

Handle gracefully the lack of spot instances

See the cluster at [0]. It spent >70 minutes provisioning. We need to add the ability to turn off spot instances, so that we can guarantee clusters to people.

Longer-term, we need to expand our spot instances. Different instance types, different regions, and more would allow us wider selection of machines and reduce this problem.

[0] https://us-west-2.console.aws.amazon.com/elasticmapreduce/home?region=us-west-2#cluster-details:j-1569LDWQXO5A7

Add an example of how to store data on S3 from Spark in the Hello World notebook

Originally reported in https://bugzilla.mozilla.org/show_bug.cgi?id=1253675

AttributeError: 'SparkJobProvisioner' object has no attribute 'stop'

While testing locally in a branch I bumped into this. I didn't test it on master but looking at the code this seems like it may also be a problem.

The SparkJob.terminate calls self.provisioner.stop(self.current_run_jobflow_id), where self.provisioner is a SparkJobProvisioner. But currently the SparkJobProvisioner has no stop method and neither does the parent class, it is only defined on the ClusterProvisioner. In my branch I changed the terminate method to call self.cluster_provisioner.stop instead and it fixed my issue locally.

Scheduled jobs should have an option to be run "right now"

https://bugzilla.mozilla.org/show_bug.cgi?id=1309567

Last run date is updated before the job runs to completion (or failure)

See https://bugzilla.mozilla.org/show_bug.cgi?id=1316042

"Go to today" button should select "today" in UTC time

See https://bugzilla.mozilla.org/show_bug.cgi?id=1322640

Add Zeppelin support to on-demand clusters.

See mozilla/emr-bootstrap-spark#59.

Create CLI

Refs #197 and https://bugzilla.mozilla.org/show_bug.cgi?id=1312424

Explain notebook flavors in UX

See https://bugzilla.mozilla.org/show_bug.cgi?id=1314677

Scheduled job notebook form entry should allow URI

See https://bugzilla.mozilla.org/show_bug.cgi?id=1338163

Failing jobs should generate alerts

See https://bugzilla.mozilla.org/show_bug.cgi?id=1312752

Improve security of telemetry

See mozilla/telemetry-dashboard#250

TemplateSyntaxError on atmo.dev.mozaws.net

I tried to go to the dev cluster to kill a cluster I spun up earlier and received the django debug error page. Looks like there's a hanging parens from the recent PR switching the template backend @jezdez

TemplateSyntaxError at /

Invalid block tag on line 6: 'url('clusters-new''. Did you forget to register or load this tag?

Request Method:     GET
Request URL:    https://atmo.dev.mozaws.net/
Django Version:     1.9.1
Exception Type:     TemplateSyntaxError
Exception Value:    

Invalid block tag on line 6: 'url('clusters-new''. Did you forget to register or load this tag?

Exception Location:     /usr/local/lib/python2.7/dist-packages/django/template/base.py in invalid_block_tag, line 568

Server time:    Thu, 22 Sep 2016 03:28:02 +0000

Template error:
In template /app/atmo/templates/atmo/new-cluster.html, error at line 6
   Invalid block tag on line 6: 'url('clusters-new''. Did you forget to register or load this tag?   1 : <button type="button" class="btn btn-md btn-primary" data-toggle="modal" data-target="#new-cluster-dialog" title="Launch Spark cluster">
   2 :   <span class="glyphicon glyphicon-play" aria-hidden="true"></span> Launch
   3 : </button>
   4 : <div class="modal fade" id="new-cluster-dialog" tabindex="-1" role="dialog" aria-labelledby="new-cluster-dialog">
   5 :   <div class="modal-dialog" role="document">
   6 :     <form class="modal-content" action=" {% url('clusters-new' %} " method="POST" enctype="multipart/form-data">
   7 :       {% csrf_token %}
   8 :       <div class="modal-header">
   9 :         <button type="button" class="close" data-dismiss="modal" aria-label="Close"><span aria-hidden="true">&times;</span></button>
   10 :         <h4 class="modal-title">Launch a Spark cluster</h4>
   11 :       </div>
   12 :       <div class="modal-body">
   13 :         <div class="form-group">
   14 :           <label>Cluster identifier</label>
   15 :           <div class="control-group">
   16 :             {{ new_cluster_form.identifier }}

Add other types of spot instances

@vitillo: Analysis jobs have different requirements in terms of hardware resources; some might benefit from more cores while others might benefit from more memory. Our users would like to select the instance type so that they can use machines that fit their job well.

Note that our Spark configuration and ETL jobs have been heavily tuned over time to run on c3.4xlarge instances. Introducing new instance types would require a significant amount of manual QA.

Add Zeppelin configuration step

See https://bugzilla.mozilla.org/show_bug.cgi?id=1337386

Create API

See https://bugzilla.mozilla.org/show_bug.cgi?id=1312423

Refactor EMR scripts

Quoting from https://bugzilla.mozilla.org/show_bug.cgi?id=1312747:

Currently airflow and atmo are using two different EMR steps [1] [2] for almost the same logic. We should refactor those into a single script and add that directly to the telemetry-analysis-service repository so that we can have different steps in different environments, like staging and production.
The bootstrap script [3] and the Spark configuration [4] should also be moved to telemetry-analysis-service.
[1] https://github.com/mozilla/emr-bootstrap-spark/blob/master/ansible/files/batch.sh
[2] https://github.com/mozilla/telemetry-airflow/blob/master/ansible/files/spark/airflow.sh
[3] https://github.com/mozilla/emr-bootstrap-spark/blob/master/ansible/files/telemetry.sh
[4] https://github.com/mozilla/emr-bootstrap-spark/blob/master/ansible/files/configuration.json

Add arguments for installing dependencies to Spark

See https://bugzilla.mozilla.org/show_bug.cgi?id=1315355

Use exponential back-off for DescribeCluster EMR call

See https://bugzilla.mozilla.org/show_bug.cgi?id=1306721

Please file issues in Bugzilla!

Here's a quick link to file a bug. Thanks!

(Issues aren't disabled to not lose the archive of already closed ones)

Should use --email arg to bootstrap script; have user directory mounted in EFS

See https://bugzilla.mozilla.org/show_bug.cgi?id=1335441

Better documentation for analysis output visibility

https://bugzilla.mozilla.org/show_bug.cgi?id=1309878

"Go to today" button would be more useful if it also selected the current time (+5 min), and not only the date

See https://bugzilla.mozilla.org/show_bug.cgi?id=1322641

Substitute sqlite with postgresql for heroku deployment

We want to deploy a stage instance of the analysis service on heroku to verify it's working properly. Due to the ephemeral nature of heroku's nodes, we need to move away from sqlite and use the heroku-provided postgresql service.

Fix log permissions

See https://bugzilla.mozilla.org/show_bug.cgi?id=1335228

Add auth credentials view for first time setup

The view would allow setting up the Google credentials via a view that we redirect to if no credentials are found in the database. That way we can simplify the setup procedure.

Store history of Spark job runs

Originally reported in https://bugzilla.mozilla.org/show_bug.cgi?id=1312426

is my job running? - overview visibility

for jobs that I have scheduled.... are they currently running? Did they ever run?

edit a job doesn't allow changing the ipynd

User story: Fixing bugs in the ipy notebook code.

Currently: 'edit job' doesn't allow changing the attached notebook. The way to edit it to make a new job.

Keep track of amazon emr/ec2 spot instances

Refs https://bugzilla.mozilla.org/show_bug.cgi?id=1312430

Billing e-mail for Spark and map-reduce jobs

Originally reported in https://bugzilla.mozilla.org/show_bug.cgi?id=1173429

Strip out Mozilla specific bits

See https://bugzilla.mozilla.org/show_bug.cgi?id=1312428

Results page of scheduled private job is updated before the job runs to completion or failure

See https://bugzilla.mozilla.org/show_bug.cgi?id=1316046

Logs of scheduled jobs should be listed in descending order

See https://bugzilla.mozilla.org/show_bug.cgi?id=1316048

Zeppelin should support R

See https://bugzilla.mozilla.org/show_bug.cgi?id=1337400

delete / edit buttons in user dash - not clear which job will be affected

Show logs in cluster detail page

See https://bugzilla.mozilla.org/show_bug.cgi?id=1310157

Display notification when cluster is ready

See https://bugzilla.mozilla.org/show_bug.cgi?id=1324464

Use Redis buildpack for secure connection

https://github.com/heroku/heroku-buildpack-redis

Add retry logic to scheduled jobs

Originally reported in https://bugzilla.mozilla.org/show_bug.cgi?id=1312427

Put LDAP login in an environment variable

See https://bugzilla.mozilla.org/show_bug.cgi?id=1333596

Store the list of available emr releases in a database table

See https://bugzilla.mozilla.org/show_bug.cgi?id=1335109

Add support for scheduling generic scripts

See https://bugzilla.mozilla.org/show_bug.cgi?id=1318665