azkaban / azkaban

Azkaban workflow manager.

Azkaban

Build

Azkaban builds use Gradle and requires Java 8 or higher.

The following set of commands run on *nix platforms like Linux, OS X.

# Build Azkaban
./gradlew build

# Clean the build
./gradlew clean

# Build and install distributions
./gradlew installDist

# Run tests
./gradlew test

# Build without running tests
./gradlew build -x test

Build a release

Pick a release from the release page. Find the tag corresponding to the release.

Check out the source code corresponding to that tag. e.g.

git checkout 3.30.1

Build

./gradlew clean build

Documentation

The current documentation will be deprecated soon at azkaban.github.io. The new Documentation site is under development. The source code for the documentation is inside docs directory.

For help, please visit the Azkaban Google Group.

Developer Guide

See the contribution guide.

Documentation development

If you want to contribute to the documentation or the release tool (inside the tools folder), please make sure python3 is installed in your environment. python virtual environment is recommended to run these scripts.

To create a venv & install the python3 dependencies inside it, run

python3 -m venv venv
source venv/bin/activate
pip3 install -r requirements.txt

After, enter the documentation folder docs and make the build by running

cd docs
make html

Find the built docs under _build/html/.

For example on a Mac, open them in browser with:

open -a "Google Chrome" _build/html/index.html

[July, 2018] We are actively improving our documentation. Everyone in the AZ community is welcome to submit a pull request to edit/fix the documentation.

azkaban's Issues

About jetty ssh

how disable jetty ssh ?

[Feature Request] Ability to run overrides to scheduled executions

I have a large flow that we use both for scheduled and manual executions of the flow. We make heavy use of the Flow Parameter override for manual executions, however it would be nice if we could set the same up for the scheduled flow.

The example use case is the priority level of hadoop jobs. By default and for all manual executions we want the hadoop tasks to run at NORMAL priority. However for the scheduled flows we want them to be a higher priority than any manual flow.

orphan processes pile up on Azkaban server machine

Happened in LinkedIn's busiest clusters that, after 6+ months and a ton of different version changes, Azkaban machine accumulated quite a number orphan processes.

[Feature Request] Download Log Files

Web-UI should allow the user to download the logs of jobs.

The mime type of zip file may be application/x-zip-compressed

--- src/java/azkaban/webapp/servlet/ProjectManagerServlet.java  (revision 164)
+++ src/java/azkaban/webapp/servlet/ProjectManagerServlet.java  (working copy)
@@ -1068,12 +1068,13 @@
                        String type = null;
 
                        final String contentType = item.getContentType();
-                       if(contentType.startsWith("application/zip")) {
+                       if(contentType.startsWith("application/zip") ||  contentType.startsWith("application/x-zip-compressed")) {
                                type = "zip";
                        }
                        else {
                                item.delete();
                                setErrorMessageInCookie(resp, "File type " + contentType + " unrecognized.");
+                               return ;
                        }
 
                        File tempDir = Utils.createTempDir();

API Problem: Executing flow

The 'execute flow' API gives no error, but does not start a flow. 'job1' is a job in the project, and I can start it by hand. The API does not start it.

[my machine]$ curl -D headers -k --data 'action=executeFlow&session.id=2ead5140-2668-4891-be58-d84e6c00827a&project=api_test&flow=job1' https://localhost:8443
[my machine]$ cat headers
HTTP/1.1 200 OK
Content-Length: 0
Server: Jetty(6.1.26)

Deadlock in Azkaban 2.1 Web Server

When accessing the Azkaban web server (after it successfully scheduled/executed a job), the web server deadlocked. The JVM stack trace (after sending SIGQUIT) is:

Full thread dump OpenJDK 64-Bit Server VM (23.7-b01 mixed mode):

"Thread-10" prio=10 tid=0x00007f95d4001800 nid=0x2738 waiting on condition [0x00007f9680228000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.mortbay.thread.QueuedThreadPool.doStop(QueuedThreadPool.java:435)
at org.mortbay.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:76)
- locked <0x00000007e6036748> (a java.lang.Object)
at org.mortbay.jetty.Server.doStop(Server.java:291)
at org.mortbay.component.AbstractLifeCycle.stop(AbstractLifeCycle.java:76)
- locked <0x00000007e5590298> (a java.lang.Object)
at azkaban.webapp.AzkabanWebServer$1.run(AzkabanWebServer.java:465)

"SIGTERM handler" daemon prio=10 tid=0x00007f9654001000 nid=0x2737 in Object.wait() [0x00007f968042a000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Thread.join(Thread.java:1258)
- locked <0x00000007556ebeb8> (a azkaban.webapp.AzkabanWebServer$1)
at java.lang.Thread.join(Thread.java:1332)
at java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:106)
at java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:46)
at java.lang.Shutdown.runHooks(Shutdown.java:123)
at java.lang.Shutdown.sequence(Shutdown.java:167)
at java.lang.Shutdown.exit(Shutdown.java:212)
- locked <0x00000007556ec008> (a java.lang.Class for java.lang.Shutdown)
at java.lang.Terminator$1.handle(Terminator.java:52)
at sun.misc.Signal$1.run(Signal.java:212)
at java.lang.Thread.run(Thread.java:722)

"2102544801@qtp-1369161839-45" prio=10 tid=0x00007f9618033000 nid=0x2546 waiting for monitor entry [0x00007f9680b30000]
java.lang.Thread.State: BLOCKED (on object monitor)
at azkaban.scheduler.ScheduleManager.getSchedules(ScheduleManager.java:120)
- waiting to lock <0x00000007e60ec4f8> (a azkaban.scheduler.ScheduleManager)
at azkaban.webapp.servlet.ScheduleServlet.handleGetAllSchedules(ScheduleServlet.java:292)
at azkaban.webapp.servlet.ScheduleServlet.handleGet(ScheduleServlet.java:80)
at azkaban.webapp.servlet.LoginAbstractAzkabanServlet.doGet(LoginAbstractAzkabanServlet.java:77)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:713)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

"162434782@qtp-1369161839-44" prio=10 tid=0x00007f9618032800 nid=0x2544 waiting for monitor entry [0x00007f968062b000]
java.lang.Thread.State: BLOCKED (on object monitor)
at azkaban.scheduler.ScheduleManager.getSchedules(ScheduleManager.java:120)
- waiting to lock <0x00000007e60ec4f8> (a azkaban.scheduler.ScheduleManager)
at azkaban.webapp.servlet.ScheduleServlet.handleGetAllSchedules(ScheduleServlet.java:292)
at azkaban.webapp.servlet.ScheduleServlet.handleGet(ScheduleServlet.java:80)
at azkaban.webapp.servlet.LoginAbstractAzkabanServlet.doGet(LoginAbstractAzkabanServlet.java:77)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:713)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

"1981941886@qtp-1369161839-43" prio=10 tid=0x00007f9618057000 nid=0x24cf waiting for monitor entry [0x00007f9680d32000]
java.lang.Thread.State: BLOCKED (on object monitor)
at azkaban.scheduler.ScheduleManager.getSchedules(ScheduleManager.java:120)
- waiting to lock <0x00000007e60ec4f8> (a azkaban.scheduler.ScheduleManager)
at azkaban.webapp.servlet.ScheduleServlet.handleGetAllSchedules(ScheduleServlet.java:292)
at azkaban.webapp.servlet.ScheduleServlet.handleGet(ScheduleServlet.java:80)
at azkaban.webapp.servlet.LoginAbstractAzkabanServlet.doGet(LoginAbstractAzkabanServlet.java:77)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:713)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

"1662427104@qtp-1369161839-42" prio=10 tid=0x00007f9618056000 nid=0x24c6 waiting for monitor entry [0x00007f968092e000]
java.lang.Thread.State: BLOCKED (on object monitor)
at azkaban.scheduler.ScheduleManager.getSchedules(ScheduleManager.java:120)
- waiting to lock <0x00000007e60ec4f8> (a azkaban.scheduler.ScheduleManager)
at azkaban.webapp.servlet.ScheduleServlet.handleGetAllSchedules(ScheduleServlet.java:292)
at azkaban.webapp.servlet.ScheduleServlet.handleGet(ScheduleServlet.java:80)
at azkaban.webapp.servlet.LoginAbstractAzkabanServlet.doGet(LoginAbstractAzkabanServlet.java:77)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:713)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

"1685440007@qtp-1369161839-41" prio=10 tid=0x00007f9618021000 nid=0x239b waiting for monitor entry [0x00007f9680e33000]
java.lang.Thread.State: BLOCKED (on object monitor)
at azkaban.scheduler.ScheduleManager.getSchedules(ScheduleManager.java:120)
- waiting to lock <0x00000007e60ec4f8> (a azkaban.scheduler.ScheduleManager)
at azkaban.webapp.servlet.ScheduleServlet.handleGetAllSchedules(ScheduleServlet.java:292)
at azkaban.webapp.servlet.ScheduleServlet.handleGet(ScheduleServlet.java:80)
at azkaban.webapp.servlet.LoginAbstractAzkabanServlet.doGet(LoginAbstractAzkabanServlet.java:77)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:713)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

"313138637@qtp-1369161839-40" prio=10 tid=0x00007f9618023000 nid=0x2398 waiting for monitor entry [0x00007f9680a2f000]
java.lang.Thread.State: BLOCKED (on object monitor)
at azkaban.scheduler.ScheduleManager.getSchedules(ScheduleManager.java:120)
- waiting to lock <0x00000007e60ec4f8> (a azkaban.scheduler.ScheduleManager)
at azkaban.webapp.servlet.ScheduleServlet.handleGetAllSchedules(ScheduleServlet.java:292)
at azkaban.webapp.servlet.ScheduleServlet.handleGet(ScheduleServlet.java:80)
at azkaban.webapp.servlet.LoginAbstractAzkabanServlet.doGet(LoginAbstractAzkabanServlet.java:77)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:713)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

"1891074395@qtp-1369161839-39" prio=10 tid=0x00007f9618053800 nid=0x2396 waiting for monitor entry [0x00007f968082d000]
java.lang.Thread.State: BLOCKED (on object monitor)
at azkaban.scheduler.ScheduleManager.getSchedules(ScheduleManager.java:120)
- waiting to lock <0x00000007e60ec4f8> (a azkaban.scheduler.ScheduleManager)
at azkaban.webapp.servlet.ScheduleServlet.handleGetAllSchedules(ScheduleServlet.java:292)
at azkaban.webapp.servlet.ScheduleServlet.handleGet(ScheduleServlet.java:80)
at azkaban.webapp.servlet.LoginAbstractAzkabanServlet.doGet(LoginAbstractAzkabanServlet.java:77)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:713)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

"1961647566@qtp-1369161839-36" prio=10 tid=0x00007f9618052800 nid=0x1ac3 waiting for monitor entry [0x00007f9682ed3000]
java.lang.Thread.State: BLOCKED (on object monitor)
at azkaban.scheduler.ScheduleManager.getSchedules(ScheduleManager.java:120)
- waiting to lock <0x00000007e60ec4f8> (a azkaban.scheduler.ScheduleManager)
at azkaban.webapp.servlet.ScheduleServlet.handleGetAllSchedules(ScheduleServlet.java:292)
at azkaban.webapp.servlet.ScheduleServlet.handleGet(ScheduleServlet.java:80)
at azkaban.webapp.servlet.LoginAbstractAzkabanServlet.doGet(LoginAbstractAzkabanServlet.java:77)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:713)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

"1896617363@qtp-1369161839-35" prio=10 tid=0x00007f9618051800 nid=0x1ac2 waiting for monitor entry [0x00007f9680328000]
java.lang.Thread.State: BLOCKED (on object monitor)
at azkaban.scheduler.ScheduleManager.getSchedules(ScheduleManager.java:120)
- waiting to lock <0x00000007e60ec4f8> (a azkaban.scheduler.ScheduleManager)
at azkaban.webapp.servlet.ScheduleServlet.handleGetAllSchedules(ScheduleServlet.java:292)
at azkaban.webapp.servlet.ScheduleServlet.handleGet(ScheduleServlet.java:80)
at azkaban.webapp.servlet.LoginAbstractAzkabanServlet.doGet(LoginAbstractAzkabanServlet.java:77)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:713)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

"317606521@qtp-1369161839-33" prio=10 tid=0x00007f9618027800 nid=0x1ac0 waiting for monitor entry [0x00007f968072c000]
java.lang.Thread.State: BLOCKED (on object monitor)
at azkaban.scheduler.ScheduleManager.getSchedules(ScheduleManager.java:120)
- waiting to lock <0x00000007e60ec4f8> (a azkaban.scheduler.ScheduleManager)
at azkaban.webapp.servlet.ScheduleServlet.handleGetAllSchedules(ScheduleServlet.java:292)
at azkaban.webapp.servlet.ScheduleServlet.handleGet(ScheduleServlet.java:80)
at azkaban.webapp.servlet.LoginAbstractAzkabanServlet.doGet(LoginAbstractAzkabanServlet.java:77)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:713)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

"262131240@qtp-1369161839-30" prio=10 tid=0x00007f9618019000 nid=0x1ab8 waiting for monitor entry [0x00007f9680c31000]
java.lang.Thread.State: BLOCKED (on object monitor)
at azkaban.scheduler.ScheduleManager.getSchedules(ScheduleManager.java:120)
- waiting to lock <0x00000007e60ec4f8> (a azkaban.scheduler.ScheduleManager)
at azkaban.webapp.servlet.ScheduleServlet.handleGetAllSchedules(ScheduleServlet.java:292)
at azkaban.webapp.servlet.ScheduleServlet.handleGet(ScheduleServlet.java:80)
at azkaban.webapp.servlet.LoginAbstractAzkabanServlet.doGet(LoginAbstractAzkabanServlet.java:77)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:713)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

"32453141@qtp-1369161839-29" prio=10 tid=0x00007f9618025000 nid=0xb7d waiting for monitor entry [0x00007f968052a000]
java.lang.Thread.State: BLOCKED (on object monitor)
at azkaban.scheduler.ScheduleManager.getSchedules(ScheduleManager.java:120)
- waiting to lock <0x00000007e60ec4f8> (a azkaban.scheduler.ScheduleManager)
at azkaban.webapp.servlet.ScheduleServlet.handleGetAllSchedules(ScheduleServlet.java:292)
at azkaban.webapp.servlet.ScheduleServlet.handleGet(ScheduleServlet.java:80)
at azkaban.webapp.servlet.LoginAbstractAzkabanServlet.doGet(LoginAbstractAzkabanServlet.java:77)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:713)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

"52225937@qtp-1369161839-27" prio=10 tid=0x00007f9618055000 nid=0xb76 waiting for monitor entry [0x00007f96831d6000]
java.lang.Thread.State: BLOCKED (on object monitor)
at azkaban.scheduler.ScheduleManager.getSchedules(ScheduleManager.java:120)
- waiting to lock <0x00000007e60ec4f8> (a azkaban.scheduler.ScheduleManager)
at azkaban.webapp.servlet.ScheduleServlet.handleGetAllSchedules(ScheduleServlet.java:292)
at azkaban.webapp.servlet.ScheduleServlet.handleGet(ScheduleServlet.java:80)
at azkaban.webapp.servlet.LoginAbstractAzkabanServlet.doGet(LoginAbstractAzkabanServlet.java:77)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:713)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

"1057409309@qtp-1369161839-3" prio=10 tid=0x00007f961802a800 nid=0xdbf2 waiting for monitor entry [0x00007f9680f34000]
java.lang.Thread.State: BLOCKED (on object monitor)
at azkaban.scheduler.ScheduleManager$ScheduleRunner.getRunnerSchedules(ScheduleManager.java:292)
- waiting to lock <0x00000007e60f5038> (a azkaban.scheduler.ScheduleManager$ScheduleRunner)
at azkaban.scheduler.ScheduleManager.getSchedules(ScheduleManager.java:120)
- locked <0x00000007e60ec4f8> (a azkaban.scheduler.ScheduleManager)
at azkaban.webapp.servlet.ScheduleServlet.handleGetAllSchedules(ScheduleServlet.java:292)
at azkaban.webapp.servlet.ScheduleServlet.handleGet(ScheduleServlet.java:80)
at azkaban.webapp.servlet.LoginAbstractAzkabanServlet.doGet(LoginAbstractAzkabanServlet.java:77)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:713)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

"DestroyJavaVM" prio=10 tid=0x00007f970800a800 nid=0xdba9 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"Thread-6" prio=10 tid=0x00007f9708738000 nid=0xdbe1 waiting for monitor entry [0x00007f96832d8000]
java.lang.Thread.State: BLOCKED (on object monitor)
at azkaban.scheduler.ScheduleManager.removeSchedule(ScheduleManager.java:139)
- waiting to lock <0x00000007e60ec4f8> (a azkaban.scheduler.ScheduleManager)
at azkaban.scheduler.ScheduleManager$ScheduleRunner.run(ScheduleManager.java:444)
- locked <0x00000007e60f5038> (a azkaban.scheduler.ScheduleManager$ScheduleRunner)

"AzkabanWebServer-Cleaner-Thread" prio=10 tid=0x00007f9708762800 nid=0xdbdb in Object.wait() [0x00007f96834da000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007e6125d60> (a azkaban.executor.ExecutorManager$CleanerThread)
at azkaban.executor.ExecutorManager$CleanerThread.run(ExecutorManager.java:966)
- locked <0x00000007e6125d60> (a azkaban.executor.ExecutorManager$CleanerThread)

"Abandoned connection cleanup thread" daemon prio=10 tid=0x00007f9708715000 nid=0xdbd5 in Object.wait() [0x00007f96836dc000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
- locked <0x00000007e6150518> (a java.lang.ref.ReferenceQueue$Lock)
at com.mysql.jdbc.AbandonedConnectionCleanupThread.run(AbandonedConnectionCleanupThread.java:40)

"net.sf.ehcache.CacheManager@71ac4afb" daemon prio=10 tid=0x00007f97085e1000 nid=0xdbd4 in Object.wait() [0x00007f96837dd000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000007e5575470> (a java.util.TaskQueue)
at java.util.TimerThread.mainLoop(Timer.java:552)
- locked <0x00000007e5575470> (a java.util.TaskQueue)
at java.util.TimerThread.run(Timer.java:505)

"RMI TCP Accept-0" daemon prio=10 tid=0x00007f97083cf000 nid=0xdbd2 runnable [0x00007f96839df000]
java.lang.Thread.State: RUNNABLE
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:522)
at java.net.ServerSocket.accept(ServerSocket.java:490)
at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:52)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.executeAcceptLoop(TCPTransport.java:387)
at sun.rmi.transport.tcp.TCPTransport$AcceptLoop.run(TCPTransport.java:359)
at java.lang.Thread.run(Thread.java:722)

"Service Thread" daemon prio=10 tid=0x00007f970814e000 nid=0xdbd1 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"C2 CompilerThread1" daemon prio=10 tid=0x00007f970814c000 nid=0xdbd0 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"C2 CompilerThread0" daemon prio=10 tid=0x00007f9708149000 nid=0xdbcf waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"Signal Dispatcher" daemon prio=10 tid=0x00007f9708147000 nid=0xdbce waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE

"Finalizer" daemon prio=10 tid=0x00007f97080f0800 nid=0xdbcd in Object.wait() [0x00007f9689fde000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:135)
- locked <0x00000007e60e0c30> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:151)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:189)

"Reference Handler" daemon prio=10 tid=0x00007f97080ee000 nid=0xdbcc in Object.wait() [0x00007f968a0df000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:503)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:133)
- locked <0x00000007e6030298> (a java.lang.ref.Reference$Lock)

"VM Thread" prio=10 tid=0x00007f97080e6000 nid=0xdbcb runnable

"GC task thread#0 (ParallelGC)" prio=10 tid=0x00007f9708018800 nid=0xdbaa runnable

"GC task thread#1 (ParallelGC)" prio=10 tid=0x00007f970801a800 nid=0xdbab runnable

"GC task thread#2 (ParallelGC)" prio=10 tid=0x00007f970801c000 nid=0xdbac runnable

"GC task thread#3 (ParallelGC)" prio=10 tid=0x00007f970801e000 nid=0xdbad runnable

"GC task thread#4 (ParallelGC)" prio=10 tid=0x00007f9708020000 nid=0xdbae runnable

"GC task thread#5 (ParallelGC)" prio=10 tid=0x00007f9708021800 nid=0xdbaf runnable

"GC task thread#6 (ParallelGC)" prio=10 tid=0x00007f9708023800 nid=0xdbb0 runnable

"GC task thread#7 (ParallelGC)" prio=10 tid=0x00007f9708025800 nid=0xdbb1 runnable

"GC task thread#8 (ParallelGC)" prio=10 tid=0x00007f9708027000 nid=0xdbb2 runnable

"GC task thread#9 (ParallelGC)" prio=10 tid=0x00007f9708029000 nid=0xdbb3 runnable

"GC task thread#10 (ParallelGC)" prio=10 tid=0x00007f970802b000 nid=0xdbb4 runnable

"GC task thread#11 (ParallelGC)" prio=10 tid=0x00007f970802c800 nid=0xdbb5 runnable

"GC task thread#12 (ParallelGC)" prio=10 tid=0x00007f970802e800 nid=0xdbb6 runnable

"GC task thread#13 (ParallelGC)" prio=10 tid=0x00007f9708030800 nid=0xdbb7 runnable

"GC task thread#14 (ParallelGC)" prio=10 tid=0x00007f9708032000 nid=0xdbb8 runnable

"GC task thread#15 (ParallelGC)" prio=10 tid=0x00007f9708034000 nid=0xdbb9 runnable

"GC task thread#16 (ParallelGC)" prio=10 tid=0x00007f9708036000 nid=0xdbba runnable

"GC task thread#17 (ParallelGC)" prio=10 tid=0x00007f9708038000 nid=0xdbbb runnable

"GC task thread#18 (ParallelGC)" prio=10 tid=0x00007f9708039800 nid=0xdbbc runnable

"GC task thread#19 (ParallelGC)" prio=10 tid=0x00007f970803b800 nid=0xdbbd runnable

"GC task thread#20 (ParallelGC)" prio=10 tid=0x00007f970803d800 nid=0xdbbe runnable

"GC task thread#21 (ParallelGC)" prio=10 tid=0x00007f970803f000 nid=0xdbbf runnable

"GC task thread#22 (ParallelGC)" prio=10 tid=0x00007f9708041000 nid=0xdbc0 runnable

"GC task thread#23 (ParallelGC)" prio=10 tid=0x00007f9708043000 nid=0xdbc1 runnable

"GC task thread#24 (ParallelGC)" prio=10 tid=0x00007f9708044800 nid=0xdbc2 runnable

"GC task thread#25 (ParallelGC)" prio=10 tid=0x00007f9708046800 nid=0xdbc3 runnable

"GC task thread#26 (ParallelGC)" prio=10 tid=0x00007f9708048800 nid=0xdbc4 runnable

"GC task thread#27 (ParallelGC)" prio=10 tid=0x00007f970804a000 nid=0xdbc5 runnable

"GC task thread#28 (ParallelGC)" prio=10 tid=0x00007f970804c000 nid=0xdbc6 runnable

"GC task thread#29 (ParallelGC)" prio=10 tid=0x00007f970804e000 nid=0xdbc7 runnable

"GC task thread#30 (ParallelGC)" prio=10 tid=0x00007f970804f800 nid=0xdbc8 runnable

"GC task thread#31 (ParallelGC)" prio=10 tid=0x00007f9708051800 nid=0xdbc9 runnable

"GC task thread#32 (ParallelGC)" prio=10 tid=0x00007f9708053800 nid=0xdbca runnable

"VM Periodic Task Thread" prio=10 tid=0x00007f97083f7800 nid=0xdbd3 waiting on condition

JNI global references: 267

Found one Java-level deadlock:

"2102544801@qtp-1369161839-45":
waiting to lock monitor 0x00007f9604002078 (object 0x00000007e60ec4f8, a azkaban.scheduler.ScheduleManager),
which is held by "1057409309@qtp-1369161839-3"
"1057409309@qtp-1369161839-3":
waiting to lock monitor 0x00007f9630005ef0 (object 0x00000007e60f5038, a azkaban.scheduler.ScheduleManager$ScheduleRunner),
which is held by "Thread-6"
"Thread-6":
waiting to lock monitor 0x00007f9604002078 (object 0x00000007e60ec4f8, a azkaban.scheduler.ScheduleManager),
which is held by "1057409309@qtp-1369161839-3"

Java stack information for the threads listed above:

"2102544801@qtp-1369161839-45":
at azkaban.scheduler.ScheduleManager.getSchedules(ScheduleManager.java:120)
- waiting to lock <0x00000007e60ec4f8> (a azkaban.scheduler.ScheduleManager)
at azkaban.webapp.servlet.ScheduleServlet.handleGetAllSchedules(ScheduleServlet.java:292)
at azkaban.webapp.servlet.ScheduleServlet.handleGet(ScheduleServlet.java:80)
at azkaban.webapp.servlet.LoginAbstractAzkabanServlet.doGet(LoginAbstractAzkabanServlet.java:77)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:713)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
"1057409309@qtp-1369161839-3":
at azkaban.scheduler.ScheduleManager$ScheduleRunner.getRunnerSchedules(ScheduleManager.java:292)
- waiting to lock <0x00000007e60f5038> (a azkaban.scheduler.ScheduleManager$ScheduleRunner)
at azkaban.scheduler.ScheduleManager.getSchedules(ScheduleManager.java:120)
- locked <0x00000007e60ec4f8> (a azkaban.scheduler.ScheduleManager)
at azkaban.webapp.servlet.ScheduleServlet.handleGetAllSchedules(ScheduleServlet.java:292)
at azkaban.webapp.servlet.ScheduleServlet.handleGet(ScheduleServlet.java:80)
at azkaban.webapp.servlet.LoginAbstractAzkabanServlet.doGet(LoginAbstractAzkabanServlet.java:77)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:713)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
"Thread-6":
at azkaban.scheduler.ScheduleManager.removeSchedule(ScheduleManager.java:139)
- waiting to lock <0x00000007e60ec4f8> (a azkaban.scheduler.ScheduleManager)
at azkaban.scheduler.ScheduleManager$ScheduleRunner.run(ScheduleManager.java:444)
- locked <0x00000007e60f5038> (a azkaban.scheduler.ScheduleManager$ScheduleRunner)

Found 1 deadlock.

Heap
PSYoungGen total 611648K, used 376044K [0x00000007c0000000, 0x00000007eaaa0000, 0x0000000800000000)
eden space 524288K, 55% used [0x00000007c0000000,0x00000007d19ebd90,0x00000007e0000000)
from space 87360K, 99% used [0x00000007e5550000,0x00000007eaa9f430,0x00000007eaaa0000)
to space 87360K, 0% used [0x00000007e0000000,0x00000007e0000000,0x00000007e5550000)
ParOldGen total 1398144K, used 490971K [0x0000000740000000, 0x0000000795560000, 0x00000007c0000000)
object space 1398144K, 35% used [0x0000000740000000,0x000000075df76d68,0x0000000795560000)
PSPermGen total 27008K, used 26818K [0x0000000735a00000, 0x0000000737460000, 0x0000000740000000)
object space 27008K, 99% used [0x0000000735a00000,0x0000000737430a78,0x0000000737460000)

Unable to run java jars against the hadoop cluster on EMR, only local

This is probably not a bug or many others would have already reported it but I have hit the wall on what else to try.

I run this on the command line and it runs against the hadoop cluster as expected:

hadoop jar my.jar com.my.JobRunner com.my.Job --hdfs

Yet when I try all of these via azkaban here is what I get:

1. type=command
   command=hadoop jar my.jar com.my.JobRunner com.my.Job --hdfs
   -- runs, but not against the cluster, just local
2. type=javaprocess
   java.class= com.my.JobRunner
   main.args= com.my.Job --hdfs
   classpath=..the cp
   -- runs fine, but not against the cluster, just local
3. type=hadoopJava
   jobtype.classpath=...the cp
   main.args=com.my.Job --hdfs
   job.class=com.my.JobRunner.class
   jobtype.class=azkaban.jobtype.HadoopJavaJob
   obtain.binary.token=true

Command: java -Xms1536m -Xmx1536m -XX:MaxPermSize=1638m -Xms64M -Xmx256M -cp /home/hadoop/project-jars/:/home/hadoop/:/home/hadoop/lib/:/home/hadoop/azkaban-2.1/plugins/jobtypes/hadoopJava/azkaban-jobtype-2.1.jar:/home/hadoop/azkaban-2.1/lib/azkaban-2.1.jar:/home/hadoop/azkaban-2.1/plugins/jobtypes/hadoopJava/azkaban-hadoopsecuritymanager-2.1.jar:/home/hadoop/azkaban-2.1/plugins/jobtypes/hadoopJava/lib/:/home/hadoop/:/home/hadoop/lib/ azkaban.jobtype.HadoopJavaJobRunnerMain com.my.Job --hdfs
12-10-2013 16:18:24 PDT my_job_name INFO - Environment variables: {JOB_NAME=my_job_name, JOB_PROP_FILE=/home/hadoop/azkaban-2.1/executions/16931/my_job_name/my_job_name_1392369104419033110_tmp, JOB_OUTPUT_PROP_FILE=/home/hadoop/azkaban-2.1/executions/16931/my_job_name/my_job_name_output_9031288206276976184_tmp}
12-10-2013 16:18:24 PDT my_job_name INFO - Working directory: /home/hadoop/azkaban-2.1/executions/16931/my_job_name
12-10-2013 16:18:25 PDT my_job_nameINFO - INFO Running job my_job_name
12-10-2013 16:18:25 PDT my_job_name INFO - INFO Class name com.my.JobRunner.class
12-10-2013 16:18:25 PDT my_job_name ERROR - Exception in thread "main" java.lang.ClassNotFoundException: com.my.JobRunner.class
12-10-2013 16:18:25 PDT my_job_name ERROR - at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
12-10-2013 16:18:25 PDT my_job_name ERROR - at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
12-10-2013 16:18:25 PDT my_job_name ERROR - at java.security.AccessController.doPrivileged(Native Method)
12-10-2013 16:18:25 PDT my_job_name ERROR - at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
12-10-2013 16:18:25 PDT my_job_name ERROR - at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
12-10-2013 16:18:25 PDT my_job_name ERROR - at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
12-10-2013 16:18:25 PDT my_job_name ERROR - at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
12-10-2013 16:18:25 PDT my_job_name ERROR - at azkaban.jobtype.HadoopJavaJobRunnerMain.getObject(HadoopJavaJobRunnerMain.java:299)
12-10-2013 16:18:25 PDT my_job_name ERROR - at azkaban.jobtype.HadoopJavaJobRunnerMain.(HadoopJavaJobRunnerMain.java:146)
12-10-2013 16:18:25 PDT my_job_name ERROR - at azkaban.jobtype.HadoopJavaJobRunnerMain.main(HadoopJavaJobRunnerMain.java:76)
12-10-2013 16:18:25 PDT my_job_name INFO - Process completed unsuccessfully in 0 seconds.
12-10-2013 16:18:25 PDT my_job_name ERROR - Job run failed!

Even if I copy my fat jar to the local dir for hadoopJava it still can't find it
jobtype.classpath=/home/hadoop/azkaban-2.1/plugins/jobtypes/hadoopJava/,/home/hadoop/,/home/hadoop/lib/*

I have had other issues as well with hadoopJava - and i'm hoping this will allow the jar to run against the cluster but I don't understand why the command type using the hadoop command does not work, if it does when executing the jar directly vs via azkaban?

The docs are not clear as to what these should be:
job.class

• is this the runner with a main method that the FQCN of the actual job class to run is passed to as a string arg?
• is it the FQCN or with .class after?
• what does this accomplish? jobtype.class=azkaban.jobtype.HadoopJavaJob

thanks

Add auto retrying of failed jobs

Re-introduce 'retry' with a retry backoff.
Should be fairly straight forward when implemented in JobRunner.

change the rules of create flow

Azkaban 2.1, the leaf nodes as the goal to create the workflow, such as five jobs: a (independent), B1 (a dependent), C1 (a dependent), B2 (B1 dependent), C2 (C1 dependent), Azkaban 2.1 will create two workflow: C1, c2.

Do you have any parameters can change the rules, to the root node a as the goal, to create the workflow?create one flow,have five node:a,b1,b2,c1,c2.

azkaban-web-start.sh breaks in zsh

azkaban-web-start.sh don't work in zsh, but works in bash. Have you ever test the scripts on other shells?

And I think is better to add

#!/bin/bash

at the top of head.

So are the problem swith other azkaban--start.sh. But azakaban--shutdown.sh scripts work all well.

Command execution off by one

Installed 2.1 release of web, executor and plugin-types.
Tried command.zip downloaded from github site.
It has three commands. The executor only ran the first two.
The job file has a typo for the last command entry:
type=command
command=echo "hello"
command.1=echo "This is how one runs a command job"
comnand.2=whoami

com-N-and rather than com-M-and
This is the log:

01-08-2013 16:39:07 PDT command INFO - Running execid:1 flow:command project:2 version:1
01-08-2013 16:39:07 PDT command INFO - Update active reference
01-08-2013 16:39:07 PDT command INFO - Updating initial flow directory.
01-08-2013 16:39:07 PDT command INFO - Fetching job and shared properties.
01-08-2013 16:39:07 PDT command INFO - Starting flows
01-08-2013 16:39:07 PDT command INFO - Submitting job command to run.
01-08-2013 16:39:07 PDT command INFO - Job Finished command with status SUCCEEDED
01-08-2013 16:39:07 PDT command INFO - Job command had output props.
01-08-2013 16:39:07 PDT command INFO - Finishing up flow. Awaiting Termination
01-08-2013 16:39:07 PDT command INFO - Flow is set to SUCCEEDED
01-08-2013 16:39:07 PDT command INFO - Setting end time for flow 1 to 1375400347232

allow job level notification

Current notification is flow level.
Would like to refactor the code to do job level notification.
So each job should be able to specify different notification methods. Say a failed job can send email or other notification to different account/services.
What needs to be done:
add fields in flow/executableflow/executablenode
pick up settings from file in project upload
executormanager when updating executableflows, fire off event if a job in a flow has failed.
this can be monitored by a trigger that is fired along with every flow execution
the trigger may send out a summary notification which list all failed jobs as flow notification.

Connection refused error from executor when executing flow from azkaban-web

I get this issue when im trying to run a flow from the azkaban web interface

I got this from the azkaban log
azkaban.executor.ExecutorManagerException: org.apache.http.conn.HttpHostConnectException: Connection to http://localhost:12321 refused
at azkaban.executor.ExecutorManager.callExecutorServer(ExecutorManager.java:419)
at azkaban.executor.ExecutorManager.submitExecutableFlow(ExecutorManager.java:390)
at azkaban.webapp.servlet.ExecutorServlet.ajaxExecuteFlow(ExecutorServlet.java:732)
at azkaban.webapp.servlet.ExecutorServlet.ajaxAttemptExecuteFlow(ExecutorServlet.java:698)
at azkaban.webapp.servlet.ExecutorServlet.handleAJAXAction(ExecutorServlet.java:293)
at azkaban.webapp.servlet.ExecutorServlet.handleGet(ExecutorServlet.java:69)
at azkaban.webapp.servlet.LoginAbstractAzkabanServlet.doGet(LoginAbstractAzkabanServlet.java:77)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:401)
at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
at org.mortbay.jetty.Server.handle(Server.java:326)
at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:928)
at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:549)
at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:713)
at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)

Documentation site redesign

• Non-documentation pages moved to azkaban.github.io
• Apply new Azkaban Jekyll theme (see davidzchen/azkaban-jekyll)
• Single-page documentation

Tracked by internal JIRA ticket: HADOOP-4410

Invalid schedules remain after updating project flow.

Steps to Reproduce

1. Upload a project flow such as:

   # test1.job
   type=command
   command=echo "Hello, world!"
   
   # test2.job
   type=command
   dependencies=test1
   command=echo ${azkaban.flow.start.day}
   
   # test3.job
   type=command
   dependencies=test1
   command=echo ${azkaban.flow.start.day}
   

2. Schedule the new flows.

3. Upload a new .zip file for the project that contains the above jobs, plus the following.

   # test4.job
   type=command
   dependencies=test2,test3
   command=echo ${azkaban.flow.start.day}
   

What Happens

The project now has only one flow named test4. The list of scheduled flows still contained the test2 and test3 flows. When the next execution time for flow test3 arrives it is not executed and a message like the following is logged:

2013/11/11 10:55:50.228 -0500 ERROR [TriggerManager] [Azkaban] Failed to do action Execute flow test3 from project test-proj
java.lang.RuntimeException: Error finding the flow to execute test3
    at azkaban.trigger.builtin.ExecuteFlowAction.doAction(ExecuteFlowAction.java:217)
    at azkaban.trigger.TriggerManager$TriggerScannerThread.onTriggerTrigger(TriggerManager.java:247)
    at azkaban.trigger.TriggerManager$TriggerScannerThread.checkAllTriggers(TriggerManager.java:229)
    at azkaban.trigger.TriggerManager$TriggerScannerThread.run(TriggerManager.java:193)

What is Expected

Ideally the GUI would give some feedback that the new flow definition invalidated the existing schedules and give the user a chance to correct the situation or abort the flow update.

[Feature Request] Attatch Logs to failed workflow email

I would like to see more context in a workflow failure email. Attaching the logs from the workflow working path should give me a better idea as to why the workflow failed.

Job Edit

Re-introduce job edit.
Along with this feature, would like to allow job edit of running flows.

Email message send failed

com.sun.mail.smtp.SMTPSendFailedException: 530 5.7.0 Must issue a STARTTLS command first. dk3sm36522188pbc.32 - gsmtp

at com.sun.mail.smtp.SMTPTransport.issueSendCommand(SMTPTransport.java:2114)
at com.sun.mail.smtp.SMTPTransport.mailFrom(SMTPTransport.java:1618)
at com.sun.mail.smtp.SMTPTransport.sendMessage(SMTPTransport.java:1119)
at azkaban.utils.EmailMessage.sendEmail(EmailMessage.java:170)
at azkaban.executor.ExecutorMailer.sendSuccessEmail(ExecutorMailer.java:167)
at azkaban.executor.ExecutorManager.finalizeFlows(ExecutorManager.java:725)
at azkaban.executor.ExecutorManager.access$1000(ExecutorManager.java:52)

my property followed:

mail settings

mail.sender=...

mail server to send job success/failure notifications to

mail.smtp.starttls.enable=true
mail.host=smtp.gmail.com
mail.user=...
mail.password=...

mail.port=587

job.failure.email=...
job.success.email=[email protected]

The version is azkaban-2.1, and ssh turned on.
Did I set the properties wrong, looking forward to the answer.

When upload big zip file, got error

When I uploaded zip file which is more than 100mb, I got

"Installation Failed.
Error chunking"

Add additional Rest API

1. Get job start time
2. Get Job Status
3. Get Job Scheduled Time
4. Get Job 95th/99th Percentile run time (might not be in first run through).
5. Access to output parameters
6. Access to compressed log (download feature)
7. Batch API - if you can

8. of mappers, # of reducers - Job tracker level if you have this information (not yet possible)

Job edit and flow parameter edit clears results if multiclicked

If clicked on twice, the edit text field for job and flow parameters clears previous results.

job dependencies bug in runtime

Recently I deployed an azkaban system, and it is really awesome. But a bug appeared yesterday nigtht. I have tested it on my local computer, and the bug is for sure there.

I tested the bug with some python jobs, as shown in the image in the attachment, job2,job3,job5 were just some python code, with time.sleep(seconds), and job4 was intended to fail because i raised an hard-coded exception in it.

When the flow began, job2 and job4 would be executed very quickly and after 5 seconds ,job3 would began. When it came to job5, it will be killed because the system assumed the job2 was failed.

of course, we do not want it that way. job5 should be launched because its parents were all successfully executed.

upload zip file error

when I upload the flow the correct zip file using my Chrome and Firefox, I comes the
"File type application/octet-stream unrecognized.",I seems that the post the brower does not recognize the zip file as application/octet-stream(http://stackoverflow.com/questions/856013/mime-type-for-zip-file-in-google-chrome). I think it is a bug.

HDFS Viewer Plugin (2.1) java.lang.NoClassDefFoundError: org/apache/hadoop/security/AccessControlException

Ran into this exception when trying to set up the hdfs viewer.

Version: Azkaban 2.1 (web server, executor and plugin)

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/security/AccessControlException
at java.lang.Class.getDeclaredConstructors0(Native Method)
at java.lang.Class.privateGetDeclaredConstructors(Class.java:2389)
at java.lang.Class.getConstructor0(Class.java:2699)
at java.lang.Class.getConstructor(Class.java:1657)
at azkaban.webapp.AzkabanWebServer.loadViewerPlugins(AzkabanWebServer.java:585)
at azkaban.webapp.AzkabanWebServer.main(AzkabanWebServer.java:446)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.security.AccessControlException
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
... 6 more

schedulermanager thread issue

in azkaban.scheduler/SchedulerManger.java line 481-482

addRunnerSchedule(runningSched); loader.updateSchedule(runningSched);

since the "addRunnerSchedule" has interrupted the thread, the next line will not be executed, so the next execute time of not be updated. small but vital bug.

forgive my stupid poor english. :) :)

Display Azkaban version in web UI footer

To make it clear which version of Azkaban is running, the Azkaban version should be displayed in the page footer in the web UI.

I removed the footer a while ago when I was making the graph and log pages fill the screen, but now, we should add the footer back in.

2.1 hdfs browser plugin download (https://s3.amazonaws.com/azkaban2/azkaban-plugins/azkaban-hdfs-viewer-2.1.tar.gz) throwing 403's

looks like a standard 403 from S3 to me... Just an FYI, thanks for the work on the new site, looks awesome.

Standardize time

Some times show up in azkaban as utc, some as whatever the default timezone is.
We should standardize these times.

change log level of Azkaban log from ERROR to INFO for output of jobs

The Azkaban logs routinely prints out messages with log level ERROR when they are not errors: e.g., Hadoop job outputs etc. This should be changed to INFO or something else suitable

Job executions have no end time on Job History page

On the Job History page for a given job, job executions, regardless of whether successful or failed, have no end time even though they have start times and durations. This also causes the time graph to be displayed incorrectly.

Add exflow on exflow dependency

Introduce the ability to have exflow dependent on the execution of another exflow.

In a more granular way, we want to have exflow jobs being able to watch other exflows jobs. This would allow pipelining, a better defined way of restarting 'failed finishing jobs' and job level blocking.

This should be fairly easy:
-if the exflow is running local, watch local.
-if the exflow is running remotely, update from the datastore.

[Feature Request] add "cleanup" or "finally" job to handle cleanup of flow in the event of failure

I would like to have a flag or some other indicator that would label a job as a "cleanup" job that would run whether or not the preceding jobs failed. Kind of like a "finally" block in Java. This would allow things like cleanup up an EMR cluster in the event of failed Hadoop job.

Ability to customize Flow name

The current implementation derives the Flow name from the job dependencies. The Flow name uses the name of the last job in the flow. It would be preferable to allow this to be customizable.

Note: The workaround solution is to add name the last step to match the desired flow name.

Ajax the upload command

Currently upload is a multipart form post http request, but assumes it was from a browser.

Add an ajax version that will return a json.

Scheduling time problem

Scheduling a job for 12:00AM (midnight) will result in a job at 12:00PM (noon)

Support .tar files for deploying projects

I'm developing and deploying to Azkaban on a Mac (for testing).
If I upload a zip file with a bash shell script that has execute permissions set, then call that script in a job, I have to use "bash scriptname" in the job. Apparently, the script file written into Azkaban's executor service ".../projects/M.N" directory doesn't have the original file permissions.

jobs depends on the failed retry job will be killed by Azkaban with no email

I

I run the big flow on the Azkaban, the red jobs are killed(Not in failed status) by Azkaban.And the Azkaban does not give the failed email. By reading the source code, it seems that the retried jobs is in failed status,so the jobs depending on the failed retried job should be killed.

html garbled

I found a bug, when VM file contains the Chinese will be messy code, the solution is to modify the azkaban.webapp.servlet.Page, in the render() method, response.setContentType (mimeType); the upper line, add the following two lines of code:
Response.setHeader ("Content-type", "text/html; charset=UTF-8");
Response.setCharacterEncoding ("UTF-8");

Cannot edit a flow to run only one job when it has siblings.

In the image above, I am in an execute workflow and have disabled all jobs. I should be able to enable one job of the workflow.

hello, there, is there an document on plugins, something like hive's configuration

i've upload jobs as the website say. the command.zip runs fine. but the others all fail.
i've modified the configuration file, but not sure about it.

such as hive-demo, i've installed hive, and set the necessary environment, but it doesn't work.

hope there is an detail document about it.

Thank you!

flow execution status stays at "PREPARING", even through the executor server side has executed the job successfully

the issue happened when executing the flow, as a result, runningFlows at webserver side( see ExecutorManager ) never remove the execId for its status is always stuck in "preparing" , then logically it goes to callExecutorServer rather than retrieve log from DB, however, callExecutorServer throws "flow not found" exception because the flow has already been removed from runningflow at server side after the execution has finished successfully at executorServer,

I don't know whether the above issue is a bug or a known issue, but it has confused me for nearly a week, i don't think it's a deploy or install issuer, i feel maybe sth need to be changed,

I really really need someone to help me solve this issue ASAP, contact me at [email protected] or mobile [phone]

Remove Ehcache

Ehcache is used in the SessionCache. It was a convenient way to have sessions expire.

However, it also writes 'update' messages, and may or may not 'phone home' which is a huge bother. Since we're not using Ehcache's complicated features (like dumping to disk), might as well replace this with a simple thread that periodically wakes and expires sessions.

Azkaban soloserver needs documentation

It appears that the Azkaban soloserver is not documented. Documentation on how to set up Azkaban in soloserver mode should be added.

TriggerRunnerManager-Trigger-Scanner-Thread FAILD

when i run bin/azkaban-web-start.sh

2014/01/28 13:57:48.357 +0800 INFO [Condition] [Azkaban] Testing condition BasicTimeChecker_1.eval()
2014/01/28 13:57:48.369 +0800 INFO [Condition] [Azkaban] Testing condition BasicTimeChecker_2.eval()
Exception in thread "TriggerRunnerManager-Trigger-Scanner-Thread" java.lang.IllegalMonitorStateException
at java.lang.Object.wait(Native Method)
at azkaban.trigger.TriggerManager$TriggerScannerThread.run(TriggerManager.java:242)
2014/01/28 13:57:48.696 +0800 INFO [log] [Azkaban] Started [email protected]:8443
2014/01/28 13:57:48.696 +0800 INFO [AzkabanWebServer] [Azkaban] Server running on ssl port 8443.

So, schedule is invalid.

Sessions don't seem to expire

Default session expiry is a day. I've gotten reports that expiry is not happening as expected.

[Feature Request] ability to deploy/undeploy jobs via API

Please ignore this request if such a feature already exists.

I'd be good to have an API that allows job manipulation such us deploy, undeploy, run, stop, etc.

We are interesting in auto-deployments of jobs driven by our build server.

Thanks.

Change KILLED states to CANCELLED for unrun failed jobs

Current behaviour is to mark any subsequent job to a failure as KILLED. However, that is also the state of any job that was running, but was killed.

We should add a Cancelled state for jobs that have not been run, but are skipped due to a previous failure state.

When unpacking the job, file modification dates are not honored

When Azkaban unpacks the job, I would expect the modification dates of the files to be the original dates kept in the ZIP file. Instead, it just creates new files with new modification dates.

Need to set the date to the original one.

Role permission to turn off 'Create Project' feature

Add a role permission that blocks the creation of new projects.