bootique / bootique-job

Provides a job execution framework with Bootique integration

Bootique is a minimally opinionated java launcher and integration technology. It is intended for building container-less runnable Java applications. With Bootique you can create REST services, webapps, jobs, DB migration tasks, etc. and run them as if they were simple commands. No JavaEE container required! Among other things Bootique is an ideal platform for Java microservices, as it allows you to create a fully-functional app with minimal setup.

Each Bootique app is a collection of modules interacting with each other via dependency injection. This GitHub project provides Bootique core. Bootique team also develops a number of important modules. A full list is available here.

Quick Links

• WebSite
• Getting Started
• Docs - documentation collection for Bootique core and all standard modules.

Support

You have two options:

• Open an issue on GitHub with a label of "help wanted" or "question" (or "bug" if you think you found a bug).
• Post a question on the Bootique forum.

TL;DR

For the impatient, here is how to get started with Bootique:

• Declare the official module collection:

<dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>io.bootique.bom</groupId>
            <artifactId>bootique-bom</artifactId>
            <version>3.0-M4</version>
            <type>pom</type>
            <scope>import</scope>
        </dependency> 
    </dependencies>
</dependencyManagement>

• Include the modules that you need:

<dependencies>
    <dependency>
        <groupId>io.bootique.jersey</groupId>
        <artifactId>bootique-jersey</artifactId>
    </dependency>
    <dependency>
        <groupId>io.bootique.logback</groupId>
        <artifactId>bootique-logback</artifactId>
    </dependency>
</dependencies>

• Write your app:

package com.foo;

import io.bootique.Bootique;

public class Application {
    public static void main(String[] args) {
        Bootique
            .app(args)
            .autoLoadModules()
            .exec()
            .exit();
    }
}

It has main() method, so you can run it!

For a more detailed tutorial proceed to this link.

Upgrading

See the "maven-central" badge above for the current production version of bootique-bom. When upgrading, don't forget to check upgrade notes specific to your version.

bootique-job's Issues

Pass parameters to ad-hoc job executions

Hi @andrus , may I propose adding an additional version of com.nhl.bootique.job.scheduler.Scheduler#runOnce with additional Map<String, Object> parameter? This will either override all "default" params, collected from environment and YML config, or - alternatively - will be merged into the default params, preserving params that are absent from the Map passed from the caller.

Exec command: Order and repetition of jobs

Talking about --exec command it would be useful to preserve the execution order and repetition of jobs came from the program arguments as --job options.

Value object: Cron

Let's create a value object for cron expressions, so that we can later use it in TriggerDescriptor.cron configuration property (which is now a String).

Bridge commons-logging to SLF4J

Looks like we inherit commons-logging via Spring in the job framework. Let's bridge that to SLF4j and remove the dependency.

Make Scheduler DI binding a singleton

This would be convenient for tests, and I don't see a reason for it not to be a singleton anyway.

Non-blocking ScheduleCommand

Per bootique/bootique#197 we can make ScheduleCommand unblock immediately after the Jetty server start, returning succeededAndForkedToBackground outcome.

Trigger: support for 'initialDelayMs' YAML parameter

Currently it is hardcoded at 10 sec. Invconvenient for unit tests, etc.

Better error reporting for unregistered jobs

A common user error is an attempt to run a job without registering it via JobModule.extender(..). Here is a typical error:

java.lang.NullPointerException
	at io.bootique.job.scheduler.execution.DependencyGraph.getOrCreateExecution(DependencyGraph.java:77)
	at io.bootique.job.scheduler.execution.DependencyGraph.populateWithDependencies(DependencyGraph.java:39)
	at io.bootique.job.scheduler.execution.DependencyGraph.<init>(DependencyGraph.java:26)
	at io.bootique.job.scheduler.execution.DefaultJobRegistry.getJob(DefaultJobRegistry.java:88)
	at io.bootique.job.scheduler.DefaultScheduler.findJobByName(DefaultScheduler.java:140)
	at io.bootique.job.scheduler.DefaultScheduler.runOnce(DefaultScheduler.java:130)
	at io.bootique.job.scheduler.DefaultScheduler.runOnce(DefaultScheduler.java:125)
	at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
	at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1374)
	at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:481)
	at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:471)
	at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
	at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
	at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:499)

So instead of a bare NPE, it would be great to throw a BootiqueException with explanation of the problem.

Upgrade to bootique 0.22 , implement own "extend" API

Upgrade to bootique-modules-parent 0.8

Upgrade to Bootique 0.20

There are some breaking change in 0.20 ... Upgrading all standard modules

ConsulLockHandler local job locking

ConsulLockHandler doesn't lock job execution on one runtime, like in LocalLockHandler. It's seems like unexpected behavior.
More details in tests: #70

Job exception is not propagated to the shell

$ java -jar target/myapp-0.1.0.jar  --exec --job=myjob
[Exception is thrown by 'myjob']

$ echo $?
0

Exist code should reflect the error.

Typo in JobRegistry.allowsSimlutaneousExecutions() method name

Upgrade to BQ 0.18 and bootique-curator

Update Bootique version, and also switch from bootique-zookeeper to bootique-curator per this issue.

Config self-documentation

Let's expand module provider and annotate factories and subfactories with BQConfig and BQConfigProperty so that -H command could print configuration details.

Do not start the scheduler before performing checks

Currently, if any problems are detected in DefaultScheduler#start() (e.g. triggers for unknown jobs), the call to start() throws an exception. The next call to start() will not even attempt to check anything, it will fail immediately, claiming that the scheduler has already been started (per previous invocation). It's not a big deal, when using the standard ScheduleCommand (as the program terminates on exception), but can be problematic, when using the Scheduler directly.

Instrumented jobs module

Via recently added JobListener's

ScheduleCommand always returns exit code 1

In a similar vein as #43 , ScheduleCommand uses Thread.currentThread().join() and therefore never returns unless interrupted, in which case it returns exit code 1. In other words, return CommandOutcome.succeeded() is effectively unreachable.

Upgrade to Bootique 0.15

Primarily needed to use bootique-test, but generally a good idea never the less.

Scheduler thread pool must be identifiable by name

Currently the scheduler thread pool is using thread names like ThreadPoolTaskScheduler-XX which does not intuitively point to its origin. So let's rename the threads to use bootique-jobs-XX format.

Move contribution API from JobBinder into static methods on JobModule

Following the pattern in Bootique core, all module contributions should be exposed on the Module class itself (JobModule in this case). JobBinder will be deprecated.

Double --job option

Looks like both "schedule" and "exec" commands contribute "--job" option, resulting in an odd help like this:

--job=job_name
   Specifies the name of the job to schedule. Available job names can be
   viewed using '--list' command.

--job=job_name
   Specifies the name of the job to run with '--exec'. Available job
   names can be viewed using '--list' command.

I guess we need to take the option out of both commands and define it independently.

Allow empty schedulers to start

Currently DefaultScheduler would blow up if attempted to start it with no triggers. I guess such second guessing the user is pointless. Starting an empty scheduler may not make much sense, but it is not invalid either.

Move JobMetadataBuilder inside JobMetadata

Move JobMetadataBuilder inside JobMetadata. This will be similar to how the latest Bootique CommandMetadat, etc. are assembled.

Use non-zero exit code in ExecCommand, when some of the jobs have failed

Simple job has no try-catch block and just throws runtime exception:

public class SomeJob extends BaseJob {

    public SomeJob() {
        super(JobMetadata.build(SomeJob.class));
    }

    private class Task {
    	public void doTask() {
    	    throw new CayenneRuntimeException();
    	}
    }

    @Override
    public JobResult run(Map<String, Object> params) {
        new Task().doTask();
        return JobResult.success(getMetadata());
    }

} 

--exec --job some
[13/Oct/2017:13:42:42,189] main ERROR i.b.j.c.ExecCommand: Finished job 'some', result: UNKNOWN, message: null

Then io.bootique.job.command.ExecCommand.run(Cli) handles exception, ignores even JobResult.failure and responses CommandOutcome.succeeded() to the environment, i.e. 0 what is considered as success.

For instance, Bamboo logs
13-Oct-2017 09:42:42 | Finished task 'Run Some Job' with result: Success

The point is if runtime exception has occurred then job result will be failure and not equals 0.

In the situation we use job to build database data and when some data is missing or impacted then deployment process should stop but with 0 the process is still continuing.

SerialJob annotation has no effect

I recently discovered this critical issue, which was supposedly introduced per #15 . I added a test to reproduce the problem and a simple fix.

Clustered execution interval

If I'm reading the code correctly, Zookeeper is used for locking (preventing two executions at once) but not for synchronising the execution timing.

So if I had a cluster of 5 applications, with a scheduled event once an hour, each application will try to run it once an hour and we'd get executions every 12 minutes on average.

A possible solution might be to store the last run timestamp in the ZK node, and make persist that node between executions.

Add integration tests for bootique-job-zookeeper

It will be nice to add integration tests for bootique-job-zookeeper module (using TestContainers lib)

ListCommand - sort jobs alphabetically

When running -l in apps with lots of jobs, it is hard to visually locate the job you need. Let's sort the job names alphabetically.

Integrate "business transaction" ids in job-instrumented

A unique ID generator is available in bootique-metrics per bootique/bootique-metrics#16 and was already integrated in jetty-instrumented. Let's integrate it in job-instrumented so that unique tx ids could be printed in the logs for job executions via %X{txid}.

Provide info on job params in YAML

Need API to retrieve job parameters from YAML config (in some sense they are perceived as "default" by users).

Filter scheduled triggers by jobs specified with --job arguments

I.e. allow to schedule only a subset of configured triggers.

Remove circular dependency DefaultJobRegistry <-> Scheduler

See comments in bootique/bootique-di#6

Upgrade to BQ 0.21

Including bootqiue-job 0.13 in in 0.21 train, so upgrading Bootique.

Switch TriggerDescriptor properties to value objects

Here is a subset of job scheduler configuration related to TriggerDescriptor

cron: <string>
fixedDelayMs: <long>
fixedRateMs: <long>
initialDelayMs: <long>

We need to change these properties to use Cron (see #65 ) and Duration (see bootique/bootique#219) value objects. Since Duration can be expressed in various units, we also need to rename properties that end in "Ms" (with full deprecation). So the resulting set of properties will look like this:

cron: # Cron String
fixedDelay: # Duration String
fixedDelayMs: # deprecated, Long, internally converted to Duration
fixedRate: # Duration String
fixedRateMs: # deprecated, Long, internally converted to Duration
initialDelay: # Duration String
initialDelayMs: # deprecated, Long, internally converted to Duration

Upgrade Bootique to 0.12

0.12 version of Bootique includes a completely redone command API. Need to upgrade Job commands to that new API.

Removing deprecated code

Removing JobBinder and JobMetadataBuilder that are deprecated since 0.9

ListCommand should not fail when there are no jobs

The current behavior of -l when no jobs are configured is to fail:

Error running command '--config=classpath:default.yml -l': No jobs are available.

Though logically this is not an error condition. The command should simply print a no-jobs message and return successfully.

Metrics renaming to follow naming convention

Renaming metrics per bootique/bootique-metrics#30 :

• io.bootique.job.instrumented.InstrumentedJobListener.[jobname]-success-counterrenamed to bq.Job.[JobName].Success
• io.bootique.job.instrumented.InstrumentedJobListener.[jobname]-failure-counterrenamed to bq.Job.[JobName].Failure
• io.bootique.job.instrumented.InstrumentedJobListener.[jobname]-active-counterrenamed to bq.Job.[JobName].Active
• io.bootique.job.instrumented.InstrumentedJobListener.[jobname]-completed-counterrenamed to bq.Job.[JobName].Completed
• io.bootique.job.instrumented.InstrumentedJobListener.[jobname]-timerrenamed to bq.Job.[JobName].Time

Provide info on currently running jobs

I can see two ways of accomplishing this:

1. Register start/end of job execution in RunnableJob.run() method and provide a simple API for retrieving a list of jobs, that were started but hadn't yet finished
2. Store JobFutures somewhere and provide an API for querying current (and possibly historical) data, that will be based on the list of active and completed futures

Scheduled job exceptions are not reported

If a job is scheduled and doesn't catch its own exceptions, the exception is being caught inside Callback.runAndNotify(..) , wrapped in FAILED JobResult, and never reported. Also a side affect of such exception is the line i.b.j.JobLogListener: job 'games' finished printed twice (once from the catch block, and another time under normal flow).

Job groups are not listed with "--list"

Doesn't look like --list shows job groups defined in yaml. It only prints actual jobs.

Cleaning up APIs deprecated since <= 0.25

Part of the overall deprecated API cleanup per bootique/bootique#214

JobGroup feature doesn't pass incoming parameters to job executions

My code uses Scheduler.runOnce(String jobName, Map<String, Object> parameters) API to manage job runs and keep them interacting with the external environment (cancellation, status reporting and etc).
It works perfectly when it runs a single job - all incoming parameters are being transferred into a running job.

But it doesn't work for job groups:
I see that JobGroup source code doesn't pass Map<String, Object> parameters downstream to a group job executions:

public JobResult run(Map<String, Object> parameters) { traverseExecution(jobExecutions -> { Set<JobResult> results = execute(jobExecutions, jobMap); ... }

Convert parameters based on their type from metadata

I accidentally introduced a bug while implementing #15 , that has broken the conversion of job params based on the Java type of parameter, specified in the job metadata

Support for listener ordering

Let's add an explicit way to sort added listeners - a MappedListener wrapper around the listener object that contains an int ordering. Lower ordering means an outer listener, higher - inner. This will help to sort out metrics listeners, MDC, etc.

Note: the issue was created to implemented listener ordering as a separate feature for bq-job (parent issue #48) by analogy with bq-jetty MappedListeners (#66). It's arisen from PR#53.

Scheduler is not shutdown properly

We are creating a dedicated thread pool for the Scheduler (encapsulated inside of ThreadPoolTaskScheduler). Yet we don't register it with ShutdownManager, so scheduler shutdown in tests or real apps doesn't work properly, leaving pool threads behind.

Job parameters can not be overridden with declared variables

Here is a test demonstrating the problem (under ExecutionIT)

@Test
public void testExecution_ParameterizedJob1_ParametersOverriddenWithVars() {
    ParameterizedJob2 job = new ParameterizedJob2();

    testFactory.app("--exec",
            "--job=parameterizedjob2")
            .autoLoadModules()
            .module(b -> JobModule.extend(b).addJob(job))
            .module(b -> BQCoreModule.extend(b)
                  .declareVar("jobs.parameterizedjob2.params.longp", "TEST_PARAM"))
            .module(b -> BQCoreModule.extend(b).setVar("TEST_PARAM", "35"))
            .createRuntime()
            .run();

   // fails here ... actual longp is null
    assertExecutedWithParams(job, Collections.singletonMap("longp", 35l));
}

The cause seems to be the user of uppercase for the job name i the resolved config:

Move to io.bootique namespace

Also implies upgrade to bootique 0.19 and curator 0.10