g-andrade / taskforce

On-demand worker pools for parallelizable tasks

taskforce

This library is not under active maintenance; if you'd like to perform maintenance yourself, feel free to open an issue requesting access.

Copyright (c) 2015-2022 Guilherme Andrade

Version: 1.2.3

Authors: Guilherme Andrade ([email protected]).

taskforce allows you to parallelise arbitrary tasks in a controlled way.

---

Creating tasks

Tasks = #{ some_task => taskforce:task(fun work/0, Args, #{ timeout => 2000 }),
           similar_task => taskforce:task(fun work/0, Args123, #{ timeout => 2500 }),
           other_task => taskforce:task(fun other_work/1, OtherArgs, #{ timeout => 500 }) }.

Executing tasks

#{ completed := Completed } = taskforce:execute(Tasks),
SomeTaskResult = maps:get(some_task, Completed),
SimilarTaskResult = maps:get(similar_task, Completed),
OtherTaskResult = maps:get(other_task, Completed).

Finely tuning execution

ExecutionOptions = #{ max_workers => 8, timeout => 5000 },
#{ completed := Completed } = taskforce:execute(Tasks, ExecutionOptions),
% ...

Individual task timeouts

% ...
#{ individual_timeouts := IndividualTimeouts } = taskforce:execute(Tasks),
(length(IndividualTimeouts) > 0
 andalso io:format("oh noes! tasks with ids ~p timed-out",
                   [IndividualTimeouts]))

Global execution timeouts

% ...
#{ global_timeouts := GlobalTimeouts } = taskforce:execute(Tasks),
(length(GlobalTimeouts) > 0
 andalso io:format("execution ran out of time; tasks with ids ~p timed-out",
                   [GlobalTimeouts]))

Full example

%
% Calculate first 200 prime numbers using 4 processes,
% with a timeout of 2s per individual calculation and 10s
% for the whole batch.
%
NrOfPrimes = 200,
Tasks =
    maps:from_list(
        [{Nth, taskforce:task(fun fancy_lib:find_nth_prime/1,
                              [Nth], #{ timeout => 2000 })}
         || Nth <- lists:seq(1, NrOfPrimes)]),

ExecutionOptions =
    #{ % default is the sum of all individual task timeouts
       timeout => 10000,

       % default is number of active schedulers
       max_workers => 4 },

#{ completed := NthPrimes } = taskforce:execute(Tasks, ExecutionOptions),
io:format("200th prime is: ~p~n", [maps:get(200, NthPrimes)]).

Also in examples/.

Modules

taskforce