kureuil / batch-rs Goto Github PK
View Code? Open in Web Editor NEWBackground job library for Rust supporting RabbitMQ
Home Page: https://kureuil.github.io/batch-rs/
License: Apache License 2.0
Background job library for Rust supporting RabbitMQ
Home Page: https://kureuil.github.io/batch-rs/
License: Apache License 2.0
The "Batch in action" example currently written in the README doesn't compile. We should set up a tool such as rust-skeptic to automatically check that the README stays up-to-date with the state of the project.
Also it would seem there's tabs instead of spaces.
Even if the library should be able to work correctly on macOS, no tests were made and the CI doesn't run the tests on a macOS system.
When running the tests on Ubuntu Trusty (14.04), Travis provides version 3.6 of RabbitMQ. We should upgrade it to 3.7 to test the same version on all 3 OS.
Also, it would seem that RabbitMQ 3.6 support has recently stopped.
When I run multiple instances, a connection error appears one after the other.
root@10f834387d04:/app/batch# cargo run --example simple-client
Finished dev [unoptimized + debuginfo] target(s) in 0.0 secs
Running `/app/target/debug/examples/simple-client`
Starting RabbitMQ client example
root@10f834387d04:/app/batch# cargo run --example simple-client
Compiling batch v0.1.1 (file:///app/batch)
Finished dev [unoptimized + debuginfo] target(s) in 19.29 secs
Running `/app/target/debug/examples/simple-client`
Starting RabbitMQ client example
root@10f834387d04:/app/batch# cargo run --example simple-client
Finished dev [unoptimized + debuginfo] target(s) in 0.1 secs
Running `/app/target/debug/examples/simple-client`
Starting RabbitMQ client example
root@10f834387d04:/app/batch# cargo run --example simple-client
Finished dev [unoptimized + debuginfo] target(s) in 0.1 secs
Running `/app/target/debug/examples/simple-client`
Starting RabbitMQ client example
^[[AERROR 2018-04-20T21:39:03Z: lapin_futures::transport: upstream poll gave error: Os { code: 104, kind: ConnectionReset, message: "Connection reset by peer" }
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Error { inner:
An error occured in the RabbitMQ broker: Connection reset by peer (os error 104) }', libcore/result.rs:945:5
note: Run with `RUST_BACKTRACE=1` for a backtrace.
Tracking issue for the upcoming futures/tokio 0.2 ecosystem migration. It should be pretty straightforward once our dependencies will be updated:
Amazon SQS seems to be a pretty popular choice when needing a managed queue service on AWS.
The simplest way to use it will probably be to use one of the Rusoto crates, the AWS SDK for Rust.
In Rust 2018 Edition, paths have been reworked and the code generated by our macros might be incompatible with this rework.
The auto-generated task name is just the type name, making it too easy to get name collisions. We should use the module_path!
macro to prefix the task name with its containing module.
There seems to be more momentum behind the term Job
, and keeping the term Task
might induce some confusion as the futures
is using the same word for their work unit.
Manual Debug
implementation of batch
's types format their output using a string as a template but they should be replaced by the debug_struct
method.
Proc macro code is currently rather messy and can't be easily refactored because it lacks tests.
We should integrate [compiletest-rs] to make it easy to work on the *-codegen
crates.
compiletest-rs: https://github.com/laumann/compiletest-rs
When the worker receives a job from the broker, it spawns a brand new process per job. While this is easy to implement and monitor, it is sub-optimal on the performance side.
One possible solution to this performance problem would be to have a pool of processes, which are kept alive as long as possible. The main worker process would send over to these "worker worker" processes the jobs to execute, and they would send a message back to acknowledge or reject the job execution. If one the processes dies, it should be restarted and the job should be sent back to the broker as "rejected".
Maintain a changelog, allowing users to quickly understand what will be required to upgrade from one version to another.
See Keep a Changelog
When building a Worker
, if a Task
cannot be associated to any Queue
using its exchange & routing key a warning should be emitted to inform the user as their's a high chance it's an error.
When using batch-derive
, the task of the name is directly inferred from the struct's location (its absolute module path) and from the struct's name which might give a name such as webapp::tasks::SendSignupEmail
. However this task name format isn't often used, the separator is usually a dot (.
) instead of a double colon (::
), this should be changed. It will probably need to use the lazy-static
crate as Task::name()
should still return a &'static str
and String
operations on literals are not executed at compile-time.
Faktory is a new message broker (written by the author of sidekiq
) that aims to be as language agnostic as possible. It probably shouldn't be too hard to write an adapter for it given how it works.
There already exists a client faktory-rs
but its a high-level one, and it only supports synchronous I/O for now.
Disque is a message broker written by the author of Redis. Disque aims to be as simple as Redis while incorporating the high-level blocks of a message broker (message TTL, acknowledgements & retries, etc.). It is currently pending a rewrite as a Redis module.
Such a backend will allow us to store the status of past, current and upcoming tasks in a sensible and will allow us to easily build dashboards to visualize tasks & stats, and more.
In the first version, the backend will probably by SQL based though we might want to loosen the requirement on this one at a later point.
During the installation of RabbitMQ 3.7. Doesn't seem to be related to batch's code:
$ if [ "$TRAVIS_OS_NAME" = linux ]
then
sudo mv /opt/jdk_switcher/jdk_switcher.sh /tmp
sudo apt-get install rabbitmq-server
sudo mv /tmp/jdk_switcher.sh /opt/jdk_switcher/
sudo service rabbitmq-server start
fi
Reading package lists...
Building dependency tree...
Reading state information...
The following packages were automatically installed and are no longer required:
couchdb-common libmozjs185-1.0
Use 'sudo apt autoremove' to remove them.
The following packages will be upgraded:
rabbitmq-server
1 upgraded, 0 newly installed, 0 to remove and 186 not upgraded.
Need to get 9,105 kB of archives.
After this operation, 4,850 kB of additional disk space will be used.
Get:1 https://dl.bintray.com/rabbitmq/debian trusty/main amd64 rabbitmq-server all 3.7.6-1 [9,105 kB]
Fetched 9,105 kB in 3s (2,465 kB/s)
(Reading database ... 100104 files and directories currently installed.)
Preparing to unpack .../rabbitmq-server_3.7.6-1_all.deb ...
* Stopping message broker rabbitmq-server * message broker already stopped
Unpacking rabbitmq-server (3.7.6-1) over (3.6.14-1) ...
Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
Processing triggers for ureadahead (0.100.0-16) ...
Setting up rabbitmq-server (3.7.6-1) ...
Installing new version of config file /etc/init.d/rabbitmq-server ...
Installing new version of config file /etc/logrotate.d/rabbitmq-server ...
/var/lib/dpkg/info/rabbitmq-server.postinst: 8: /etc/profile.d/travis-haskell.sh: [[: not found
* Starting message broker rabbitmq-server * FAILED - check /var/log/rabbitmq/startup_\{log, _err\}
fail]
invoke-rc.d: initscript rabbitmq-server, action "start" failed.
dpkg: error processing package rabbitmq-server (--configure):
subprocess installed post-installation script returned error exit status 1
Processing triggers for ureadahead (0.100.0-16) ...
Errors were encountered while processing:
rabbitmq-server
E: Sub-process /usr/bin/dpkg returned an error code (1)
* Starting message broker rabbitmq-server
* FAILED - check /var/log/rabbitmq/startup_\{log, _err\}
...fail!
See C-QUESTION-MARK in the API Guidelines:
Like it or not, example code is often copied verbatim by users. Unwrapping an error should be a conscious decision that the user needs to make.
Celery can use the setproctitle
package to rename a child process, making it easier to identify the task being executed by a worker child process when visualizing them in ps
or top
for example.
setproctitle
is based on code written by the PostgreSQL project: https://doxygen.postgresql.org/ps__status_8c_source.html
It could be interesting to port this code to Rust and integrate its functionality to batch
.
We should document:
This might reduce duplication and remove some string-y API.
let exchanges = vec![exchange("default.exchange")];
let queues = vec![
queue("default.queue")
.task::<SayHello>("default.exchange")
];
WorkerBuilder::new()
.exchanges(exchanges)
.queues(queues)
.build();
Following the API docs, it might not be obvious that Exchange
is the result of ExchangeBuilder::build()
and that Queue
is the result of QueueBuilder::build()
.
A simple solution for now would be to remove the new
constructors on ExchangeBuilder
& QueueBuilder
and replace them by a builder
static function on Exchange
& Queue
respectively.
RabbitMQ natively supports priority queues, adding support in batch
should be rather straightforward.
Probably at least Chain
and Group
, and be able to compose these primitives.
e.g when a task execution fails, we should provide a simple hook for people to publish the error to their favourite error reporting service (Sentry, Raygun, Rollbar, Bugsnag, etc.). Integrations should not be tied to the batch crate.
Travis recently announced support for Windows on their CI platform. Might be interesting to have all CI on one service ?
Even if the library should be able to work correctly on Windows, no tests were made and the CI doesn't run the tests on a Windows system.
Depends on #13.
Add a web dashboard, allowing administrators to:
The dashboard will be made available through middlewares for web frameworks, allowing for maximum flexibility.
Since the task name refactor (#45), users of this library are required to specify lazy-static as a direct dependency of their project, even though they never have to explicitly use it.
A way of having the same behavior without forcing users to install lazy-static should be found.
Currently, the Worker
only executes jobs one by one. It should be possible to spawn as many jobs as available core on the current machine, greatly improving the performance.
Create a Scheduler
object, able to schedule tasks execution.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.