tolerance / tolerance

Fault tolerance library and micro-services helpers

Tolerance

Tolerance is a PHP library that provides fault tolerance and microservices related tools.

Getting started

• Documentation

License

Contributors

• Samuel ROZE @samuelroze
• Other awesome developers

tolerance's Issues

Circuit breaker pattern to prevent self-killing

Stackphp Middleware bridge

Are you interested by a stackphp bridge? For non symfony MS, we need to achieve the same logic as event listener do in symfony bridge to enrich / trigger some stuff before and after the business logic.

I want to use the middleware way via stackphp to handle that.

Placeholder responses in case of exception

For non-critical feature even if the operation failed we want to be able to display something to the user. So what we can do is to have a placeholder response, such as an empty image URL, or an empty list, etc...

This operation runner would decorate an existing one and return something that is compatible with the expected answer.

v0.4.1 requires php 7

In this commit, a float type hint has been added. But the composer.json files doesn't specify that php 7 is required.

So when I ran composer update on my project running on php 5.6, tolerance has been updated (0.4.0 => 0.4.1), and I get this fatal error: Fatal error: Default value for parameters with a class type hint can only be NULL in /var/www/html/vendor/tolerance/tolerance/src/Tolerance/Waiter/ExponentialBackOff.php on line 41

So, could you remove the php 7 syntax? Or specify a php 7 requirement in composer.json?
Thanks.

Rename it to "Tolerance"

Add a random factor parameter to ExponentialBackOff

Currently the ExponentialBackOff constructor is the following:

Tolerance/src/Tolerance/Waiter/ExponentialBackOff.php

Line 41 in 6242d43

public function __construct(Waiter $waiter, $initialExponent, $step = 1.0)

I suggest adding a new parameter drift at the end with a default value of 0.0:

public function __construct(Waiter $waiter, $initialExponent, $step = 1.0, $stepRandomFactor = 0.0)

What is this value?

The goal is to add some randomness to the waiter next value. This is useful for concurrent calls to avoid them to be too much massively synchronized.

How?

$stepRandomFactor should be a value between 0.0 and 1.0 that will behave like in this code for example:

// Template method that computes and returns the next backoff delay in
// milliseconds.
BackoffStrategy.prototype.next = function() {
    var backoffDelay = this.next_();
    var randomisationMultiple = 1 + Math.random() * this.randomisationFactor_;
    var randomizedDelay = Math.round(backoffDelay * randomisationMultiple);
    return randomizedDelay;
};

Maybe it should even be more useful to have this argument moved directly into the waiter?

Symfony integration

Creates operation runners from a YML configuration.

An operation should know about its state

• Did I started?
• Did I success?
• Did I failed?

Add Zipkin adapter for MessageProfile

At the moment, the MessageProfile component only pushes something to Neo4j. Even if this is very useful, we need to be able to support Zipkin and OpenTracing.

ZipKin are looking for PHP implementation, and Tolerance is probably one very good candidate.
openzipkin/zipkin#1330

Tracer: support opentracing

opentracing is kind of emerging interface for tracers, thus not only zipkin would be supported if Tolerance adopt it.

About the bad news: no official php lib has been released yet. There's a try with some unofficial but very incomplete implementation.

Report & Metrics

We can decorates an operation runner to collect/send metrics about the success/failure/count/details of these operations.

Enable A/B and validation testing via operation runners

Operation runners can be an interesting way of actually setting up candidate releases and validating these releases against new ones.

GitHub's experiment about merge release is interesting and might be too complicated to setup some times, FaultTolerance needs to answer that:
http://githubengineering.com/move-fast/

The RabbitMq rate is incorrect

It should probably be split into these two metrics:

• backing_queue_status.avg_ingress_rate
• backing_queue_status.avg_egress_rate

Missing Metrics & Tracer documentation

Php-Http integration

Support ZMQ messaging

Hello,

There is a plan to support ZMQ messaging?

Move a request identifier across services

• Guzzle middleware

Terminates the buffered operations

We should be able to terminate buffered operations, by using a terminate method on the operation buffer.

Case of synchronous operations

Provided we have the following $runner object:

$callbackRunner = new CallbackOperationRunner();
$waitStrategy = new CountLimited(
    new ExponentialBackOff(
        new SleepWaiter(),
        1
    ),
    3
);

$runner = new RetryOperationRunner(
    $callbackRunner,
    $waitStrategy
);

When we do:

$operation = new Callback(function() {
    return $this->client->locate('80.87.18.226');
});
$this->runner->run($operation);

We can then access to the result via $operation->getResult() which is a CallbackOperation method and not one of the Operation interface.

The issue here is that $this->runner->run($operation); is incoherent with the OperationRunner::run(Operation): Operation signature that says it returns an Operation object. The signature implies that the runner returns a new Operation instance, potentially different from the was passed as an input. It makes sense if we pass an operation meant to be run asynchronously for example. But in the case of a CallbackOperation that we want to run synchronously to get a result, this is not ideal.

So I would rather have two different runner interfaces, one for synchronous operations and in which case the run() signature would be OperationRunner::run(Operation): void, i.e. the operation passed is the same as the one returned. And a different one for asynchronous operations with the same signature as the actual runner, for which the returned operation may be a different as the one passed to the runner.

Run Operation based on the result of previous operation

Hi @sroze ,

I have a use case where I need to insert an element in a database based on a previous condition, for example :

I have an API endpoint that receives events from a website (GoogleAnalyticsTagManager events).
For the same page load I can receive multiple events with roughly the same data (for example I can receive events when the user is scrolling the page with additional data likes elements shown on his page area).

This endpoint will just dispatch the event data to RabbitMQ.

On the other side I have a RMQ worker, which will read and process the event messages. In the current scenario, I can have 5 or 6 messages in sequence that will represent actions on the same website page. I am only interested in inserting 1 click event in my database.

My current application checks whether the last clicked page id (in the database) is the same as the event data page id, if no, then I write the new click into the database. We are not using Tolerance right now but we are investigating it a lot for a data ingestion platform.

So, the current need is to only perform the ClickInsert operation only if the CheckLastVisitedPropertyIsNotSameAsCurrent operation returns true.

I'm aware Tolerance doesn't have such OperationRunners right now and I would like to know your advise about this need and your current vision of Tolerance's architecture about this need as we would definitely be happy to contribute back the created runners if you find them generic enough.

I see multiple ways of achieving it, one of them is based on other needs which would need another ticket on this topic.

Plan A)

This would basically be a kind of BufferedOperationRunner where operations have to be run in sequence, however you would be able to pass PreflightOperations that should all return true in order to run the normal Operations.

Plan B)

There could be a ChainedOperationRunner (actually your current ChainOperationRunner doesn't have the meaning I have in mind for chains) where every operation would have to return another Operation, something like :

$nextOperation = $this->operation;
do {
  $nextOperation = $nextOperation->run();
} while ($nextOperation instanceof Operation);

Do you have other ideas in mind ?

Thanks for your thoughts on this and thanks for making this library.

Redis storage for the rate counter

PHP Microservice Platform aka Torelance Platform

I open an new issue related to the discussion in #52 PR.

I agree with you about the php ecosystem, we lack this kind of initiative as there is a lot of how to do an api rest application but we don't have anything to make them talk at scale.

It could be interesting to have a sort of "microservice platform" that can leverage all features from symfony and will be based on small microservice components if you love laravel or code igniter :)

PHP Microservice Platform (microservice chassis framework)

• Endpoint (they all use RPC messaging pattern, i don't know if it's the best solution)
• Authentication system
• log(monolog multiple output)
• monitoring/health check
• metrics
• circuit breaker
• rate limiter (you already have one)
• transport (you already have one)
• service discovery (client side discovery or server side discovery) use it with etcd or another key/value store.
• router/load balancing

I'm not a master in microservices, i just read some articles a long time ago :).

What do you think, this looks completely wrong ?

Linear waiter documentation

The documentation is missing for the linear waiter.

Support Google Stackdriver Traces

https://cloud.google.com/trace/docs/faq

Add a request profile component

Goals

• Being able to trace a request profile (response time, from where, extra metadata...) (array)
• Being able to store these request profiles

  Google Cloud Datastore
  Elastic Search

TimeRate's getTicks() method incorrect behaviour

In order to return number of seconds, division must be used instead of multiplication.
Currently new TimeRate(0.5, TimeRate::PER_SECOND) is not equal to new TimeRate(30, TimeRate::PER_MINUTE).