awolden / brakes Goto Github PK

Brakes version: 3.0.1

After having updated to Node v12.16.0 globalStatsStream does not track the instances correctly. I have two brakes instantiated but only the first brake does notify via stream for only one first event. Example:

        const brake1 = new Brakes(promiseCall, {
            name: "MyDefault",
            bucketSpan: 5000,
            bucketNum: 3,
            waitThreshold: 5,
            timeout: 200,
            threshold: 0.5,
            circuitDuration: 30000,
        });

        const brake2 = new Brakes(promiseCall, {
            name: "XXXX",
            bucketSpan: 5000,
            bucketNum: 3,
            waitThreshold: 5,
            timeout: 200,
            threshold: 0.5,
            circuitDuration: 30000,
        });

        Brakes.getGlobalStats().getHystrixStream().on('data', (stats) => {
            console.log('received global stats ->', stats);
        });

With Node v12.15.0 it is working as expected. Alle events are propagated via the stats stream. Any known issues about this?

Hi! 👋

I'm planning to use Brakes in a TypeScript project, and noticed that there are no typings available in the repo or via DefinitelyTyped. Before spending a lot of time, I have two questions.

First: Would you be interested in migrating this repo to TypeScript?

I naïvely cloned and started typing the whole project, in the hopes that I would be able to annotate a few variables and get type definitions for free. A couple of hours later, I realised the error of my ways, and gave up. The almost-functional result is available at https://github.com/awolden/brakes/compare/master...theneva:typescript?expand=1

If no, I won't spend any more time on typing the whole thing. For now, I've resigned to manually writing a type definition file for the Brakes class.

Second question: Would you accept the module type definition file directly into this repo (similar to what is done in for example https://github.com/sindresorhus/copy-text-to-clipboard), or should I go via DefinitelyTyped to publish it?

CircuitBrokenError is taking threshold as failure rate, however it is success rate.
The error message is wrong in that way.

I'm using the following code:

const Brakes = require('brakes');
const fallback = require('../lib/circuit-breaker-fallback');

const breakerOpts = {
      timeout: 150,
      fallback,
};

const promisifiedPUT = () => {
  return new Promise((resolve, reject) => {
    serviceCall(optsObj, (err, res) => {
      if (err) {
          reject(err);
      } else {
          resolve(res);
      }
    });
  });
};

const brake = new Brakes(breakerOpts);
const slaveCircuit = brake.slaveCircuit(promisifiedPUT);

slaveCircuit.exec()
  .then(handleSuccess)
  .catch(handleFailure);
...

I get the following error:

/path/node_modules/brakes/lib/Brakes.js:174
      throw new Error(consts.NO_FUNCTION);
      ^

Error: This instance of brakes has no master circuit. You must use a slave circuit.
    at Brakes.fallback (/path/node_modules/brakes/lib/Brakes.js:174:13)
    at new Brakes (/path/node_modules/brakes/lib/Brakes.js:71:12)
    at Object.<anonymous> (/path/server/models/greeting.js:17:23)

Following the trail, I get to here and here. This suggests to me that the constructor requires a function and a settings object, but the examples only show a setting object being passed in:

const brake = new Brakes({
  statInterval: 2500,
  threshold: 0.5,
  circuitDuration: 15000,
  timeout: 250
});

const greetingBrake = new Brakes(() => {}, breakerOpts); seems to work, but I'm not sure that's how I'm supposed to be using the library.

Any thoughts? Suggestions?

Hi,

I've implemented brakes in a Lambda function in AWS, and have seen a gradual performance degradation over the course of about an hour.

We can observe the lambda running slower and slower up until a point where it takes about 10-30 seconds for it to execute, whereas on a fresh startup, requests are processed within a few milliseconds.

Memory consumption appears to increase linearly.

Our usage looks like this:

class Foo extends Service {

    init(stuff) {
        return super.init(stuff)
        .then(res => {
            this.getFooCB = new Brakes(
                 this.getFoo.bind(this), // Bind the scope, or won't have access to class vars
                 {
                     circuitDuration: 15000,
                     timeout: 10000,
                     waitThreshold:0 // After how many requests should we check?
                 });            
        });
    }

   getFoo(stuff) {
      // do some async thing
   }

  useFoo() {
    return this.getFooCB.exec(stuff).then(res => {...});
  }

Found an edge case where the circuit can be opened, but brakes is showing request stats that indicate that it shouldn't have opened (because failure percentage is below threshold).

I think what might be happening is:

1. With several requests pending, one comes back as a failure.
2. Brakes sees this puts the server over the user-specified failure threshold (e.g. 50%) and opens the circuit, reporting a failure rate of 51%.
3. The other pending requests return successfully after the circuit has opened, which alters the request stats (bringing failure percentage down to 48%).
4. Brakes rejects all new requests to the server, but reports that it's doing so because the failure rate (48%) is higher than the failure threshold (50%).

This isn't a case of brakes doing the wrong thing. It's just a rare case (which will probably only happen at low numbers of requests) where the way brakes reports things could be confusing.

I'm using version 2.5.1 and I try to view the stream from the hystrix dashboard but it seems to me that the requestCount never goes down even though the no one calls the Brakes "command":

The data that is reported by the hystrix stream looks like this:

data: {"type":"HystrixCommand","name":"MyName","group":"MyGroup","currentTime":1487861651232,"isCircuitBreakerOpen":false,"errorPercentage":0,"errorCount":0,"requestCount":53,"rollingCountBadRequests":0,"rollingCountCollapsedRequests":0,"rollingCountExceptionsThrown":0,"rollingCountFailure":0,"rollingCountFallbackFailure":0,"rollingCountFallbackRejection":0,"rollingCountFallbackSuccess":0,"rollingCountResponsesFromCache":0,"rollingCountSemaphoreRejected":0,"rollingCountShortCircuited":0,"rollingCountSuccess":53,"rollingCountThreadPoolRejected":0,"rollingCountTimeout":0,"currentConcurrentExecutionCount":0,"latencyExecute_mean":758,"latencyExecute":{"0":600,"25":664,"50":724,"75":868,"90":906,"95":912,"99":951,"100":951,"99.5":951},"latencyTotal_mean":15,"latencyTotal":{"0":600,"25":664,"50":724,"75":868,"90":906,"95":912,"99":951,"100":951,"99.5":951},"propertyValue_circuitBreakerRequestVolumeThreshold":100,"propertyValue_circuitBreakerSleepWindowInMilliseconds":5000,"propertyValue_circuitBreakerErrorThresholdPercentage":0.5,"propertyValue_circuitBreakerForceOpen":false,"propertyValue_circuitBreakerForceClosed":false,"propertyValue_circuitBreakerEnabled":true,"propertyValue_executionIsolationStrategy":"THREAD","propertyValue_executionIsolationThreadTimeoutInMilliseconds":800,"propertyValue_executionIsolationThreadInterruptOnTimeout":true,"propertyValue_executionIsolationThreadPoolKeyOverride":null,"propertyValue_executionIsolationSemaphoreMaxConcurrentRequests":20,"propertyValue_fallbackIsolationSemaphoreMaxConcurrentRequests":10,"propertyValue_metricsRollingStatisticalWindowInMilliseconds":10000,"propertyValue_requestCacheEnabled":false,"propertyValue_requestLogEnabled":false,"reportingHosts":1}

My Brakes "command" looks like this (defined "globally" in the module):

const myHystrixCommand = new Brakes(queryBitBucket, {
    statInterval: 1000,
    threshold: 0.5,
    circuitDuration: 5000,
    timeout: 5000,
    name: 'MyName',
    group: 'MyGroup',
    isPromise: true
});

Am I doing something wrong or why is the requestCount always showing 53? (which is the total number of calls I've made).

Hi.
So your circuit breaker has the following flow:

closed -> errorsOverThreshold -> open ->

• if has healthcheck: _setHealthInterval (https://github.com/awolden/brakes/blob/master/lib/Brakes.js#L122), if ok, close circuit.
• if no healthcheck, use a timer. (https://github.com/awolden/brakes/blob/master/lib/Brakes.js#L150), if T time passed, close circuit.

This could potentially do more harm than good.

I'd expect the circuit breaker to NOT have a timer (the healthcheck is a nice addition), and let a request come in once in a while to check if the circuit can potentially be closed, if so, let a bit more requests come in and so on. This should be throttled. (hystrixjs does it,albeit not ideal but better: https://bitbucket.org/igor_sechyn/hystrixjs/src/a8e30f520660a696362b3605d959794883c5368b/lib/command/CircuitBreaker.js?at=master&fileviewer=file-view-default#CircuitBreaker.js-53)

I think you should reconsider that design on your end.

Currently a rolling count is kept, but not a cumulative one.

It would be great of both were available.

https://github.com/Netflix/Hystrix/wiki/Metrics-and-Monitoring#cumulative-and-rolling-event-counts

https://github.com/awolden/brakes/blob/master/lib/Brakes.js#L248

This function should be renamed to remove terminology related to slavery and white supremacy.

Probably it is my lack in understanding the circuit brake pattern, but let's take examples/healthCheck-example.js. What happen when the circuit is opened?

Sending a fat arrow function without parens (one arg only) into brakes causes an exception: TypeError: Cannot read property '1' of null

It appears to be breaking on the regex in getFnArgs in lib/utils.js:

/^[function\s]?.*?\(([^)]*)\)/

When a circuit breaker is opened the module should return a more meaninful error object than it currently does. Right now, it just returns a string. It should return an actual error that includes a snapshot of the current stats.

When a call is completed the timer is not cleared. Under high load this leads to a large amount of extra processing and the promise is rejected on timeout (after already being resolved)

We'd like to have a sliding window being checked for percentage of failures, instead of a definite number of requests. Levee has this, for instance.
Any particular reason why this is not implemented here? Would you be opposed to a PR adding it?

E.g. There is more than 25% failures the last second, open the circuit.

When using fetch to do remote http calls we will get a promise back. If the service response is a 5xx satus this is still considered a "resolving" promise. When this happens i would brakes to consider this as a failure. Today this means I have convert the resolving promise in to a rejecting one, which feels a bit awkward.

If there where an isSuccess option I could easily verify that the response code is not in the 500-range.

When a circuit is created you should be able to pass a name and have that state stored internally within the circuit breaker.

Why in the examples/hystrix-example.js there are two Hystrix servers created? One is not enough?

Hi.
I see that there is not support for cluster mode. Could this be done with minimum overhead (aggregate before sending to master)?
Because stats are collected on workers, and master will expose the SSE stream, I'm unsure this works out of the box.

As well as context support. for example, I'd like to execute a method from my class. Currently, I see that you're using .apply(this (where this is Circuit.js), args);. Is it possible to configure this per circuit?

thanks!

2.6.0 introduced "Add the breaker name to circuit rejections".

This breaks existing installations where err.message is not a string:

"[Breaker: Basket] [object Object]" is not helpful. Please either check if err.message is a String, or make adding the circuit-name optional.

The default setting for waitThreshold is currently set to 100 which is way too high IMO.
Imagine a scenario in which the backend system becomes unresponsive and stays broken for a few hours. Assuming you set the circuitDuration to 30 seconds, you would dish out 100 failing requests before opening the circuit breaker again - every 30 seconds. Don't you think that 100 requests is a bit too much for a system which is already struggling?

Of course it is up to the user to tweak the waitThreshold according to their needs but speaking of myself personally, I was very confused when I tried the library in my project and the circuit breaker wouldn't open immediately even though the failure rate was at 100%.

I would think that a waitThreshold of 5 to 10 would be much more appropriate.

enabling isFailure setting by calling a function (the same way available for healthCheck and fallback) would make the structure more consistent.
Thanks!

Is there a plan to transpile code for supporting browser usage or maybe atleast providing support in some areas? Lots of times node_modules are excluded in webpack settings and code that isn't transpiled (static or =>) end up throwing errors in isomorphic land...

const Brakes = require('brakes');

const prom = (foo) => {
    return new Promise((resolve, reject) => {
        resolve(foo);
    });
}

const brake = new Brakes(prom);

brake.exec('foo').then(result => {
    console.log(result)
}, err => {
    console.error(err);
});

Causes the following error:

./node_modules/brakes/node_modules/promisify-node/utils/args.js:9
  var args = func.toString().match(/function\s.*?\(([^)]*)\)/)[1];
                                                              ^

TypeError: Cannot read property '1' of null
    at module.exports (./node_modules/brakes/node_modules/promisify-node/utils/args.js:9:63)
    at processExports (./node_modules/brakes/node_modules/promisify-node/index.js:61:29)
    at module.exports (./node_modules/brakes/node_modules/promisify-node/index.js:164:10)
    at new Brakes (./node_modules/brakes/lib/Brakes.js:32:25)
    at Object.<anonymous> (./index.js:10:15)
    at Module._compile (module.js:435:26)
    at Object.Module._extensions..js (module.js:442:10)
    at Module.load (module.js:356:32)
    at Function.Module._load (module.js:311:12)
    at Function.Module.runMain (module.js:467:10)

This seems to be caused by reflection that promisify-node is doing under the covers to detect meta data about the passed in function.

Hello!

I am implementing this circuit breaker for node.js and i noted something strange. In a normal flow for my, in this case, API call i always see that the fallback functions is being called, the service is working ok and the response is totally fine, but whenever i set a breakpoint in the fallback, it stops right there, even if the flow goes ok.
I am not sure why is it executing if there are not errors in the api call.

Thanks!

nvm use says 4.8.4 but the syntax used isn't being transpiled for that.. what is the last version of node that we are supposed to provide support for?

In short, I am struggling to understand how to minimize redundancy in the data/stats within the "snapshot" events.

My goal is to ensure that each "snapshot" contains only data which has been measured after the previous "snapshot." In other words, I do not want a "snapshot" to report on data which has already been reported on in a previous "snapshot."

My issue is that I cannot seem to properly assess which bucket or buckets are included in each "snapshot" event.

For example, if I configured the brakes instance options to have say:
{ bucketNum: 6, bucketSpan: 60000, statInterval: 6 * 60000 }, would each "snapshot" event report on all the data measured by all 6 buckets in the last 6 minutes? Or would it report on only the data measured in the very last bucket?

Again, my goal is to select the above options such that my "snapshot" stats do not report data that has already been included in a previous "snapshot," while also not missing any unreported data in the time elapsed since the last "snapshot."

Any insight into this would be very much appreciated, and thank you in advance!

Right now when a circuit is broken it registers it as being closed, and when a circuit is still viable it registers it as open. These are backwards. The terminology should be changed such that:

open: requests automatically fail and and circuit is tripped
closed: requests are properly flowing through to the request function passed

There is a example based aother library.It is more elegant.

Working with request-promise ;

e.g.

var rp = require('request-promise');
var Brakes = require('brakes');

var options = {
  uri: 'http://some-json-api.com/',
  json: true
};

var brake = new Brakes(request, { timeOut: 1000 });

return brake.exec(options)
  .then( response => {
    return processResponse(response);
  })
  .catch(e => {
    return processError(e.statusCode);
  });

request-promise wraps the normal request object, which is a function that has a callback parameter, and makes it return a promise. Wrapping it causes the behaviour to change because Circuit prefers async callback functions ; instead of the usual behaviour with these options which is for response to be an object representing the processed JSON response, and for errors to end up in .catch, all responses including errors end up in the .then() block because Circuit uses the default promise callback style instead of request-promise's Promise wrapping.

Not sure the best way to resolve this but maybe an option to force the Promise treatment instead of using the callback style? Not a big thing to write code to work around this (and ditch request-promise in the process, since we're not using it's features).

Using the default setting, we experienced growth in CPU usage and increasing performance degradation throughout the day. We were forced to add a cron job to bounce the service every 6 hours. When I removed brakes, CPU usage returned to normal.

In my code I need to update options for a brake already instantiated (to be more specific I need to update the timeoutvalue).

I'm using Brakes internals to do that:

brake._masterCircuit._opts = Object.assign( {}, brake._masterCircuit._opts, newOptions );

Could be a nice to have an updateOptions(opts) method attached to brake instance to have the same result without touching internals.

Hi

brakes modifies error messages coming from a downstream, my downstream service returns
{ message: 'We are unable to process your request, please try again later', status: 400 }, then brakes change the message to [Breaker: myservice] [object] [object]. I can't display that message to the user. How can I get the original error message?