Exceptions within test callbacks cause unexpected behavior about fh-health

[evanshortiss]

Hey @pete-thompson, could you let me know your scenario. Happy to help understand this and address it.

It sounds like your test might be encountering an "uncaughtException". The library could attempt to catch this, but if the operation that throws the error is in an asynchronous function e.g a http.request then this module might not be able to catch it, for example:

function asyncOpExample () {
  setTimeout(function () {
    throw new Error('the catch block below cannot catch this');
  });
}

try {
  asyncOpExample()
} catch (e) {
  console.log(e.stack);
}

In such a scenario you'll need to wrap the offending piece of code in your asynchronous function and return the error accordingly, e.g:

function asyncOpExample (callback) {
  setTimeout(function () {
    try {
      throw new Error('this will be caught and passed safely back to the callback');
    } catch (e) {
      callback(e.toString(), null);
    }
  });
}

asyncOpExample(callback)

[pete-thompson]

Thanks Evan. I've now had a good read about exceptions and asynchronous Java - all rather ugly! The specific problem I had was that the health check function had an error in it that only became obvious when the health check failed. The code was making a connection to an Oracle database and calling the health method to pass an error if the connection failed. But the code then dropped through the IF statement without returning and therefore ran the code that attempted to run a SQL statement on the failed connection.

So - a philosophical question... Do you want the health check library to defend against poorly written tests that fail by exceptions, or do you make it the responsibility of the health test developer to thoroughly debug their code? As a developer I'd probably feel happy with taking the responsibility to test my code, but as someone who's normally an architect/lead I usually like defensive solutions that don't rely on all my developers being superstars :).

I'm guessing that the only way to get defensive around the health tests would be to use Node's Domain API? The interesting/sad thing about Domain is that it appears to be deprecated in the latest version without a replacement being specified - so they're basically saying that if you need it you need to use it, but expect to be forced into changing your code soon...

[evanshortiss]

I've now had a good read about exceptions and asynchronous Java(script?) - all rather ugly!

@pete-thompson, maybe. It's certainly a mental shift from more traditional environments, and can take some time to adjust!

@pete-thompson, this module could wrap tests in a try/catch in an attempt to capture synchronous errors. I'm not opposed to that idea (and will happily add it), but ultimately, the best solution would be to unit test your health checks to ensure each conditional statement is executed, here's an example of how you can achieve that without hitting a real system. It relies on proxyquire and sinon to achieve mocking out of dependencies.

Apologies if the code below has errors, I have not run it.

// health/critical/get-google.js
var request = require('request')
module.exports = function (callback) {
  request.get('http://www.google.com', function (err, res, body) {
    if (err) {
      callback('GET to get google.com failed');
    } else if (res.statusCode !== 200) {
      callback('GET to google.com returned status ' + res.statusCode.toString());
    } else {
      callback(null, 'GET to google.com was successful');
    }
  });
}

// health/critical/get-google.test.js
var proxyquire = require('proxyquire');
var sinon = require('sinon');
var expect = require('chai').expect;

describe('google health test', function () {
  var requestStub;
  var healthCheck;

  // Called before each "it" to ensure clean state for each test case
  beforeEach(function () {
    requestStub = sinon.stub();
    healthCheck = proxyquire('./get-google.js', {
      // Stub out request module
      request: requestStub
    });
  });

  it('should return error', function (done) {
    requestStub.yields(new Error('fake http error'));

    healthCheck(function (err, msg) {
      expect(err).to.equal('GET to get google.com failed');
      done();
    });
  });

  it('should return status error', function () {
    requestStub.yields(null, {statusCode: 404}, null);

    healthCheck(function (err, msg) {
      expect(err).to.equal('GET to google.com returned status 404');
      done();
    });
  });

  it('should return success', function () {
    requestStub.yields(null, {statusCode: 200}, 'ok');

    healthCheck(function (err, msg) {
      expect(err).to.equal(null);
      done();
    });
  });
});

Usage:

// health/main.js
var health = require('fh-health');
health.addTest('GET google', require('health/critical/get-google.js'));

You could also write a loader to automatically add all tests in a folder like so (this is what I tend to do):

// health/main.js
var health = require('fh-health');
var fs = require('fs');
var path = require('path');

fs.readdirSync(path.join(__dirname, './critical')).forEach(function (filename) {
  // Auto require the critical test
  health.addCriticalTest(filename, require(path.join(__dirname, './critical', filename)))
});

[pete-thompson]

Thanks Evan. It's not my health checks I'm worried about - I'm a perfect programmer and always unit test everything I write ;). Thanks for the info on unit testing though - I've spent a fair amount of time this past couple of weeks learning all about the joys of Mocha, Proxyquire, Supertest etc.

The specific problem we had that triggered my request was in some code that Todd Wardzinski's been building for us to make it easy to configure some health checks (e.g. ping tests) without the need for coding. When I find some time I'm going to see if it makes sense to send you some pull requests to add capabilities. So blame Todd for the problem :). Anyway, goes to show that even with good developers and unit testing you can still get unexpected crashes in the real environment even in code that looks relatively simple...

[evanshortiss]

@pete-thompson absolutely, these things can still happen. Feel free to open a PR or Issue in the future if necessary.