wooparadog / doctor

Metric based in-memory circuit breaker for python, from zeus_core/health

doctor (split from zeus_core/health)

Health is described with current errors percentage, if the health status turns bad, actions like “refuse service” should be taken, mainly to protect our backend databases.

You must invoke on_* like methods of doctor.checker.HealthTester.metrics(doctor.metrics.Metrics) to record metrics, then HealthTester.is_healthy calculate api call status by thresholds, and HealthTester.test based the flowing policy to decide whether the current request can be passed.

Install

pip install doctor -i http://pypi.dev.elenet.me/eleme/eleme

Policy

Current detail policy to test health description:

• if current api is heavily under errors, disallow it to pass the test(), and further incoming requests should be refused ( in at least MIN_RECOVERY_TIME).
• if current api has recoveried from bad health, allow it to pass the test() gradually (via random.random() with priority).

Current errors threholds:

• Errors contains system errors and gevent timeouts.
• Threholds are percentages: errors / requests.
• Errors threholds are checked only if the current requests count is greater than THRESHOLD_REQUEST.

Health check interval:

• Calculated by METRICS_GRANULARITY * METRICS_ROLLINGSIZE, in seconds.

Settings

MIN_RECOVERY_TIME         min recovery time (in seconds)
MAX_RECOVERY_TIME         max recovery time (in seconds)
THRESHOLD_REQUEST         min requests to trigger a health check. (per INTERVAL)  # noqa
THRESHOLD_TIMEOUT         gevent timeout count threshold (per INTERVAL)
THRESHOLD_SYS_EXC         sys_exc count threshold (per INTERVAL)
THRESHOLD_UNKWN_EXC       unkwn_exc count threshold (per INTERVAL)

Examples

# callbacks, take *doctor.checker.APIHealthTestCtx* as arguments
def on_api_health_locked(result):
    pass
def on_api_health_unlocked(result):
    pass
def on_api_health_tested(result):
    pass
def on_api_health_tested_bad(result):
    pass
def on_api_health_tested_ok(result):
    pass

# you can custom the settings, see doctor/configs.py
configs = Configs()

# callbacks order matters.
tester = HealthTester(
    configs,
    on_api_health_locked,
    on_api_health_unlocked,
    on_api_health_tested,
    on_api_health_tested_bad,
    on_api_health_tested_ok,
)


def api_decorator(func):
    @functools.wraps(func)
    def _wrapper(service, *args, **kwargs):
        service_name, func_name = service.name, func.__name__
        if not tester.test(service_name, func_name):
            print('Oh! No!!!')
            return

        result = None
        try:
            result = func(service, *args, **kwargs)
        except UserError:
            tester.metrics.on_api_called_user_exc(service_name, func_name)
        except TimeoutError:
            tester.metrics.on_api_called_timeout(service_name, func_name)
        except SysError:
            tester.metrics.on_api_called_sys_exc(service_name, func_name)
        except Exception:
            tester.metrics.on_api_called_unkwn_exc(service_name, func_name)
        else:
            tester.metrics.on_api_called_ok(service_name, func_name)
        finally:
            tester.metrics.on_api_called(service_name, func_name)

        return result
    return _wrapper


@api_decorator
def api(service):
    client.connect(service.addr)

Ports

• Go

Authors

• @Damnever
• @xiangyu.wang
• @hit9