unixjunkie / cpmlib

Classification and Regression Performance Metrics library

cpmlib

Classification and Regression Performance Metrics library

cpmlib's Issues

shuffle_then_cut and shuffle_then_nfolds

Generic versions of those two functions could come in a Utils module.

let shuffle_then_cut seed p train_fn =
  match Utls.lines_of_file train_fn with
  | [] | [_] -> assert(false) (* no lines or header line only?! *)
  | (csv_header :: csv_payload) ->
    let rng = BatRandom.State.make [|seed|] in
    let rand_lines = L.shuffle ~state:rng csv_payload in
    let train, test = Utls.train_test_split p rand_lines in
    train_test_dump csv_header train test

let shuffle_then_nfolds seed n train_fn =
  match Utls.lines_of_file train_fn with
  | [] | [_] -> assert(false) (* no lines or header line only?! *)
  | (csv_header :: csv_payload) ->
    let rng = BatRandom.State.make [|seed|] in
    let rand_lines = L.shuffle ~state:rng csv_payload in
    let train_tests = Utls.cv_folds n rand_lines in
    L.rev_map (fun (x, y) -> train_test_dump csv_header x y) train_tests

add MAE

add logAUC

cf. http://wiki.bkslab.org/index.php/LogAUC

update opam package

implement precision-recall AUC

we already have ROC AUC but...

add R^2

add MCC for classification models

add rmse

Lorenz curve and Gini coefficient

Can be used to compare the ranking power of methods.

Kappa coefficient?

re-implement Platt scaling

Currently there is a hack using gnuplot.
If we had an OCaml implementation of Platt scaling, this would remove the dependency to gnuplot.

implement REC

Regression Error Characteristic Curve

Bi, J. and Bennett, K.P., 2003. Regression error characteristic curves. In Proceedings of the 20th international conference on machine learning (ICML-03) (pp. 43-50).

logAUC

 $LogAUC_\lambda=\frac{\sum_{i}^{where~x_i\ge\lambda} (\log_{10} x_{i+1} - \log_{10} x_i)(\frac{y_{i+1}+y_i}{2})}{\log_{10}\frac{1}{\lambda}}$

add Initial Enhancement

RIE = S/<S>
S = sum_over_i (exp (-rank(active_i) / a))
<S> = average S for 1000 trials where molecules (actives and inactives) are randomly sorted.
a << N.
for example, with a of 50, S almost vanishes after 300.
This is a kind of early EF measure.

replace the TopKeeper module

by the one from RanKers.
Write a test suite for it.

I need AUC for an array of score labels

instead of a list

new release in opam v5.0.0

after tests

also, it will be useful to version constrained to the previous major
version software that were already using this library

minor release

new release in opam

cf. latest tag

do a bugfix release in opam

fast_enrichment_factor is missing

release v8.0.0 in opam

add Net Reclassification Index (NRI) for two sorted list of score-labels

NRI = p(up/event) - p(down/event) + p(down/nonevent) -p(up/nonevent)

For the layman: actives should go up, inactives should go down.
If they do, bonus points. If they don't, malus points.

This allows to compare two classifiers based on their ranking performance; not weighted more
towards early recovery.