tdhock / aum

Area Under the Minimum (AUM) of False Positives and Negatives

tests
coverage

This R package provides an efficient C++ implementation of the AUM, which can be used as a surrogate loss for optimizing Area Under the ROC Curve (AUC) in supervised binary classification and changepoint detection problems.

Installation

install.packages("aum")
## OR:
if(!requireNamespace("remotes"))install.packages("remotes")
remotes::install_github("tdhock/aum")

Usage

Converting binary labels to aum_diffs

The code below creates an aum_diffs data table which represents error functions for two labeled examples in binary classification.

> (bin.diffs <- aum::aum_diffs_binary(c(0,1)))
   example pred fp_diff fn_diff
1:       0    0       1       0
2:       1    0       0      -1

• The first row above means there is a false positive difference of 1 at a predicted value of 0. This is the error function for each example with a negative label in binary classification (no error if predicted value less than 0, changes up to 1 false positive for larger predicted values).
• The second row means there is a false negative difference of -1 at a predicted value of 0. This is the error function for each example with a positive label in binary classification (1 false negative if predicted value less than 0, changes down to no errors for larger predicted values).

Computing AUM from aum_diffs table and prediction vector

Next we assume predicted values of 0 for both examples, and then compute Area Under the Minimum (AUM) of False Positives and False Negatives and its directional derivatives.

> aum::aum(bin.diffs, c(0,0))
$aum
[1] 0

$derivative_mat
     [,1] [,2]
[1,]    0    1
[2,]   -1    0

The result above is a named list with two elements.

• aum is a numeric value giving the AUM for the specified error functions and predicted values.
• derivative_mat is a matrix of directional derivatives, one row for each example (first column for left directional derivative, second column for right). In the example above we can see that decreasing the first prediction (entry 1,1) and/or increasing the second prediction (entry 2,2) results in no change to AUM. Since the right directional derivative of the first example is positive (entry 1,2), that implies an increased prediction would result in an increased AUM. Similarly the left directional derivative for the second example is negative (entry 2,1), indicating that a decreased prediction would result in an increased AUM.

Changepoint detection

See ?aum::aum_diffs_penalty for documentation about how to compute the AUM for supervised penalty learning in changepoint detection problems.

Line search

An exact line search can be computed using time which is log-linear in the number of step sizes, see ?aum::aum_line_search for a single line search, and ?aum::aum_linear_model_cv for using the line search in each step of gradient descent when learning a linear model.

Related Work

• penaltyLearning::ROChange provides an alternative implementation of AUM and its directional derivatives in R data.table code (slower, also includes ROC curve computation).
• The speed comparison vignette has an alternative implementation of AUM and its directional derivatives in base R code.
• max-generalized-auc repo provides code for making the figures in https://arxiv.org/abs/2107.01285
• A python/torch implementation of AUM for binary classification.

aum's Issues

vignette about asymptotic complexity of line search

after going through a few iterations of the first for loop in https://github.com/tdhock/aum/blob/main/vignettes/line-search.Rmd
I executed this code

  N.seq <- as.integer(10^seq(2,log10(max(diff.list$subtrain$example)),l=10))
  N.seq <- as.integer(10^seq(log10(100),log10(1600),l=10))
  atime.list <- atime::atime(
    N=N.seq,
    setup={
      maxIterations <- N*(N-1)/2
      X.subtrain <- X.keep[index.list$subtrain,]
      X.sub <- X.subtrain[1:N,]
      diff.sub <- diff.list$subtrain[example %in% seq(0,N-1)]
    },
    result=TRUE,
    seconds.limit=1,
    times=1,
    aum_line_search={
      nb.weight.search <- aum::aum_line_search(
        diff.sub,
        maxIterations=maxIterations,
        feature.mat=X.sub,
        weight.vec=weight.vec)
      list(
        total.iterations=nrow(nb.weight.search$line_search_result),
        iterations.to.min=nb.weight.search$line_search_result[,which.min(aum)])
    })
  atime.list$measurements[, total.iterations := sapply(
    result, function(L)L$total.iterations)]
  atime.list$measurements[, iterations.to.min := sapply(
    result, function(L)L$iterations.to.min)]

  best.list <- atime::references_best(atime.list, unit.col.vec=c("total.iterations","iterations.to.min"))
best.ref <- best.list$ref[each.sign.rank==1]
  library(ggplot2)
  gg <- ggplot()+
    theme_bw()+
    facet_grid(unit ~ ., scales="free")+
    geom_line(aes(
      N, empirical, color=expr.name),
      data=best.list$meas)+
    scale_x_log10()+
    scale_y_log10("median line, min/max band")
  gg.show <- gg+
     directlabels::geom_dl(aes(
       N, empirical, color=expr.name, label=expr.name),
       method="right.polygons",
       data=best.list$meas)+
     theme(legend.position="none")+
     coord_cartesian(xlim=c(min(N.seq),max(best.list$meas$N)*5))
  gg.ref <- gg.show+
    geom_line(aes(
      N, reference, group=paste(fun.name, expr.name)),
      color="grey",
      data=best.ref)
    gg.ref+
      directlabels::geom_dl(aes(
        N, reference,
        label.group=paste(fun.name, expr.name),
        label=fun.name),
        data=best.ref,
        color="grey",
        method="left.polygons")

and I got this plot

which suggests that the number of iterations in line search is quadratic, and so is the number of iterations to get to the min.
@phase Would be nice to have a vignette that explores this more systematically,

• do this asymptotic analysis for every step of gradient descent on full data?
• do gradient descent on different data sizes, and keep track of these metrics at each step?

compute AUC in addition to AUM in line search C++ code

During the algorithm which computes the update for AUM at every iteration of the line search, we could probably compute an AUC update for free (constant factors), and that would be interesting (choose step size that maximizes AUC, rather than minimizes AUM).
@phase
Need to derive the equations that tell us how to code those updates.