ishmaelbelghazi / bigoptim

Large Scale Machine learning Optimization through Stochastic Average Gradient

BigOptim – Large Scale Finite Sums Cost functions Optimization for R

Description

BigOptim is an R package that implements the Stochastic Average Gradient(SAG)[1] optimization method. For strongly convex problems, SAG achieves batch gradient descent convergence rates while keeping the iteration complexity of stochastic gradient descent. This allows for efficient training of machine learning algorithms with convex cost functions.

Setup

install.packages("devtools")
devtools::install_github("hadley/devtools")  ## Optional
devtools::install_github("IshmaelBelghazi/bigoptim")

Example: Fit with Linesearch

## Loading Data set
data(covtype.libsvm)
## Normalizing Columns and adding intercept
X <- cbind(rep(1, NROW(covtype.libsvm$X)), scale(covtype.libsvm$X))
y <- covtype.libsvm$y
y[y == 2] <- -1
## Setting seed
#set.seed(0)
## Setting up problem
maxiter <- NROW(X) * 10  ## 10 passes throught the dataset
lambda <- 1/NROW(X) 
sag_ls_fit <- sag_fit(X=X, y=y, lambda=lambda,
                      maxiter=maxiter, 
                      tol=1e-04, 
                      family="binomial", 
                      fit_alg="linesearch",
                      standardize=FALSE)
## Getting weights
weights <- coef(sag_ls_fit)
## Getting cost
cost <- get_cost(sag_ls_fit)

Example: Demo – Monitoring gradient norm

demo("monitoring_training")

Runtime comparison

Ran on intel i7 4710HQ 16G with intel MKL and compilers.

demo("run_times")

Dense dataset: Logistic regression on covertype

Logistic Regression on Covertype – 581012 sample points, 55 variables

	constant	linesearch	adaptive	glmnet
Cost at optimum	0.513603	0.513497	0.513676	0.513693
Gradient L2 norm at optimum	0.001361	0.001120	0.007713	0.001806
Approximate gradient L2 norm at optimum	0.001794	0.000146	0.000214	NA
Time(seconds)	1.930	2.392	8.057	8.749

Sparse dataset: Logistic regression on rcv1_train

Logistic Regression on RCV1_train – 20242 sample points, 47237 variables

	constant	linesearch	adaptive	glmnet
Cost at optimum	0.046339	0.046339	0.046339	0.046342
Gradient L2 norm at optimum	3.892572e-07	4.858723e-07	6.668943e-10	7.592185e-06
Approximate gradient L2 norm at optimum	3.318267e-07	4.800463e-07	2.647663e-10	NA
Time(seconds)	0.814	0.872	1.368	4.372

References

[1] Mark Schmidt, Nicolas Le Roux, and Francis Bach. Minimizing Finite Sums with the Stochastic Average Gradient. arXiv:1309.2388 [cs, math, stat], September 2013. arXiv: 1309.2388. [ bib | http ]

bigoptim's Issues

Bug in Logistic Regression cost gradient

Strange behavaviour of the L2 regularizied logistic regression gradient in the tests. Norm of the true gradient is larger than that of the approximate one.

Pass R CMD check --as-cran

could not find function "get_cost"

Hey @IshmaelBelghazi nice to see that you wrote some demos. However I am getting an error when I try to run them. Maybe you need to export get_cost?

thocking@silene:~/R$ R --vanilla

R version 3.2.1 (2015-06-18) -- "World-Famous Astronaut"
Copyright (C) 2015 The R Foundation for Statistical Computing
Platform: x86_64-unknown-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

> library(bigoptim)
Loading required package: Matrix
demo> demo("example_SAG", package="bigoptim")


    demo(example_SAG)
    ---- ~~~~~~~~~~~

Type  <Return>   to start : 

> library(Matrix)

> data(rcv1_train)

> X <- rcv1_train$X

> ##X <- cBind(rep(1, NROW(X), X), X)
> y <- rcv1_train$y

> n <- NROW(X)

> p <- NCOL(X)

> ## Setting seed
> ##set.seed(0)
> maxIter <- n * 20 

> lambda <- 1/n

> tol <- 0

> print("Running Stochastic average gradient with constant step size\n")
[1] "Running Stochastic average gradient with constant step size\n"

> ## -----------------------------------------------------------------------------
> ## SAG with Constant step size
> sag_constant_fit <- sag_fit(X=X, y=y, lambda=lambda, maxiter=maxIter,
+                             tol=0, fit_alg="constant", model="binomial")

> cost_constant <- get_cost(sag_constant_fit, X=X, y=y) 
Error in eval(expr, envir, enclos) : could not find function "get_cost"
> demo("example_SAG2", package="bigoptim")


    demo(example_SAG2)
    ---- ~~~~~~~~~~~~

Type  <Return>   to start : 

> ## Loading Data set
> data(covtype.libsvm)

> ## Normalizing Columns and adding intercept
> X <- cbind(rep(1, NROW(covtype.libsvm$X)), scale(covtype.libsvm$X))

> y <- covtype.libsvm$y

> y[y == 2] <- -1

> n <- NROW(X)

> p <- NCOL(X)

> ## Setting seed
> #set.seed(0)
> ## Setting up problem
> maxiter <- n * 20  ## 10 passes throught the dataset

> lambda <- 1/n 

> tol <- 1e-4

> ## -----------------------------------------------------------------------------
> ## SAG with Constant step size
> print("Running Stochastic Average Gradient with constant step size")
[1] "Running Stochastic Average Gradient with constant step size"

> sag_constant_fit <- sag_fit(X=X, y=y, lambda=lambda, maxiter=maxiter,
+                             tol=tol, family="binomial",
+                             fit_alg="constant", standardize=FALSE)

> cost_constant <- get_cost(sag_constant_fit, X, y)
Error in eval(expr, envir, enclos) : could not find function "get_cost"
In addition: Warning message:
In sag_fit(X = X, y = y, lambda = lambda, maxiter = maxiter, tol = tol,  :
  Optimisation stopped before convergence. Try incrasing maximum number of iterations
> demo("monitoring_training", package="bigoptim")


    demo(monitoring_training)
    ---- ~~~~~~~~~~~~~~~~~~~

Type  <Return>   to start : 

> suppressPackageStartupMessages(library(ggplot2))

> suppressPackageStartupMessages(library(glmnet))

> ## Loading Data set
> data(covtype.libsvm)

> ## Normalizing Columns and adding intercept
> X <- cbind(rep(1, NROW(covtype.libsvm$X)), scale(covtype.libsvm$X))

> y <- covtype.libsvm$y

> y[y == 2] <- -1

> n <- NROW(X)

> p <- NCOL(X)

> ## Setting seed
> #set.seed(0)
> ## Setting up problem
> n_passes <- 50  ## number of passses trough the dataset

> maxiter <- n * n_passes

> lambda <- 1/n 

> tol <- 0

> family <- "binomial"

> fit_algs <- list(constant="constant",
+                  linesearch="linesearch",
+                  adaptive="adaptive")

> sag_fits <- lapply(fit_algs, function(fit_alg) sag_fit(X, y,
+                                                    lambda=lambda,
+                                                    maxiter=maxiter,
+                                                    family=family,
+                                                    fit_alg=fit_alg,
+                                                    standardize=FALSE,
+                                                    tol=tol, monitor=TRUE))

> print(lapply(sag_fits, function(sag_fit) get_cost(sag_fit, X, y)))
Warning message:
In sag_fit(X, y, lambda = lambda, maxiter = maxiter, family = family,  :
  Optimisation stopped before convergence. Try incrasing maximum number of iterations
Error in print(lapply(sag_fits, function(sag_fit) get_cost(sag_fit, X,  : 
  error in evaluating the argument 'x' in selecting a method for function 'print': Error in FUN(X[[i]], ...) : could not find function "get_cost"
> sessionInfo()
R version 3.2.1 (2015-06-18)
Platform: x86_64-unknown-linux-gnu (64-bit)
Running under: Ubuntu precise (12.04.5 LTS)

locale:
 [1] LC_CTYPE=en_CA.UTF-8       LC_NUMERIC=C              
 [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_CA.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_CA.UTF-8   
 [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C            
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] glmnet_1.9-5        ggplot2_1.0.1       bigoptim_0.0.0.9000
[4] Matrix_1.2-1       

loaded via a namespace (and not attached):
 [1] Rcpp_0.11.6        lattice_0.20-31    digest_0.6.4       MASS_7.3-40       
 [5] grid_3.2.1         plyr_1.8.1         gtable_0.1.2       scales_0.2.3      
 [9] reshape2_1.2.2     labeling_0.2       proto_1.0.0        RColorBrewer_1.0-5
[13] tools_3.2.1        stringr_0.6.2      dichromat_2.0-0    munsell_0.4.2     
[17] colorspace_1.2-4  
> 

Improve numerical stability of log expressions

Expression log(1 + x) can be replaced by log1p(x) to avoid loss of precision.

C interface of multiple cost functions

Pass struct of function pointers to a model blind sag optimizer in C.

Change C code Variable names to be more intuitive.

Some of the variables names are too short and not descriptive enough.

C version of GLMs cost functions

C version of GLMs cost functions as well as their gradients

Improve R code documentation

Improve R code Documentation.

Add sparse Matrices support

Use the Matrix Package C API to add Compressed Sparse Column matrices support.

Set Doxygen documentation for C code.

LICENCE is not a really a licence

bigoptim/LICENSE

Lines 1 to 2 in 720e80b

	YEAR: <2015>
	COPYRIGHT HOLDER: <Mohamed Ishmael Diwan Belghazi>

Looks more like a copyright notices. What is the licence of this work?

Tidy up C code

Tidy up C code before first milestones. This inclues:

• Improve variable naming
• Improve code structure

Add GLM cost functions in R

R version of GLMs cost functions as well as their gradients

Add SAG with line-search and adaptive sampling