airoldilab / ai-sgd Goto Github PK

I've been tweaking sgd.R over the past few days and it has gotten absurdly lengthy (have not pushed the latest yet).

It would be natural to use update functions, e.g., in the main loop, you would have something such as theta.new <- sgd.update(...) for each if statement. However, the number of parameters the update function takes causes them to be very convoluted.

Here's an example of what the main loop would look like:

# Run the stochastic gradient method.
# Main iteration: i = #iteration
for (i in 1:(n*npass)) {
  # Update y matrix if method uses least squares estimate.
  if (sgd.method %in% c("LS-SGD", "LS-ISGD")) {
    y[, i] <- ls.update(i, data, theta.sgd, n)
  }
  # Update parameters.
  if (sgd.method %in% c("SGD", "ASGD", "LS-SGD")) {
    theta.new <- sgd.update(i, data, theta.sgd, lr, lambda, n, ...)
  } else if (sgd.method %in% c("ISGD", "AI-SGD", "LS-ISGD")) {
    theta.new <- isgd.update(i, data, theta.sgd, lr, lambda, n, ...)
  } else if (sgd.method == "SVRG") {
    theta.new <- svrg.update(i, data, theta.sgd, lr, lambda, n, ...)
  }
  theta.sgd[, i+1] <- theta.new
}

Is this an acceptable solution? It makes the sgd() function more readable at the cost of making the update functions themselves a bit convoluted.

It is probably better to have a new folder with the experiments and then
follow a better naming scheme. For example, it is more descriptive to have a name
such as exp_model_params.R e.g., exp_normal_n5p2.R to know that this is an experiment
on the normal model with n=1e5 examples and p=1e2 parameters.

See Bach and Moulines (2013). Compare AI-SGD to Bach's method for logistic regression and SVRG.

svrg.update() may possibly be slower due to loading in the entire DATA object per iteration of the main loop. This may be negligible unless the number of passes is arbitrarily large.

:)

It's preferable to have multiple functions that are roughly of the same category belonging to the same file (or directory). One file should not collect everything however.

Only the Gaussian case currently incorporates the snr argument.

This should be script optimal_aisgd in the theory/ folder.

sgd.R should be generic enough to work for all experiments.

Not sure why, but in my experiments setting lambda to be anything other than zero leads to worse performance. Also the implicit case simply doesn't run.

The DATA object was changed to include a definition of the GLM model
that generates the data. The batch learning needs to change so that
to use this information and feed it into a call glm(..., family=...)
where family will depend on the GLM model.

To address the comment on f9395ac:

NOTE(ptoulis): This was a bit confusing. Where are we using no-slope?

This is used in examples/exp_normal_n5p2.R. Agreed that it's confusing though.

Also, experiment with optimal rate schedules for aiSGD?

I think we should look for some interesting datasets for estimation.
Either from the Pascal challenge or from UCI ?

Especially we would like models where explicit would have trouble with;
Poisson regression is a prime example.