openopt / copt

A Python library for mathematical optimization

Note: This package is no longer actively maintained. I won't be actively responding to issues. If you'd like to volunteer to maintain it, please drop me a line at [email protected]

copt: composite optimization in Python

copt is an optimization library for Python. Its goal is to provide a high quality implementation of classical optimization algorithms under a consistent API.

Docs | Examples

Installation

If you already have a working installation of numpy and scipy, the easiest way to install copt is using pip

pip install -U copt

Alternatively, you can install the latest development from github with the command:

pip install git+https://github.com/openopt/copt.git

Citing

If this software is useful for your research, please consider citing it as

@article{copt,
  author       = {Fabian Pedregosa, Geoffrey Negiar, Gideon Dresdner},
  title        = {copt: composite optimization in Python},
  year         = 2020,
  DOI          = {10.5281/zenodo.1283339},
  url={http://openopt.github.io/copt/}
}

Development

The recommended way to work on the development versionis the following:

1. Clone locally the github repo. This can be done with the command:

   git clone https://github.com/openopt/copt.git
   

This will create a copt directory.

2. Link this directory to your Python interpreter. This can be done by running the following command from the copt directory created with the previous step:

python setup.py develop

Now you can run the tests with py.test tests/

copt's Issues

SFW very slow

The example https://storage.googleapis.com/copt-doc/auto_examples/plot_sfw.html#sphx-glr-auto-examples-plot-sfw-py currently takes 37 minutes to run, that is way too slow. Ideally it should run under 5 minutes.

I think the example is fine, and the issue is in the SFW implementation, that could be made faster. I propose the following TODO:

1. Profile minimize_sfw to find the bottlenecks
2. Figure strategies to make it faster. For example, batching and/or jit'ing the relevant parts using JAX or numba

refactor all proximal gradient methods in a single one

remove return_gradient={True, False}

For API compatibility

Verbose not used in minimize_proximal_gradient

Thank you for the open source. The verbose argument of minimize_proximal_gradient() of copt/proximal_gradient.py is not being used.

It's a minor problem and not urgent.

h_lipschitz in minimize_three_split undefined

The argument doesn't appear in the docstring; I'm not sure what it's supposed to be.
Perhaps linked with #77

check stopping criterion for SAGA

Adversarial Image Example

It would be nice to have an example on how to generate an adversarial image

Http error for downloading the datasets

Got the following error when trying to load the Madelon dataset.

madelon dataset is not present in the folder /home/geoff/copt_data/madelon. Downloading it ...
---------------------------------------------------------------------------
HTTPError                                 Traceback (most recent call last)
<ipython-input-5-4c820c436ec3> in <module>
----> 1 A, b = cp.datasets.load_madelon()

~/PycharmProjects/copt/copt/datasets.py in load_madelon(subset, data_dir)
    153         * :ref:`sphx_glr_auto_examples_frank_wolfe_plot_vertex_overlap.py`
    154     """
--> 155     return _load_dataset("madelon", subset, data_dir)
    156 
    157 

~/PycharmProjects/copt/copt/datasets.py in _load_dataset(name, subset, data_dir)
     58         )
     59         url = "https://storage.googleapis.com/copt/datasets/%s.tar.gz" % name
---> 60         local_filename, _ = urllib.request.urlretrieve(url)
     61         print("Finished downloading")
     62 

~/anaconda3/envs/copt/lib/python3.7/urllib/request.py in urlretrieve(url, filename, reporthook, data)
    245     url_type, path = splittype(url)
    246 
--> 247     with contextlib.closing(urlopen(url, data)) as fp:
    248         headers = fp.info()
    249 

~/anaconda3/envs/copt/lib/python3.7/urllib/request.py in urlopen(url, data, timeout, cafile, capath, cadefault, context)
    220     else:
    221         opener = _opener
--> 222     return opener.open(url, data, timeout)
    223 
    224 def install_opener(opener):

~/anaconda3/envs/copt/lib/python3.7/urllib/request.py in open(self, fullurl, data, timeout)
    529         for processor in self.process_response.get(protocol, []):
    530             meth = getattr(processor, meth_name)
--> 531             response = meth(req, response)
    532 
    533         return response

~/anaconda3/envs/copt/lib/python3.7/urllib/request.py in http_response(self, request, response)
    639         if not (200 <= code < 300):
    640             response = self.parent.error(
--> 641                 'http', request, response, code, msg, hdrs)
    642 
    643         return response

~/anaconda3/envs/copt/lib/python3.7/urllib/request.py in error(self, proto, *args)
    567         if http_err:
    568             args = (dict, 'default', 'http_error_default') + orig_args
--> 569             return self._call_chain(*args)
    570 
    571 # XXX probably also want an abstract factory that knows when it makes

~/anaconda3/envs/copt/lib/python3.7/urllib/request.py in _call_chain(self, chain, kind, meth_name, *args)
    501         for handler in handlers:
    502             func = getattr(handler, meth_name)
--> 503             result = func(*args)
    504             if result is not None:
    505                 return result

~/anaconda3/envs/copt/lib/python3.7/urllib/request.py in http_error_default(self, req, fp, code, msg, hdrs)
    647 class HTTPDefaultErrorHandler(BaseHandler):
    648     def http_error_default(self, req, fp, code, msg, hdrs):
--> 649         raise HTTPError(req.full_url, code, msg, hdrs, fp)
    650 
    651 class HTTPRedirectHandler(BaseHandler):

HTTPError: HTTP Error 403: Forbidden

Stochastic Frank-Wolfe variants

It would be nice to be able to benchmark variants of Stochastic Frank-Wolfe algorithms. Assigning this to myself.

2nd order methods?

Are second-order methods in the scope of the project?

Docstring doesn't match function signature

h_prox is not documented as a parameter in the function string as well as the function documentation

copt/copt/gradient.py

Line 238 in 1636ad4

def minimize_TOS(

minimize_frank_wolfe doesn't take lipschitz as parameter, unlike doc announces

Cf title. lipschitz isn't in the function arguments.

Master branch borked

Hi everyone,

I'm really sorry for this distraction, but I pushed to master by accident before I was ready. I think the way to solve this would be

git checkout -b tmpbranch
git branch -D master
git checkout e5cb995  # this is fabian's last commit
git checkout -b master
git push -f

But I am hesitant to do this, especially if someone has unfortunately grabbed my commits by accident.

Any tips?

Sorry again and thanks for the help,
Gideon

@fabianp I think that you should be able to guide me here.

GitHub Actions?

I see that copt no longer has CI working. Would you like me to add configuration for GitHub Actions?

Three operator splitting with adaptative step size: prox on x inside loop ?

Hello,

I am trying to apply the TOS algorithm from your paper https://arxiv.org/pdf/1804.02339.pdf by using the COPT implementation. There is one detail that I do not really catch here.

Based on your article, it seems to me that the steps (lines 2-7) in Algorithm 1 of https://arxiv.org/pdf/1804.02339.pdf do not match the implementation:

copt/copt/splitting.py

Lines 130 to 144 in 28903ab

	x = prox_1(z - step_size * (u + grad_fk), step_size, *args_prox)
	incr = x - z
	norm_incr = np.linalg.norm(incr)
	ls = norm_incr > 1e-7 and line_search
	if ls:
	for it_ls in range(max_iter_backtracking):
	rhs = fk + grad_fk.dot(incr) + (norm_incr ** 2) / (2 * step_size)
	ls_tol = f_grad(x, return_gradient=False) - rhs
	if ls_tol <= LS_EPS:
	# step size found
	# if ls_tol > 0:
	# ls_tol = 0.
	break
	else:
	step_size *= backtracking_factor

I might be wrong but shouldn't the line:

(

copt/copt/splitting.py

Line 130 in 28903ab

x = prox_1(z - step_size * (u + grad_fk), step_size, *args_prox)

)

come after the loop begins:

(

copt/copt/splitting.py

Line 135 in 28903ab

for it_ls in range(max_iter_backtracking):

)

so as to match the steps (lines 2-7) in Algorithm 1 of https://arxiv.org/pdf/1804.02339.pdf

As it is, it seems to me that the variable x is never updated during the backtracking. Am I missing something ?

Thank you for the fantastic work !

Homotopy Smoothing Methods for Frank-Wolfe

Hi everyone,

I am taking the first steps to adding homotopy smoothing methods. I don't have a timeline but will probably work on this from time to time over the next days/weeks.

At some point when its ready for review / merging, we might need to have a discussion regarding organization since I aim to incorporate stochastic homotopy methods as well. Then it is somewhat unclear to me whether to include such methods in randomized.py or homotopy.py.

Happy to have a further discussion but just be aware that the code is far from ready.

Thanks!
Gideon

"sklearn" dependency

Hello! Would it be possible to cut a new release soon? The most recent release, version 0.8.4 (from 2020), has "sklearn" as a dependency, which causes issues for our downstream dependencies when we try to install copt: https://github.com/yeatmanlab/AFQ-Insight/actions/runs/4484579586/jobs/7885232689?pr=138#step:4:577

Doc site breaks for constraints since refactoring

Cf http://openo.pt/copt/loss_functions.html

Implement generalized "linear minimization oracles"

Lu and Freund, 2020 use a generalization of the LMO found in Frank-Wolfe optimization. It generalizes minimizing a sum of a linear function and an indicator function by minimizing a sum of a linear function and a bounded domain function (potentially strongly convex / smooth).

It seems from their paper that they obtain linear convergence in the case of a strongly convex regularizer thus defined. It may work also for other stochastic variants.

Also see Bach, 2012 and Yu et al. 2017

Allow group lasso with non-contiguous groups

Allow to store data in arbitrary location via environment variable

primal-dual solver broken on some examples

The example http://openo.pt/copt/auto_examples/proximal_splitting/plot_tv_deblurring.html#sphx-glr-auto-examples-proximal-splitting-plot-tv-deblurring-py suggests the primal-dual solver is broken

CC @vene since he might be using it

Good practice on logging

What's the easiest / more practical way to log results, and keep them locally?
For now, we have the Trace objects, but no writing to file. Should this be part of a logging.py file, or should this functionality be something the user imports from another library?

Benchmarking algorithms of various families

It would be nice to have an example to compare speed of convergence of SAGA/SVRG/SFW on problems attaining the same optimum.

Travis failing since last refactoring of utils

see https://travis-ci.org/github/openopt/copt/builds

Uniform styling for citations

For instance,

Amir is Beck's name, or a 2nd author?

Issue with coveralls

Error when running coveralls in Travis. Unclear if this is important or not.

Sparse Group Lasso

There's some nice code at richford@7ff9dbd

how to deal with intercept?

Premature termination with callbacks

The issue

When I execute minimize_proximal_gradient with a copt.utils.Trace() callback, I get early termination. For example, when trying to run an updated version of the gradient descent example:

import numpy as np
import copt as cp

# .. construct (random) dataset ..
n_samples, n_features = 1000, 200
np.random.seed(0)
X = np.random.randn(n_samples, n_features)
y = np.random.rand(n_samples)

f = cp.utils.LogLoss(X, y)
step_size = 1. / f.lipschitz

cb_pgd = cp.utils.Trace(f)
result_pgd = cp.minimize_proximal_gradient(
    f.f_grad, np.zeros(n_features), step_size=step_size,
    callback=cb_pgd, tol=0, accelerated=False)

cb_apgd = cp.utils.Trace(f)
result_apgd = cp.minimize_proximal_gradient(
    f.f_grad, np.zeros(n_features), step_size=step_size,
    callback=cb_apgd, tol=0, accelerated=True)

This throws no error but the traces cb_apgd and cp_pgd contain only one element, suggesting that minimize_proximal_gradient terminated early.

I think the issue is with this line in minimize_proximal_gradient and may have been introduced in 36d7090b. To illustrate, the old check would pass

if cb_pgd({"x": np.zeros(n_features), "step_size": 42}) == False:
    print("Early Termination!")

with no output. Conversely, the new check is

if cb_pgd({"x": np.zeros(n_features), "step_size": 42}) is None:
    print("Early Termination!")

with output Early Termination!. So this updated version triggers the break in minimize_proximal_gradient.

Suggested Fix

Depends on what the intent is for the callback functions' return values. Currently copt.utils.Trace() implicitly returns None but minimize_proximal_gradient is expecting the callback to return a truthy value. So one could fix this by either (A) making Trace return a truthy value or by (B) changing minimize_proximal_gradient to test for

if callback(locals()) is not None:

or simply

if callback(locals()):

I'm happy to submit a PR with either fix if you can tell me which direction you'd like to go. I can also submit a PR with the updated example above once Trace() and minimize_proximal_gradient are playing nicely again.

plot_saga_vs_svrg.py broken

with numba installed, I get the error
"""
File "copt/utils.py", line 379:
def log_deriv(p, y):

        tmp[p > 0] = np.exp(-p[p > 0])
        ^

[1] During: resolving callee type: type(CPUDispatcher(<function LogLoss.partial_deriv..log_deriv at 0x7f9d81486710>))
[2] During: typing of call at /home/pedregosa/dev/copt/copt/randomized.py (179)

[3] During: resolving callee type: type(CPUDispatcher(<function LogLoss.partial_deriv..log_deriv at 0x7f9d81486710>))
[4] During: typing of call at /home/pedregosa/dev/copt/copt/randomized.py (179)

File "copt/randomized.py", line 179:
def _saga_epoch(x, idx, memory_gradient, gradient_average, grad_tmp, step_size):

p += x[j_idx] * A_data[j]
grad_i = f_deriv(p, b[i])
"""

Regression seems to be introduced in 2873039

pairwise_lmo outputs 0 update_direction

This breaks the 'DR' step scheme, which is proportional to1/norm(update_direction).
It happens even when we are not at the optimum.

In particular, where does this part of the code come from?

stopping criterion for BCD

Installation issue in new environment

I'm having trouble installing copt in a new python environment. Steps to reproduce (using conda in bash):

conda create --name copt-test python=3.6
conda activate copt-test
echo copt > requirements.txt
pip install -r requirements.txt

Output:

Collecting copt (from -r requirements.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/95/00/b7bb95dd4c2310c8113bc54982b2fc33f1c79753110f441e2f7380cfad87/copt-0.6.0.tar.gz
    ERROR: Complete output from command python setup.py egg_info:
    ERROR: Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/0s/g8g2r38n4451t0gd0txvzr6r0000gn/T/pip-install-6q1flaok/copt/setup.py", line 1, in <module>
        import copt
      File "/private/var/folders/0s/g8g2r38n4451t0gd0txvzr6r0000gn/T/pip-install-6q1flaok/copt/copt/__init__.py", line 3, in <module>
        from .proxgrad import minimize_PGD, minimize_APGD
      File "/private/var/folders/0s/g8g2r38n4451t0gd0txvzr6r0000gn/T/pip-install-6q1flaok/copt/copt/proxgrad.py", line 2, in <module>
        import numpy as np
    ModuleNotFoundError: No module named 'numpy'
    ----------------------------------------
ERROR: Command "python setup.py egg_info" failed with error code 1 in /private/var/folders/0s/g8g2r38n4451t0gd0txvzr6r0000gn/T/pip-install-6q1flaok/copt/

I believe this could be fixed by including numpy and scipy in setup.py using the install_requires parameter. I will submit a PR to that effect.

Error loading img1

I get the following traceback when calling copt.datasets.load_img1:

AttributeError                            Traceback (most recent call last)
<ipython-input-29-0e616505f996> in <module>
----> 2 X_img1 = copt.datasets.load_img1()
~/copt/copt/datasets.py in load_img1(n_rows, n_cols)
     28     dim1 = int(np.sqrt(grid.shape[0]))
     29     grid = grid.reshape((dim1, dim1))
---> 30     return misc.imresize(grid, (n_rows, n_cols))
     31 
     32 

AttributeError: module 'scipy.misc' has no attribute 'imresize'

`partial_deriv` computation for `batch_size > 1` doesn't work

In the mini-batch case, we want partial_deriv to return the vector {f_i(p_i, y_i)}, where p = A_i^T x and i in batch_idx.
According to http://fa.bianp.net/blog/2019/evaluate_logistic/#sec5, this vector should exactly be given by LogLoss.expit_b(p, y[batch_idx]), with p = A[batch_idx].dot(x).

1. Why is the implementation of partial_deriv different than this?
2. Both implementations give the same result. (Tested with numpy.allclose)
3. Using this does not make the FW gap go to zero as it should in examples/frank_wolfe/plot_sfw.py.

This is linked with #55, #56.

The plot uses batch_size = 10.

Trouble reproducing Stochastic FW benchmarks

Hello!

I'm trying play around with the stochastic Frank-Wolfe benchmarks you have in this example and I'm having trouble reproducing the figure by running the provided example. The expected results are as follows:

However when I run the script out of the box, the first thing to note is that it takes a very long time; for 10 iters, these are the timing results:

CPU times: 
     user 16min 12s
     sys: 7min 29s
     total: 23min 42s
Wall time: 12min 25s

Perhaps this is fine seeing as ~10 steps should be sufficient (this would correspond to ~10^8 gradient evaluations which would match the figure in the example), but then the problem comes with the actual convergence results; if I set max_iter=500 (vs 1e4) and let it run for the full 2h x 3 runs, I get worse results:

In particular, after 10^8 gradient evaluations, the relative suboptimality

Some additional info that may be useful:

• Python 3.7.6
• SciPy 1.7.3
• NumPy 1.21.6
• Latest (development) version of COPT.

If useful, we can play around with the example in a colab notebook.

Thanks for any help!

potential issue in SFW example

the FW gap doesn't go to zero in this example

http://openo.pt/copt/auto_examples/frank_wolfe/plot_sfw.html#sphx-glr-auto-examples-frank-wolfe-plot-sfw-py

I suspect there's a bug in the code

lmo name is misleading

In the L1ball, the lmo in update_direction is solving $\max_s \langle s,u\rangle$ instead of $\min_s \langles,u\rangle$. The name and docs are incoherent with the code.
The functions in frank_wolfe.py use lmo(-grad), consistent with this remark.

Numba uses up all cores

It looks like numba uses up all available cores. It may be because of the utils.sample_batches function, with the @njit(parallel=True) decorator.

Any idea on how to limit this / performance impact?

Refactor utils.py

Right now, utils.py contains objects for loss functions, constraints, and helper functions for speeding up computation. It should probably be split in 3 files: functions.py, constraints.py and utils.py.

Adding this as a general TODO, and up for discussion. (@fabianp )

`pip install copt` yields `ModuleNotFoundError`. Same issue when installing from github

Update direction example is broken

The example
https://github.com/openopt/copt/blob/master/examples/frank_wolfe/plot_vertex_overlap.py

(see output here )

does not produce the expected output.

@gideonite: do you want to take a look into it? This looks like a good issue to get familiar with the codebase

Implement partial_deriv method for Square Loss

Adding this as a todo for myself. This is needed for applying randomized methods (SAGA, SAG, Stochastic FW) on least squared models, such as Elastic Net, LASSO or Ridge regression.

Different versions of code in pypi vs GitHub

We're using copt (on a project led by @FedericoV) for a project, and have found differences between the package imported from pypi and directly from GitHub. We're fixing the version to 0.8.3 since we use the copt.loss function, which disappears in 0.8.4. However, I get different behavior depending on how the package is installed, despite the fact that the version of the code is supposedly the same.

PyPI

$ python -m venv copt_pypi; source copt_pypi/bin/activate;

# Install directly from PyPI
$ pip install copt==0.8.3

# Check version
$ python -c "import copt; print(copt.__version__)"
0.8.3

# Throws error
$ python -c "import copt.loss"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'copt.loss'

GitHub

$ python -m venv copt_gh; source copt_gh/bin/activate;

# Install GitHub version
$ pip install git+https://github.com/openopt/copt.git

# Check version
$ python -c "import copt; print(copt.__version__)"
0.8.3

# No errors; import is successful
$ python -c "import copt.loss"

The latter works for implementation, but I want to be able to set this in a requirements.txt file or something similar and have repeatable behavior. Do you know why the two versions are different and, if so, what the best way would be of making sure we pull in the correct version consistently?

SAGA / SVRG example plot and SAGA Trace ?

From the docs on the SAGA / SVRG methods, it seems like max_iter is the number of epochs (full passes over the dataset). Therefore, the x_axis values in plt_saga_vs_svrg.py should be multiplied by n_samples for consistency. Do I understand this well?

Cf https://storage.googleapis.com/copt-doc/auto_examples/plot_saga_vs_svrg.html#sphx-glr-auto-examples-plot-saga-vs-svrg-py

Is it possible to get the trace of function values within an epoch?

Tables in document aren't rendered properly

Hi,

Several tables in the document aren't rendered properly; the content of the first column exceeds its area and overlap the second column.

For example,

Passing extra arguments to optimization methods

Most optimization methods do not allow the user to pass extra parameters to the optimization function - requiring the user to instead provide the parameters through a functor or a closure.

Unfortunately, this does not play nicely with the Pytorch JIT, which requires that all input parameters be passed explicitly to the function. For example:

@torch.jit.script
def predict_X(alpha: torch.Tensor,
              B: torch.Tensor,
              X: torch.Tensor,
              allowed_indexes: torch.Tensor,
              t0: int):
    
    t, m = X.shape
    X_pred = torch.zeros((t, m))
    
    for i in range(m):
        Xi_slice = X[:, allowed_indexes[i, :]]
        Bi_slice = B[i, allowed_indexes[i, :]]
        Ai_slice = alpha[allowed_indexes[i, :]]
        
        X_pred[:, i] = Xi_slice @ Bi_slice
        X_pred[t0:, i] +=  Xi_slice[t0:] @ Ai_slice
        
    return X_pred

@torch.jit.script
def calculate_mse(alpha: torch.Tensor,
                  B: torch.Tensor,
                  X: torch.Tensor,
                  allowed_indexes: torch.Tensor,
                  t0: int):
    X_pred = predict_X(alpha, B, X, allowed_indexes, t0)
    res = (X - X_pred)
    mse = torch.mean(res**2)
    return mse


def make_loss_fcn(X: torch.Tensor,
                  allowed_indexes: torch.Tensor,
                  t0: int):
    X = torch.as_tensor(X)
    allowed_indexes = torch.as_tensor(allowed_indexes)
      
    #@torch.jit.script
    def mse_fcn(theta: np.array):
        # We pack all the parameters (alpha and beta) into
        # a single vector of dimension m*(m+1)
        theta = torch.tensor(theta, requires_grad=True)
        
        # solve for m with quadratic: a=1, b=1
        c = theta.shape[0]
        m = int((torch.sqrt(1+4*c)-1) / 2)
        
        # unpack theta from 1d to actual model parameters.
        B = theta[:m*m].reshape((m, m))
        alpha = theta[m*m:]
        
        mse = calculate_mse(alpha, B, X, allowed_indexes, t0)
        mse.backward()
        # return mse, theta.grad
        return mse.detach().clone().numpy(), theta.grad.detach().clone().numpy()

    return mse_fcn

f_grad = make_loss_fcn(torch.as_tensor(X),
                         torch.as_tensor(allowed_indexes),
                         t0)
sol = cp.minimize_proximal_gradient(f_grad, theta, prox=l1_ball.prox, callback=cb)

This works: but requires an annoying pattern of creating a closure which creates a loss function which then calls into a jitted function. One way to fix it would be to allow the user to pass function_args and function_kwargs which are passed through to f_grad automatically.

Mini-batches for SAGA/SVRG

Both methods use batch size=1.

`lmo` returns the update direction, and not the argmax of the lmo

The constraint.lmo(u, x) methods for constraint objects in utils.py return -x + argmax_{s\in constraint} < u, s >. Some methods do not directly update the current iterate using this (e.g. the Lu and Freund Stochastic FW paper) ; it would be nicer to have a constraint.lmo(u) that returned argmax_{s\in constraint} < u, s >.