ollieboyne / sslap

Auction Algorithm for Sparse Linear Assignment Problems

SSLAP

This library provides implementations for solvers for Super Sparse Linear Assignment Problems.

An assignment problem is one where a one-to-one assignment has to be made between two sets, where each assignment has an associated cost.

In super sparse assignment problems, typically less than 1% of all feasible assignments are allowed.

This library provides a Cython implementation of the Auction Algorithm [1], which is well suited for super sparse problems. It is one in which people in one set 'bid' for objects in the other set, driving up their prices in order to find an optimal assignment.

Also provided is an implementation of the Hopcroft-Karp Algorithm [2] for finding a maximum matching in a bipartite graph. This is used by the Auction solver to check that a given problem has a valid solution.

Installation

Tested on Windows (Python 3.8):

pip install sslap

Tested on Linux (Python 3.7):

pip install git+https://github.com/OllieBoyne/sslap.git

Usage

• For usage of the Auction Algorithm, view examples/test_auction.py
• For usage of Hopcroft-Karp, view examples/test_feasibility.py

Benchmarking

The algorithm is best suited for large and sparse problems, where it outperforms scipy.optimize.linear_sum_assignment.

See below for some timed comparisons of the runtime for problems of varying density (% of valid entries in the matrix) and matrix size.

Notes

• A matrix passed into from_matrix requires positive values only, and -1 indicates invalid values.
• If the matrix is sufficiently large (experiments show N > 120k), auction_solve may crash unexpectedly. To avoid this, pass in the argument cardinality_check=False to auction_solve

[1] Bertsekas, D. A Distributed Algorithm for the Assignment Problem (1979)

[2] Hopcroft J. Karp, R. An n^(5/2) algorithm for maximum matchings in bipartite graphs (1973)

sslap's Issues

Using sparse matrices as input

Hi, thanks for your work this looks really interesting! I have a question however:

As far as I understand the input at the moment is a numpy array where "-1" defines disallowed assignments. Would it be possible to actually give a sparse matrix (e.g. from scipy sparse) as an input for the algorithm where a cost of 0 would define that no assignment is allowed and the higher the value the more likely is the assignment?

I am asking this because one of my main problems with the traditional LAP solvers is that my cost matrices become too large to fit in memory. Therefore I am looking for something that can handle sparse matrices.

Thank you!

Rectangular problems

First of all, thank you for your great piece of code!
I'm wondering whether it is possible or not to handle rectangular sparse problems, i.e. people < objects.

Buffer dtype mismatch Linux-Ubuntu

Hi,

I've run into the following problem: When using the solver on a Windows machine everything is fine. However, trying it on a machine running Linux-Ubuntu I get the following error:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/data/homes/atschan/.conda/envs/python_3_7/lib/python3.7/site-packages/sslap/auction_solve.py", line 49, in auction_solve
    solver = _from_sparse(loc=loc, val=val, size=size, **kw)
  File "sslap/auction_.pyx", line 567, in sslap.auction_._from_sparse
  File "sslap/auction_.pyx", line 596, in sslap.auction_._from_sparse
ValueError: Buffer dtype mismatch, expected 'long' but got 'int'

Maximum allowed dimension exceeded

Hi, when I use auction_solve to run 60000 x 60000 matrix, I got "Maximum allowed dimension exceeded" error. I checked your code in sslap/auction_.pyx, but I didn't find any limitation of the maximum dimension. Would you kindly tell me how to solve it? Thank you so much.

wrong matching result

Hi,

I was trying to use auction solver in one of my solution. While comparing it to scipy linear sum assignment I saw results are not same. Attached is an example with 1 object and 8 people. Scipy is picking the right value (40.4) and auction solver failed to pick it.

Segmenation fault (core dumped) when using large sparse matrix

I did some testing with the sparse matrix input for the solver and came across something. Currently I am just using sparse identity matrices for testing purposes and whenever I go above n=130'000 something seems to break. On Linux I typically get a Segmentation fault (core dumped) and on a Windows machine the python kernel seems to just break down.

Python 3.6.8 |Anaconda, Inc.| (default, Dec 30 2018, 01:22:34)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from scipy import sparse
>>> from sslap import auction_solve
>>>
>>> Cs = sparse.coo_matrix(sparse.identity(130000))
>>>
>>> sol = auction_solve(coo_mat=Cs, problem='max')
Segmentation fault (core dumped)

Rectangular sparse problems?

First of all, thank you for your great piece of code!
I'm wondering whether it is possible or not to handle rectangular sparse problems, i.e. people < objects.

Possible memory leak issue

I've been trying to use this package to solve a very large (1M x 1M cost matrix, though very sparse) linear assignment problem. doing this causes a segmentation fault even when passing cardinality_check=False.

To get around this, I cut up my problem into many smaller problems of varying sizes (10kx10k to 300x300) because this is suboptimal but feasible for my use-case. My program will start to solve the smaller problems, but will eventually crash, but never at one particular problem.

To isolate the problem, I wrote the following code to remove the other elements of my code:

import numpy as np
from scipy.sparse import random as sparse_random
import sslap

size = 1000
density = 0.01
loop_count = 100

for i in range(loop_count):
    dense_matrix = sparse_random(size, size, density=density, format='coo', data_rvs=np.random.rand).todense()
    result = sslap.auction_solve(dense_matrix, cardinality_check=False, fast=True)
    print(f"Loop {i + 1} completed")

When this is done, memory usage climbs very quickly. Our compute has 200gb ram and a larger swap, and it quickly exceeds this.

Notably this only happens when passing a sparse matrix, and not a dense matrix. I believe it still happens with the dense matrix but it appears to be much much slower.