dzhang314 / dzoptimization.jl

Fast in-place nonlinear optimization with fine-grained user control

DZOptimization.jl

Copyright © 2019-2021 by David K. Zhang. Released under the MIT License.

DZOptimization.jl is a Julia package for smooth nonlinear optimization that emphasizes performance, flexibility, and memory efficiency. In basic usage examples (see below), DZOptimization.jl has 6x less overhead and uses 10x less memory than Optim.jl.

Unlike traditional optimization libraries which only provide a black-box optimize function (e.g., Optim.jl and NLopt.jl), DZOptimization.jl gives you full control of the optimization loop. This allows you to:

• interactively monitor the progress of an optimizer,
• interleave nonlinear optimization with other tasks,
• save/load data in the middle of optimization,
• run multiple optimizers in parallel, and
• terminate optimization whenever you want (as opposed to a predetermined list of convergence criteria).

DZOptimization.jl is designed to minimize overhead. It uses static data structures and in-place algorithms to ensure that memory is never dynamically allocated (outside of optimizer constructors). This makes DZOptimization.jl especially suitable for both small-scale optimization problems, since repeatedly allocating small vectors is wasteful, and large-scale optimization problems, since memory usage will never shoot up unexpectedly.

Usage Example

The following example illustrates the use of DZOptimization.BFGSOptimizer to minimize the Rosenbrock function, starting at a random initial point.

using DZOptimization

rosenbrock_objective(x::Vector) =
    (1 - x[1])^2 + 100 * (x[2] - x[1]^2)^2

function rosenbrock_gradient!(g::Vector, x::Vector)
    g[1] = -2 * (1 - x[1]) - 400 * x[1] * (x[2] - x[1]^2)
    g[2] = 200 * (x[2] - x[1]^2)
end

opt = BFGSOptimizer(rosenbrock_objective,
                    rosenbrock_gradient!,
                    rand(2), # starting point
                    1.0)     # initial step size

while !opt.has_converged[]
    println(opt.current_objective_value[], '\t', opt.current_point)
    step!(opt)
end

Benchmarks

Compared to Optim.jl, the BFGS implementation in DZOptimization.jl is 6x faster and uses 10x less memory to minimize the Rosenbrock function.

using BenchmarkTools                            | using Optim
@benchmark begin                                |
    opt = BFGSOptimizer(rosenbrock_objective,   | @benchmark optimize(rosenbrock_objective,
                        rosenbrock_gradient!,   |                     rosenbrock_gradient!,
                        rand(2), 1.0)           |                     rand(2), BFGS())
    while !opt.has_converged[]; step!(opt); end |
end                                             |
                                                |
# BenchmarkTools.Trial:                         | # BenchmarkTools.Trial: 
#   memory estimate:  1.14 KiB                  | #   memory estimate:  9.88 KiB
#   allocs estimate:  12                        | #   allocs estimate:  163
#   --------------                              | #   --------------
#   minimum time:     2.800 μs (0.00% GC)       | #   minimum time:     8.599 μs (0.00% GC)
#   median time:      5.563 μs (0.00% GC)       | #   median time:      30.300 μs (0.00% GC)
#   mean time:        5.374 μs (0.64% GC)       | #   mean time:        31.826 μs (4.44% GC)
#   maximum time:     182.925 μs (95.70% GC)    | #   maximum time:     3.824 ms (98.29% GC)

dzoptimization.jl's Issues

Question: Wondering if DZOptimization.jl allows box constraints?

Hi. I really like the idea of DZOptimization.jl -- fast, low memory, lightweight, full user control over termination.

It would be super-valuable to me if it allows box constraints, especially on LBFGS. I see that the function signatures allow constraint_function!::C that returns a boolean, but I didn't see any examples of how to use it. Would you mind letting me know if it does support box constraints and if so, would you mind sharing an example of how to code them if you have one available?

Many thanks.

TagBot trigger issue

This issue is used to trigger TagBot; feel free to unsubscribe.

If you haven't already, you should update your TagBot.yml to include issue comment triggers.
Please see this post on Discourse for instructions and more details.

If you'd like for me to do this for you, comment TagBot fix on this issue.
I'll open a PR within a few hours, please be patient!