Benchmarks against other Gaussian Processes packages about gaussianprocesses.jl

[chris-nemeth]

Hi @dburov190 thanks for bringing this to our attention. We'll look into this and see if it's possible to speed-up the predict function.

[chris-nemeth]

Another Matlab package to consider is GPstuff https://github.com/gpstuff-dev/gpstuff.

[maximerischard]

Here's a cursory benchmark against GPy, obtaining gradients of the marginal log-likelihood:

using GaussianProcesses
nobsv=3000
X = randn(2,nobsv)
Y = randn(nobsv)
se = SEIso(0.0, 0.0)
gp = GP(X, Y, MeanConst(0.0), se, 0.0)
buf1=Array(Float64, nobsv,nobsv)
buf2=Array(Float64, nobsv,nobsv)
update_mll_and_dmll!(gp, buf1, buf2) # warm-up
@time update_mll_and_dmll!(gp, buf1, buf2)

1.281482 seconds (32.98 k allocations: 1.099 MB)

import GPy
import numpy as np
nobsv=3000
X = np.random.normal(size=(nobsv,2))
Y = np.random.normal(size=(nobsv,1))
m = GPy.models.GPRegression(X, Y, GPy.kern.RBF(2, lengthscale=1.0))
%time m.parameters_changed()

CPU times: user 2.44 s, sys: 213 ms, total: 2.65 s
Wall time: 1.36 s

I think the python and julia code do the same thing, though I'm not 100% sure on the details. The julia version is only a little bit faster, partly because GPy seems to be using some kind of parallelisation (if I understand CPU time > Wall time correctly), which could be worth exploring at some point for us.

[chris-nemeth]

I'll have to check whether GPy uses parallelisation as a default and what sort of speed up it gives. For small jobs, the communication cost of parallelisation often outweighs the benefits.

When you run GPy.models.GPRegression(), does it calculate the gradients? Perhaps a fairer comparison would be against our update_mll!() function instead.

I wonder how GPy scales with dimension? I know that our ARD kernels are slower than the ISO versions, perhaps there's a bigger time difference between GaussianProcesses.jl and GPy when using ARD kernels.

[maximerischard]

I don't know exactly what GPy.models.GPRegression() does, but GP.parameters_changed() does this:

Method that is called upon any changes to :class:~GPy.core.parameterization.param.Param variables within the model.

In particular in the GP class this method re-performs inference, recalculating the posterior and log marginal likelihood and gradients of the model

.. warning::

This method is not designed to be called manually, the framework is set up to automatically call this method upon changes to parameters, if you call

this method yourself, there may be unexpected consequences.

so I think it does the same as update_mll_and_dmll! in the julia package.

[maximerischard]

Rerunning the julia snippet now that #35 is fixed, we're now significantly faster:

nobsv=3000
X = randn(2,nobsv)
Y = randn(nobsv)
se = SEIso(0.0, 0.0)
gp = GP(X, Y, MeanConst(0.0), se, 0.0)
buf1=Array(Float64, nobsv,nobsv)
buf2=Array(Float64, nobsv,nobsv)
update_mll_and_dmll!(gp, buf1, buf2)
@time update_mll_and_dmll!(gp, buf1, buf2)

0.916660 seconds (32.98 k allocations: 1.099 MB)
(although I'm not entirely certain #35 was making it slower)

[chris-nemeth]

That's fantastic! I can't imagine why it's suddenly faster, but I wouldn't be surprised if over time we start to see speed improvements for free as Julia continues to develop.

[maximerischard]

Some more benchmarks, still with N=3000, with various combinations of SE and RQ kernels.

N=3000	julia	GPy
se	0.93	1.25
rq	1.22	1.78
se+rq	1.50	1.92
se*rq	2.10	1.95
se+se2+rq	1.77	2.01
(se+se2)*rq	2.94	2.14
mask(se, [1])	1.03	1.31
mask(se, [1])+mask(rq, [2])	1.48	1.88
fix(se, σ)	0.90	1.27

It looks like we have some work to do on product kernels, but it's looking pretty solid otherwise.

[maximerischard]

I had a look into GPflow. Here's my attempt at computing the gradient:

import GPflow
import numpy as np

nobsv=3000
X = np.random.normal(size=(nobsv,2))
Y = np.random.normal(size=(nobsv,1))

se = GPflow.kernels.RBF(2, lengthscales=1.0)
gp = GPflow.gpr.GPR(X, Y, se)
x=gp.get_free_state()
gp._compile()
%timeit gp._objective(x)

1 loop, best of 3: 5.23 s per loop

gp._objective(x) computes the likelihood and its gradient, so it looks like GPflow is not a serious contender, but of course I may very well have done something wrong. Has anyone used GPflow before and know how it works?

Note: my tensorflow installation doesn't use the GPU, which seemed fair to the other packages, but of course it could also be interesting to try it with GPU support.

[chris-nemeth]

The GPy comparisons seem fair. It's possible that there's something we could do to improve the product, and possibly the sum kernels. I don't think anything special was done when implementing these composition kernels.

I've been working with GPflow a little recently, as one of my collaborators is the main developer. I'm still getting to grips with TensorFlow, but it looks like your implementation is correct. I think the problem you're seeing here is related to the initial cost of TensorFlow calculating the gradients and compiling the model. A follow-up comparison would be to see how the package works once the model is compiled (e.g. optimising the model parameters or using a gradient based MCMC algorithm).

There's a paper which gives some details on GPflow, https://arxiv.org/pdf/1610.08733.pdf. The two points that standout are that, (i) on CPUs the speed is comparable to GPy, which is fine if we're faster and (ii) it seems that the GPU implementation of Tensorflow does offer significant improvements. I think that for our purposes we can just focus on CPU performance.

[maximerischard]

I call the _compile() function just before timing the _objective() function, so I think what I'm timing already excludes the compilation check! It doesn't seem to be as fast as GPy.

[maximerischard]

Here's my attempt at doing the same thing in matlab:

run('gpml-matlab-v4.0-2016-10-19/startup.m')

nobsv=3000;
X = randn(nobsv,2);
Y = randn(nobsv);

meanfunc = [];                    % empty: don't use a mean function
covfunc = @covSEiso;              % Squared Exponental covariance function
likfunc = @likGauss;              % Gaussian likelihood

hyp = struct('mean', [], 'cov', [0 0], 'lik', 0);

gp(hyp, @infGaussLik, meanfunc, covfunc, likfunc, X, Y);

f = @() gp(hyp, @infGaussLik, meanfunc, covfunc, likfunc, X, Y);;
timeit(f)

ans =

    1.7894

I've not used matlab before, so let me know if anything looks off.

[maximerischard]

Here's the updated table:

N=3000	julia	GPy	gpml
se	0.93	1.25	1.65
rq	1.22	1.78	1.65
se+rq	1.50	1.92	1.85
se*rq	2.10	1.95	1.84
se+se2+rq	1.77	2.01	1.92
(se+se2)*rq	2.94	2.14	1.96
mask(se, [1])	1.03	1.31	1.73
mask(se, [1])+mask(rq, [2])	1.48	1.88	1.85
fix(se, σ)	0.90	1.27

the timings I get from matlab are pretty inconsistent, so I've used the lower number when I get different results. I've not worked out yet how to fix a hyperparameter, but I think this already gives us a good idea of how our package compares to gpml. Again, we're doing very well on simple problems, and not so great when product kernels are involved.

[fairbrot]

This is good news. Do you have scripts to automate the running of these tests and the compilation of the results? If so, it may be worthwhile including them somewhere in the perf directory. In any case, I would like to run these myself.

[maximerischard]

I've dumped my benchmarking notebooks (and nbconverted scripts) in https://github.com/STOR-i/GaussianProcesses.jl/tree/master/perf/benchmarks . Let me know if you have any trouble running them.

[maximerischard]

I've cleaned up some of the code in those benchmarks, and rerun them (now with d=10 instead of d=2). There's good news and bad news. The good news is there's been some improvements in GaussianProcesses.jl timings. The bad news (kind of), is that gpml is much faster than my previous benchmarks indicated, and is now the winner in most benchmarks. In a way, this is good news: it means our implementation could be faster. But I'm not quite sure with what black magic they use to achieve these times.

label	GaussianProcesses.jl	GPy	gpml
fix(se, σ)	750	1272
mat12	840	1254	847
se	842	1267	811
mask(se, [1])	877	1286	742
rq	1256	1820	782
se+rq	1579	1855	950
mask(se, [1])+mask(rq, [2:10])	1769	1882	902
se+se2+rq	1937	1977	1070
se*rq	2568	1946	943
(se+se2)*rq	4132	2063	991

[chris-nemeth]

I'm quite surprised by gpml. I wonder if it's worth looking at a few additional scenarios where we vary d and N. I'm not surprised by our performance on the sum and product kernels as I've noticed myself that they can be slow.

[nxtruong]

I've been using GPML for quite a while (and contributed a few speedup tricks to GMPL). I'm exploring Julia currently. Have you tried to profile the Julia and GPML code, to see which part of the GPML computation is faster than Julia (compute the covariance matrix, Cholesky decomposition)? My wild guess is the covariance matrix computation. As I understand, Julia encourages the use of loops, which often results in very fast code, while in Matlab, you need to vectorize. So your code fills the covariance matrix element-by-element, while GPML uses vectorized operations to fill the matrix at once. In this particular case for very large covariance matrices, probably Matlab's way is slightly faster. In addition, looking at the numbers, I think your RQ kernel implementation is much slower than GPML. Perhaps you can inspect your RQ kernel and see if you can improve its computation. Just my guess though.

P.S: many GPML kernel functions are implemented based on Mahalanobis distance computation. Not sure if it makes any difference; maybe it's one of their tricks?

[chris-nemeth]

Thanks for the suggestions, we'll look into these ideas. @maximerischard has spent more time than me on this, so he may have more insight as to where the bottleneck is. My guess would be the same as yours that the issue is related to the matrix computation as this is always the most costly step of GPs. I think what's most odd is the difference in the results between Jan 2017 and now.

[maximerischard]

Hi. Thank you for those insights into the matlab implementation @nxtruong. You're indeed right that most of the time in GaussianProcesses.jl is spent computing covariance matrices and their derivatives. Profiling the rational quadratic case, I get:

• 22% computation of the covariance matrix
• 65% computation of the gradients of the covariance matrix
• 13% linear algebra to compute the derivatives of the log likelihood

I don't know how easy profiling code is in matlab, but I'll have a look into it. One of the things I've noticed is that matlab passes the covariance to the gradient computation functions, which removes some redundancies. I don't really understand how the Mahalanobis distance idea works, could you shed some light on that @nxtruong? I don't think vectorization is likely to make matlab faster than julia, but hopefully profiling the matlab implementation will shed some more light on that.

@chris-nemeth, I think my earlier benchmarks of gpml were wrong. Again, I'm not a matlab programmer at all, so I made some mistakes. I had assumed that randn(100) creates a vector of length 100 in matlab, just as it does in julia, but apparently it creates a 100x100 matrix instead.

[maximerischard]

In another twist, I discovered than in gpml, calling the gp function with no outputs only computes the log likelihood, and not its derivatives. As far as I can tell this behaviour is not documented. We get a very different story once we also compute the derivative.

label	GaussianProcesses.jl	GPy	gpml
fix(se, σ)	750	1255
mat12	840	1254	1246
se	842	1225	1131
mask(se, [1])	877	1327	1075
rq	1256	1845	1292
se+rq	1579	1937	1679
mask(se, [1])+mask(rq, [2:10])	1769	1893	1659
se+se2+rq	1937	1953	2127
se*rq	2568	1929	1779
(se+se2)*rq	4132	2042	2206

However, I've switched to re-using a single dataset instead of generating random numbers for each benchmark, and I've noticed that I get different outputs. For the SE kernel, GPy and gpml give the same log-likelihood and derivatives, but GaussianProcesses.jl yields something different (mll=4676.5 instead of 4536.3). This would be worth investigating in my opinion.

[chris-nemeth]

Thanks for looking into this and it's certainly worth investigating the difference in the mll. Good spot on the derivatives, these results are a lot more in line with what I would have expected. Based on these results, it would seem that we should look at trying to make the gradient calculations faster.

[maximerischard]

I was setting the noise parameter wrong in my julia benchmarks, and I now get the same numbers (log likelihood and gradient) for all three packages. This doesn't affect the timings. I would say our main weakness remains composite kernels. I have some ideas I'm working on in the faster_sum branch, which I'm hoping will help in this respect.

[maximerischard]

I've now completed my work on the faster_sum branch. The new timings are below, with very pleasing improvements across the board. The fundamental idea is to get rid of the buffer arrays for covariance gradients, and instead do the computations online (see the new dmll_kern! function in GPE.jl). This allows a focus on the elementwise covariance and derivative functions, instead of operating at the matrix level. To make this work, I made composite kernels much more parametric. I've preserved the old types for comparison (for now), but the new types AddKern and MultKern now only take two kernels as argument, and are parametrized by the types of the component kernels. Similar changes were made to Masked kernels and Fixed kernels. Finally, to prevent slowdowns in Masked kernels, I've introduced static vector from the StaticArrays package. It's a lot of changes, but there's also some nice simplifications in the code that came out of it, which I think will make our lives easier in the future.

label	GaussianProcesses.jl	GPy	gpml
fix(se, σ)	705	1255
mask(se, [1])	812	1327	1075
se	834	1225	1131
mat12	835	1254	1246
rq	1252	1845	1292
se+rq	1382	1937	1679
mask(se, [1])+mask(rq, [2:10])	1638	1893	1659
se+se2+rq	1669	1953	2127
se*rq	2154	1929	1779
(se+se2)*rq	3191	2042	2206

[chris-nemeth]

That's fantastic! I take it we'll need to make the same changes to 'GPMC'?

[maximerischard]

GPMC still works (or at least all tests are passing, I don't know how comprehensive they are), but it'd be worth thinking about whether similar changes could be implemented to get faster gradient computations.

[maximerischard]

I've done some work this week-end on product kernels. The gradient computations for product kernels had a lot of redundancies. This required going back to computing the gradient as a vector rather than parameter-by-parameter. The change has the added benefit that it will make auto-differentiated kernels (#38) even easier to reimplement. Our package now provides the fastest implementation for every kernel in my benchmarks.

label	GaussianProcesses.jl	GaussianProcesses.jl gradients	GPy	gpml
fix(se, σ)	724	730	1255
se	831	800	1225	1131
mat12	862	836	1254	1246
mask(se, [1])	871	819	1327	1075
rq	1326	1252	1845	1292
se+rq	1450	1351	1937	1679
mask(se, [1])+mask(rq, [2:10])	1680	1562	1893	1659
se+se2+rq	1703	1682	1953	2127
se*rq	2170	1614	1929	1779
(se+se2)*rq	3188	1977	2042	2206

[chris-nemeth]

Fantastic!!!

[maximerischard]

Update with the latest master:

label	GaussianProcesses.jl	GaussianProcesses.jl gradients	GPy	gpml
fix(se, σ)	724	702	1255
se	831	811	1225	1131
mat12	862	804	1254	1246
mask(se, [1])	871	781	1327	1075
rq	1326	1165	1845	1292
se+rq	1450	1313	1937	1679
mask(se, [1])+mask(rq, [2:10])	1680	1392	1893	1659
se+se2+rq	1703	1551	1953	2127
se*rq	2170	1492	1929	1779
(se+se2)*rq	3188	1847	2042	2206

[maximerischard]

I've updated all benchmark code. In particular, I realized that because some of the parameters in the benchmarks were set to log(1)=0, matlab was artificially faster, as it seems to skip some of the multiplications. I've now set all parameter to 0.3. I also am now taking the minimum of 20 (rather than 10) runs, and I closed all other programs on my laptop to reduce interference. I've also updated gpml to version 4.2, and GPy to version 1.9.6. The results now look like this:

label	GaussianProcesses.jl	GPy	gpml
fix(se, σ)	688	1126
mask(se, [1])	784	1122	837
se	810	1073	833
mat12	835	1119	973
se+mat12	990	1201	1252
se*mat12	1115	1255	1349
rq	1136	1511	1097
se+rq	1288	1565	1385
mask(se, [1])+mask(rq, [2:10])	1339	1588	1385
se*rq	1469	1649	1477
se+mat12+rq	1513	1677	1803
(se+mat12)*rq	1828	1697	1891

I'm not sure what to attribute the significant speed-ups in gpml and GPy to, possibly they've implemented some optimizations in the last year.

[theogf]

GPyTorch would be a new serious candidate to consider! https://gpytorch.ai/

[dburov190]

What about the prediction times? I've bumped into a 10-fold slowdown compared to sklearn.
sklearn:
0.035790 seconds (2.08 k allocations: 43.422 KiB)
GaussianProcesses.jl:
0.498640 seconds (1.68 M allocations: 65.687 MiB, 2.69% gc time)
I'm using ScikitLearn.jl in the first case, and pure GaussianProcesses.jl code in the second one. I can provide an MWE if you want.

It would be great if you could look into that... For my purposes, I actually don't care about the learning but prediction (simply GP's mean) is needed several thousand times per computation so it becomes a huge bottleneck. Thanks!

[maximerischard]

That's interesting @dburov190. Are you obtaining the predictive mean vector and full covariance matrix from both sklearn and GaussianProcesses.jl? If you have an MWE, I would be interested to see it.

[dburov190]

That's interesting @dburov190. Are you obtaining the predictive mean vector and full covariance matrix from both sklearn and GaussianProcesses.jl? If you have an MWE, I would be interested to see it.

I believe I was computing mean and variance but not the full matrix. I only need means but there's no way of getting them alone in GaussianProcesses.jl, right? So, to make it fair, both mean and variance were computed in both cases.

I'll post an MWE a bit later, no problem.

[dburov190]

Below is an MWE (sorry for it taking me so long, I forgot about it), and this is what I usually see as output:

Timing sklearn:
0.091156 seconds (165 allocations: 26.297 KiB)
Timing GP.jl:
0.708394 seconds (4.08 M allocations: 141.961 MiB, 1.93% gc time)
true

Here's an MWE:

import GaussianProcesses
import NPZ
const GP = GaussianProcesses

### basic setup ###
using Random

rng = MersenneTwister(42)
N = 1000
x_samples = sort(rand(rng, N) * 16 .- 4)
y_samples = x_samples .* sin.(x_samples) .+ randn(N)
sample = [x_samples y_samples]

regularization_noise = 0.5

### GaussianProcesses.jl ###
GP_WK = GP.Noise(log(1.0))
GP_mean = GP.MeanZero()
GP_kernel = GP.SEIso(log(1.0), log(1.0)) + GP_WK
GP_regressor = GP.GP(
                     sample[:,1],
                     sample[:,2],
                     GP_mean,
                     GP_kernel,
                     log(sqrt(regularization_noise))
                    )
GP.optimize!(GP_regressor; noise = false) # do not optimize noise

### scikit-learn ###
using ScikitLearn
const sklearn = ScikitLearn

@sk_import gaussian_process : GaussianProcessRegressor
@sk_import gaussian_process.kernels : (RBF, Matern, WhiteKernel)

sk_WK = WhiteKernel(1, (1e-10, 10))
sk_kernel = 1.0 * RBF(1.0, (1e-10, 1e+6)) + sk_WK

sk_regressor = GaussianProcessRegressor(
                                        kernel = sk_kernel,
                                        n_restarts_optimizer = 7,
                                        alpha = regularization_noise
                                       )
sklearn.fit!(sk_regressor, sample[:,1:1], sample[:,2])




### comparison of prediction speed ###
xx = sort(rand(rng, N) * 12 .- 2)
xx_vec = reshape(xx, (N, 1))

# stupid JIT compilation
sk_regressor.predict(xx_vec, return_std = true)
GP.predict_y(GP_regressor, xx)

println("Timing sklearn:")
@time sk_m, sk_std = sk_regressor.predict(xx_vec, return_std = true)
println("Timing GP.jl:")
@time GP_m, GP_std = GP.predict_y(GP_regressor, xx)

using LinearAlgebra
println(isapprox(sk_m, GP_m, atol = 1e-6, norm = x -> norm(x, Inf)))

[maximerischard]

Thank you for the clean MWE, it made it very easy to try this out. I get similar timing results as you. The source of the problem is our assumption that obtaining 1,000 separate predictions with 1x1 covariance will be faster than obtaining a single prediction with 1000x1000 covariance and returning the diagonal. In fact calling GP.predict_y(GP_regressor, xx; full_cov=true) is a lot faster than GP.predict_y(GP_regressor, xx; full_cov=false). To match the sklearn speed, I had to do a bit more linear algebra:

module Diag
    using GaussianProcesses
    using GaussianProcesses: GPE, KernelData, cov, mean, EmptyData, cov_ij
    using LinearAlgebra
    using LinearAlgebra: dot, ldiv!
    import PDMats
    using PDMats: whiten!
    function predict_marginal!(Kxx, Kff, Kfx, mx, αf)
        nobs, npred = size(Kfx)
        mu = mx + Kfx' * αf
        whiten!(Kff, Kfx)
        @inbounds for k in 1:npred
            @simd for i in 1:nobs
                Kxx[k] -= Kfx[i,k]^2
            end
        end
        return mu, Kxx
    end

    function predict_marginal(gp::GPE, xpred::AbstractMatrix)
        dim, npred = size(xpred)
        meanf = gp.mean
        kernel = gp.kernel
        xtrain = gp.x
        alpha = gp.alpha
        Ktrain = gp.cK
        crossdata = KernelData(kernel, xtrain, xpred)
        Kfx = cov(kernel, xtrain, xpred, crossdata)
        mx = mean(meanf, xpred)
        Kpred = [cov_ij(kernel, xpred, xpred, EmptyData(), k, k, dim)
                 for k in 1:npred]
        predict_marginal!(Kpred, Ktrain, Kfx, mx, alpha)
    end
end

X = Matrix(xx')
Diag.predict_marginal(GP_regressor, X)
println("Timing predict_marginal:")
@time GP2_m, GP2_var = Diag.predict_marginal(GP_regressor, X);

with results

Timing sklearn:
  0.069160 seconds (112 allocations: 21.297 KiB)
Timing GP.jl:
  1.047039 seconds (70.02 k allocations: 19.326 MiB, 0.57% gc time)
Timing predict_marginal:
  0.047918 seconds (48 allocations: 15.292 MiB)

I will leave this here for now as I have run out of time, and this needs further thought before making changes to the package, but I hope this helps you.

Side-note:
I do not understand why you have a separate Noise kernel and a regularisation term. This should give the same results:

GP_simpler = GP.GP(
                     sample[:,1],
                     sample[:,2],
                     GP_mean,
                     GP.SEIso(log(1.0), log(1.0)),
                     log(sqrt(1.0))
                    )
GP.optimize!(GP_simpler)