jamesrobertlloyd / gpss-research

Kernel structure discovery research code - likely to be unstable

This is part of the automatic statistician project

Automatic Bayesian Covariance Discovery

This repo contains the source code to run the system described in the paper

Automatic Construction and Natural-Language Description of Nonparametric Regression Models by James Robert Lloyd, David Duvenaud, Roger Grosse, Joshua B. Tenenbaum and Zoubin Ghahramani, appearing in AAAI 2014.

Abstract

This paper presents the beginnings of an automatic statistician, focusing on regression problems. Our system explores an open-ended space of statistical models to discover a good explanation of a data set, and then produces a detailed report with figures and natural-language text. Our approach treats unknown regression functions nonparametrically using Gaussian processes, which has two important consequences. First, Gaussian processes can model functions in terms of high-level properties (e.g. smoothness, trends, periodicity, changepoints). Taken together with the compositional structure of our language of models this allows us to automatically describe functions in simple terms. Second, the use of flexible nonparametric models and a rich language for composing them in an open-ended manner also results in state-of-the-art extrapolation performance evaluated over 13 real time series data sets from various domains.

Feel free to email the authors with any questions:
James Lloyd ([email protected])
David Duvenaud ([email protected])
Roger Grosse ([email protected])

Data used in the paper

• Full time series
• Extrapolation
• Interpolation

Related Repo

Source code to run an earlier version of the system, appearing in Structure Discovery in Nonparametric Regression through Compositional Kernel Search by David Duvenaud, James Robert Lloyd, Roger Grosse, Joshua B. Tenenbaum, Zoubin Ghahramani
can be found at

github.com/jamesrobertlloyd/gp-structure-search/.

gpss-research's Issues

Changepoints confuse discussion of monotonicity

Before computing stats python can work out where a kernel applies - potentially even a sum of kernels

Check that python is using the random seed correctly

Will it ever be useful to separate the constant component of the linear kernel?

The marginal variance of covCentredPeriodic is not the output_variance

Reparametrise it

Unexpected number of expansions in additive mode

/scratch/home/Research/GPs/gpss-research/experiments/2013-09-26.py

( M(0, SE(ell=0.3, sf=5.5)) + ( M(0, FT(ell=-1.9, p=-0.0, sf=3.2)) x M(0, LN(off=-0.8, ell=1.3, loc=1950.7)) ) )

yielded as much as

( M(0, SE(ell=0.3, sf=5.5)) + M(0, FT(ell=-1.9, p=-0.0, sf=3.2)) )

Seems wrong - but I might have been mistaken

If we want an additive model should we always have purely additive components?

Might help for parsimony but does not feel like the right way forward

Blackout (and others) are numerically unstable

Derivative w.r.t location can be infty * 0 - can either be fixed by changing order of calculation or by thresholding quantities by realmax

Another (fiddly) way would be to use signed log transforms

Remove redundancy operation should occur after random restarts

Make more things experiment parameters

Search operators
Whether or not to include the MAE best kernel in the search as well as marginal likelihood

Kernels should not have scale factors

Only the constant kernel should - fits better with symbolic regression grammars

Try the log transform of some data

A teaser for learning output warping or a demonstration of deficiency?

How would we compare marginal likelihoods? Check out the warped GP paper.

Should the search always use the best kernel found so far for expansion?

Unstandardised data?

Most aspects of the algorithm can scale appropriately but hard to control everything - should we just standardise data before running the search?

Translation should recognise soft changepoints and change its text accordingly

Make suitable for multi-d again

Changepoints etc. should select a dimension to act upon (but should pass all data shape and variables downstream)

The 10 fold cross validation needs to be updated

The new data shape parameters need to behave correctly

SE*Per translation is broken

Better to think of it as variable phase - comment about the period in terms of Fourier transforms

Output variances stored on log scale - make sure summary routines aware of this

e.g. product kernel incorrectly uses output_variance

Combine anticorrelated components

If the sum of two kernel components dramatically reduces uncertainty (at points where the uncertainty is greater than zero e.g. blackouts / changepoints) then these components probably belong together e.g. A + A + B -> 2A + B

Reintroduce the laplace approx

Will increase the need for anticorrelated component detection (and combination) since laplace will recognise this as being ok

Hessian should be computed on full data when subsetting?

Or only when a few full data iters are done as well?

XKCDify output

SE lengthscale restarts

Should sometimes be very large e.g. twice the data range - this is the neutral value (ie.. inifintiy)

Translation should comment about the future

e.g. SE * Per - the posterior mean tends to zero but this is just due to uncertainty about the period

In general, SE*Per should talk about a range of plausible periods

Cosine kernel period out by factor of 2

Whoops!

Pick best marginal likelihoods over several runs of the algorithm

Solar takes a surprisingly long time to expand at lower levels

Is there a bug in the change window expansion code?

Expand all subkernel expressions

e.g. should consider the expansion A + B + C -> (A + B) * D + C

Logistic blackout kernel still numerically unstable

Mixture of lengthscales kernel

Rather than the broad mixture that is RQ - maybe try a tighter mixture e.g. a Gaussian centred on a particular lengthscale?

R2 can be negative when combined with changepoints

Need to restrict to relevant regions - plug in a constant to the changepoints to see where the variance should be measured?

Use spectrogram / other approaches to find candidate frequencies

Periodic components seem the most difficult to find - probably requiring good initial values of hyperparameters

Linear kernel translation sometimes wrong

Need to look at gradient - can't just re-use SE*something logic

Add an exponential function?

SE*Lin - linear standard deviation

Not variance - change the text accordingly

English language kernel descriptions

Limit the number of decomposition components plotted

Bunch all the other components together when demonstrating a decomposition

Plot decomp not stable when searching over noise

Remove redundancy from additive kernel expressions

e.g. multiplying by const, SE*SE

Safest to do when in additive form since it won't affect the search

Fourier kernel not initisliased properly and numerically unstable

Lengthscale parameter is invariant to scale - relative to period - default value of zero makes sense - also check that no out of bounds checks are made

It is also numerically unstable - past a certain lengthscale the function might as well just call CovCos

Superimpose approximately periodic components to visualise the trend

e la Bayesian Data Analysis 3

Allow min period heuristic type to be an experiment parameter

e.g. when data is known to be on a regular grid but is sparse

Linear kernel restarts not great

Location should place mass outside of data range since this is what results in linearly increasing / decreasing variance

Sum, Product, Changepoint should be operators

This will likely tidy up some duplicate code since we know that all operators have operands etc

Also - if we record properties like commutativity / distributivity etc. we can abstract their behaviour.

Does the jitter heuristic work

Does the search multiply by Lin again?
Is the jitter size correct (too big and we lose optimised values - too small and spurious Lins will appear again)

Optimise parameters using random subset of data

For speed, but then compute nll using full data - randomising the subset should also guard against an unlucky subset being chosen at the beginning of the search

Mask kernels should allow None dimensions

e.g. for Const kernel which does not depend on dimension

Alternatively - we should not always use masks - only when appropriate

One of these solutions needed to make hashing in multi-d correct

cblparallel should sanitise its code input

% signs!

Changepoint etc. kernels are slow

Use bsxfun and generally optimise the formulae

More data

Earthquakes
EEG
Changepoint papers?
Fault detection papers?
Multiresolution paper?
Fix some of the current data sets that were subsampled.

Noise should be associated with kernels

It is optimised along with the other parameters and should be treated similarly (e.g. jitter)