Fix RBF controller about pilco

[kyr-pol]

Hi @voop, good catch, I'll get to it, thank you.

[nrontsis]

@kyr-pol I fixed the comment of @voop in branch rbf_controller.

However, it seems that the refactoring I made in the RBF controller (the one to avoid rewriting gp0) messed up with GPflow's optimization.

When trying to optimize the controller I get: No variables to optimize.

I think that whatever Param is inside a GPR model is not visible in the optimization of the policy. Maybe we could avoid having GPR models inside RBF but only use kernels?

[kyr-pol]

Hm if that is the case with Params inside GPR objects, yes it makes sense to just use the kernels. It is a bit weird and it seems worth looking into, but if we can get a quick fix we can look into it further later.

[petrikvladimir]

I think if ParamList from GPFlow is used instead of simple 'list' for storing models, it will be visible to the optimization.
However, changing it will cause other issues which I was not able to resolve...

[nrontsis]

Thanks @voop. I believe that we want to use ParamList only for the RbfController. This can be done by overriding the create_models function of MGPR.

I'll try to see if that works.

[nrontsis]

Okay, ParamList made the parameters visible to the optimiser. I still had to create a fake GPR class, where X and Y are Param and not DataHolders, as I think that GPflow does not optimises over DataHolders.

However, I am now getting NaNs on the cost of the policy. @kyr-pol can you take a look?

[kyr-pol]

Yes I will get to it later tonight and see what I can do.

[kyr-pol]

It seems that the NaNs were a numerical issue. By reducing the number of basis functions significantly (from 100 to 10, but 20 work too) the optimisation progresses.

Still, there are frequent failures, either in the Cholesky decompositions or the controller optimisation, which appear sometimes when using the LinearController as well. We should investigate these further.

[nrontsis]

Okay, that's good news. Let's close this and open a separate issue on numerical stability.