flowersteam / explauto Goto Github PK

I am now looking into the testcases generation. I find the method to generate uniform testcases over the sensory space very appealing. Code is here: https://github.com/flowersteam/explauto/blob/master/explauto/environment/testcase.py

My understanding is that a grid of a given resolution is projected on the sensory space, and that each cell is associated with only one observation from within that cell. My questions concern the resolution parameter:

• is it the number of cut per dimension?
• what is the logical behind the automatic calculation of resolution: resolution = max(2, int((1.3*n)**(1.0/len(robot.s_feats)))) ?

I also noticed this: # TODO : change obs only if nearer from center of coo.

From what I understand is that in each cell, the corresponding observation will be the last observation encountered in the _populate process. The todo is to replace that by keeping the closest to the center of the cell?

in sensorimotor models there is a check for wether we do a forward or inverse prediction like

if in_dims == self.m_dims and out_dims == self.s_dims: # forward
elif in_dims == self.s_dims and out_dims == self.m_dims: # inverse

if s_dims and m_dims happen to be equal i guess this will fail?

Motor babbling is almost always better evaluate than goal babbling, whereas the latter covers the sensory space much better

Hi,

Right after a fresh install of explauto, running:

python -c "import explauto"

raises the following error:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "explauto/__init__.py", line 7, in <module>
    from .sensorimotor_model.sensorimotor_model import SensorimotorModel
  File "explauto/sensorimotor_model/__init__.py", line 7, in <module>
    module = importlib.import_module('explauto.sensorimotor_model.{}'.format(mod_name))
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "explauto/sensorimotor_model/non_parametric.py", line 5, in <module>
    from ..third_party.models.models.learner import Learner
ImportError: No module named third_party.models.models.learner

From what I remember, we removed the third_party package more than a year ago. However, it seems that it is still used in non_parametric.py(in particular the Learner class).

@pierre-rouanet: do you remember the status of this?
@sebastien-forestier: do you still use the non_parametric module? (e.g. for your LWR+CMAES inverse model?). Or do you rely on another module for LWR?

@jypuigbo: as a temporary fix you can:

• install the library in developer mode, as explained here. Then comment the problematic line

 File "explauto/sensorimotor_model/non_parametric.py", line 5, in <module>
    from ..third_party.models.models.learner import Learner

and reinstall again. The error should disappear (you won't be able to use the non_parametric module though. You can also check with @santmarti who recently installed it without mentioning this problem.

Cheers

@clement-moulin-frier: Looking into gmm_progress, I am wondering what the n_sample parameter stand for and what is a reasonable value for it.

https://github.com/flowersteam/explauto/blob/master/explauto/interest_model/gmm_progress.py#L38L46

From what I understood it is the size of the window used, that is only the last n_sample are considered when fitting the GMM. Hence the self.data[self.t % self.n_samples, ... with the % used as a rolling index?

The condition to trigger the update rule is also a bit hard to follow:

if abs(self.t % (self.n_samples * self.scale_t / 4.)) < self.scale_t:
            self.update_gmm()

I think it updates every n_samples/4 steps, because scale_t is always 1.
Does this mean the first updates are done while self.data is not fully filled yet, hence full of zero? What is the potential effect of this on the algorithm, I guess it will just put a gaussian at [0,0,...,0] and maybe not find it 'interresting'.
(See https://github.com/flowersteam/explauto/blob/master/explauto/interest_model/gmm_progress.py#L24: self.data = numpy.zeros((n_samples, len(expl_dims) + 2)))

I migth propose a simplification for this and add the update frequency in the parameters.

Change the relationship between the number of nearest neighbors and the number of gaussian. The number of nearest_neighbors should be inffered from the number of gaussians (fixed to 2 or 3) and the dimensionality.

I have met @sebastien-forestier last week and we have a proposition of architecture for the development of GMR. As it has been said before, it should be possible to use GMR in any case. That is why a GMR class will be created in models and will be herited from sklearn.mixture.GMM like GMM class from gmminf.py. Plus, it will contained regressions methods LSE, SLSE, stochastic sampling and the jonathan sampling method that has been described by @oudeyer inspired from @jgrizou noteboook . The number of gaussians are given in parameters.

NB :

• LSE : moyenne pondérée
• SLSE : consiste à faire de la régression LSE en utilisant seulement une Gaussienne: celle dont la projection sur y a le plus grand « poids ».
• Stochastic sampling : consiste à tirer le x dans la distribution P(x sachant y) encodée par le mélange de Gaussienne.
• Jonathan sampling (I do not know how to name it) : la méthode consistant à trouver le x qui maximise la probabilité P(x sachant y), ce dont parle Jonathan dans son notebook (mais Gharamani n’en parle pas). Pour cela il faut utiliser un alto d’optimisation stochastique,
  car pas de solution analytique. Méthode aussi utilisable pour apprendre directement des modèles inverses redondants.

A second class ILOGMR will be implemented and would be herited from SensorimotorModels abstract class . It will be available for inverse and forward model. Plus, users will choose the way they want to compute their inverse model (directly or using optimization).

Do you think this structure is fine ? Do I forget something ?

Me spamming again!

I looked quickly into gmminf.py

Are all the part of the code actually in use?

The conditional functions seem not in use inside gmminf as well as not in use in ilogmm. There are the following:

def conditional(mean, covar, dims_in, dims_out, covariance_type='full'):
    """ Return a function f such that f(x) = p(dims_out | dims_in = x) (f actually returns the mean and covariance of the conditional distribution
    """
    in_in = covar[ix_(dims_in, dims_in)]
    in_out = covar[ix_(dims_in, dims_out)]
    out_in = covar[ix_(dims_out, dims_in)]
    out_out = covar[ix_(dims_out, dims_out)]
    in_in_inv = inv(in_in)
    out_in_dot_in_in_inv = out_in.dot(in_in_inv)

    cond_covar = out_out - out_in_dot_in_in_inv.dot(in_out)
    cond_mean = lambda x: mean[dims_out] + out_in_dot_in_in_inv.dot(x - mean[dims_in])
    return lambda x: [cond_mean(x), cond_covar]

and

    def conditional(self, in_dims, out_dims):
        conditionals = []

        for k, (weight_k, mean_k, covar_k) in enumerate(self):
            conditionals.append(conditional(mean_k, covar_k,
                                            in_dims, out_dims,
                                            self.covariance_type))

        cond_weights = lambda v: [(weight_k * Gaussian(mean_k[in_dims].reshape(-1,),
                                  covar_k[ix_(in_dims, in_dims)]).normal(v.reshape(-1,)))
                                  for k, (weight_k, mean_k, covar_k) in enumerate(self)]

        def res(v):
            gmm = GMM(n_components=self.n_components,
                      covariance_type=self.covariance_type,
                      random_state=self.random_state, thresh=self.thresh,
                      min_covar=self.min_covar, n_iter=self.n_iter, n_init=self.n_init,
                      params=self.params, init_params=self.init_params)
            gmm.weights_ = cond_weights(v)
            means_covars = [f(v) for f in conditionals]
            gmm.means_ = array([mc[0] for mc in means_covars]).reshape(self.n_components,
                                                                       -1)
            gmm._set_covars(array([mc[1] for mc in means_covars]))
            return gmm

        return res

        self.in_dims = array(in_dims)
        self.out_dims = array(out_dims)
        means = zeros((self.n_components, len(out_dims)))
        covars = zeros((self.n_components, len(out_dims), len(out_dims)))
        weights = zeros((self.n_components,))
        sig_in = []
        inin_inv = []
        out_in = []
        mu_in = []
        for k, (weight_k, mean_k, covar_k) in enumerate(self):
            sig_in.append(covar_k[ix_(in_dims, in_dims)])
            inin_inv.append(matrix(sig_in).I)
            out_in.append(covar_k[ix_(out_dims, in_dims)])
            mu_in.append(mean_k[in_dims].reshape(-1, 1))

            means[k, :] = (mean_k[out_dims] +
                           (out_in *
                            inin_inv *
                            (value - mu_in)).T)

            covars[k, :, :] = (covar_k[ix_(out_dims, out_dims)] -
                               out_in *
                               inin_inv *
                               covar_k[ix_(in_dims, out_dims)])
            weights[k] = weight_k * Gaussian(mu_in.reshape(-1,),
                                             sig_in).normal(value.reshape(-1,))
        weights /= sum(weights)

        def p(value):
            # hard copy of useful matrices local to the function
            pass

        return p

The second one is specially cryptic to me, I don't understand what is going on there. Maybe those are archaeological remainings :D ?

Arguments can't be understood before an introduction to the general explauto architecture

• rebuild doc

I just tried to install explauto. To have import explauto working, I had to also install matplotlib and pypot. Should we add those to the dependency list?

Hi guys,

If I want to run an experiment over a few days, is there a way to stop, save, reload, and start again an experiment?

I found fast_forward(self, log) in experiment.py, it resets an experiment state given a log. Could you confirm this is all we need and thus the way to go?

If yes, I will implement a save and load function in the log class. What would be the best way in your opinion (pickle.. ) ?

Tutorials:

1. Setting a basic experiment (with default config)
   (+Q: why are the reached points are concentrated on the front (high x values) ?)
2. Setting a basic experiment with another config
3. Comparing exp. with pool
4. Using another env. (pypot & pendulum)
5. Customising conf, env and agents

Just looked into the competence_dist function: https://github.com/flowersteam/explauto/blob/master/explauto/interest_model/competences.py#L4

def competence_dist(target, reached, dist_min=0., dist_max=1.):
    return (min(- dist_min, - np.linalg.norm(target - reached))) / dist_max

I don't get this division by dist_max, what is it meant for? Also why is the division not casted in float?
It results in very weird behavior like:

competence_dist(0,0,2,10.0) -> -0.2 (expected -2 ?)
competence_dist(0,0,2,10) -> -1 (expected -2 ?)
competence_dist(0,100,0,1) -> -100 (expected -1 ?)

I would have assumed that we rather would like to bound the error between dist_min and dist_max, isn't it? But I might be missing something deeper here.

The patch would be:

def competence_dist(target, reached, dist_min=0., dist_max=1.):
    return max((min(- dist_min, - np.linalg.norm(target - reached))), -dist_max)

I also assume the move to negative domain is because competence should improve when error reduces.

It seems rather complex to make all environment works with processes. For instance, the pypot environment doesn't work.

One option would be to choose when running an ExperimentPool if you want to use Thread or Process.

Maybe the environment (models ?) should themselves declare if they support processes or not.

Because of the boundary constraints on the motor/sensory dimension, the sampling of a new motor/sensory command/goal is biased with increase probability to sample a point at the boundary.

This is due to line https://github.com/flowersteam/explauto/blob/master/explauto/interest_model/gmm_progress.py#L34L35 where the sampled point is constrained to be within the boudaries. A quick visualization of the problem is below.

It leads to the following patterns of goal selection in a 7 arm experiments. The right plot show in red the goal selected, we clearly see an oversampling on the boundaries, here [-1, 1] in each dimension.

The simplest solution to unbias this sampling step is to keep sampling until the point sampled is within the boundary. The resulting resampling strategy looks much better/less biased. It was run with the same seed, the effect is the same for many seeds.

This has been implemented in pull request #79 by:

• adding a resample_if_out_of_bounds arguments for the GmmInterest class, defaulting to False for consistency with previous version. I suggest to turn it True by default if you agree.
• updating the sample function accordingly
• adding a is_within_bounds function in utils/utils.py

Should we put explauto/sensorimotor_model/forward and inverse in explauto/models ?
Pro: models should be in /models
Con: forward and inverse models are only used in sensorimotor_model

@pierre-rouanet
@clement-moulin-frier

J'aimerais pouvoir porter sur Python le code SGIM-ACTS (https://flowers.inria.fr/Nguyen-Oudeyer-Paladyn2013.pdf).
J'aurai quelques questions concernant votre librairie exploauto.

Avez-vous déjà fait des tests en utilisant votre librairie et un agent pouvant changer de stratégie ?
(choisir activement entre différentes stratégies comme imitation, random, SAGG-RIAC)
Si oui, comment avez-vous appliquer les interest trees pour qu'ils représentent différentes stratégies en même temps ?
J'ai pas l'impression que le code soit compatible avec des actions complexes (séquences de taille variable composées de primitives motrices), pouvez-vous confirmer ?
Comme je devrais à terme utiliser des actions complexes, me conseillez-vous du coup d'utiliser votre librairie pour cela ?
Avez-vous également fait des tests pour un agent pouvant changer activement de task space ?

Voilà, j'hésite à m'efforcer de comprendre votre librairie et faire quelque chose de compatible ou faire quelque chose d'entièrement différent.

from a quick check it appears to me that explauto explicitly assumes a fixed starting state or, equivalently, kinematic motion?

explauto: m -> FWD -> s_{t+1}, s_{t+1} -> INV -> m

desired: (s_t, m) -> FWD -> s_{t+1}, (s_t, s_{t+1}) -> INV -> m) with (): vector concatenation

could this be done in a minimally invasive manner?

Hi guys,

I started using ILOGMR to compare it with various regression algortihm in sci-kit learn. For this I had to create a class wrapping the ilo_gmm from explauto to an ILOGMR estimator in sklearn.

Should I had this file in the explauto repo? If yes, where should it land?

Currently the class is as follow:

import numpy

from sklearn.base import BaseEstimator

from explauto.sensorimotor_model import ilo_gmm


class ILOGMR(BaseEstimator):

    def __init__(self, conf, n_components=3, n_neighbors=100, random_state=None):
        self.conf = conf
        self.explauto_ilo_gmm = ilo_gmm.IloGmm(conf, n_components)
        self.explauto_ilo_gmm.n_neighbors = n_neighbors
        self.random_state = random_state

    def fit(self, X, y):
        self.explauto_ilo_gmm.dataset.reset()
        for n in range(X.shape[0]):
            self.explauto_ilo_gmm.update(X[n, :], y[n, :])

    def compute_conditional_gmm(self, x):
        return self.explauto_ilo_gmm.compute_conditional_gmm(
            self.conf.m_dims, self.conf.s_dims, x)

    def predict(self, X):
        y_pred = numpy.zeros((X.shape[0], len(self.explauto_ilo_gmm.s_dims)))
        for n in range(X.shape[0]):
            gmm = self.compute_conditional_gmm(X[n, :])
            y_pred[n, :] = numpy.sum(gmm.means_.T * gmm.weights_)
        return y_pred

This use the new interface for ilo_gmm.py that is currently under pull request #49

It still lacks the set_param, get_parm method so we can use it with the really convenient GridSearchCV but it will come at some point.

Hi again,

I have been looking into sklearn recently and there is not gaussian mixture inference implemented. In explauto you implemented one, quite nicely by constructing on top of the GMM class from sklearn.

I think adding this to the official sklearn repo would be of tremendous help for many other people. Do you think the code is mature enough? Are there specific basic cases that are not handled? Is it worth our time?

In the same line, once gmminf is available, adding ilo_gmm shoudl be easy and could help diffusing this algortihm in the community.

I was wondering if the computation of progress in the discretized interest model is the intended one here.
Indeed, the np.cov is computing the covariance between [1, 2, 3, 4, 5] and the last competences e.g. [0.8, 0.6, 0.7, 0.9, 0.95].
An issue with that behavior is at the beginning of the exploration when few cells are sampled and few points are sampled in those cells.
The np.cov is in fact increasing even if progress is constant:

np.cov(range(3), [0.5, 0.6, 0.7])[0,1] = 0.099
np.cov(range(4), [0.5, 0.6, 0.7, 0.8])[0,1] = 0.166
np.cov(range(5), [0.5, 0.6, 0.7, 0.8, 0.9])[0,1] = 0.25

which leads to the exploration of n points in each randomly chosen cell with n corresponding roughly to the window's size.
That can be seen for instance in the notebook about curiosity in the scatter plots.

I tried another behavior in another branch here where the initial guessed progress of each unsampled cell is a constant (e.g. 10), and the progress is computed as the mean of the 5 last competencecs in the cell minus the mean of the 10 to 5 last ones.
In that case, there is an exploration biais that pushes to explore at least once in each cell before using progress based sampling.

What do you think ?
@clement-moulin-frier
@pierre-rouanet

Hi,

I am currently trying to run explauto on Poppy using your tutorial but I think it's not updated : poppy-software repository doesn't exist anymore. I think it has been replace by poppy-humanoid-beta so I cloned it and changed the config path '../../poppy-software/poppytools/configuration/poppy_config_linux.json' by '../poppy-humanoid-beta/software/poppytools/configuration/poppy_config.json'

But when I try to run

poppy = pypot.robot.from_json(config_path)

I got this

 File "", line 3, in 
    poppy = pypot.robot.from_json(config_path)

  File "/home/roxane/Logiciels/anaconda/lib/python2.7/site-packages/pypot/robot/config.py", line 244, in from_json
    return from_config(config, sync=sync, strict=strict, use_dummy_io=use_dummy_io)

  File "/home/roxane/Logiciels/anaconda/lib/python2.7/site-packages/pypot/robot/config.py", line 60, in from_config
    dxl_io = dxl_io_from_confignode(config, c_params, attached_ids, strict)

  File "/home/roxane/Logiciels/anaconda/lib/python2.7/site-packages/pypot/robot/config.py", line 139, in dxl_io_from_confignode
    port = pypot.dynamixel.find_port(ids, strict)

  File "/home/roxane/Logiciels/anaconda/lib/python2.7/site-packages/pypot/dynamixel/__init__.py", line 105, in find_port
    ids, list(set(ids) - set(ids_founds))))

IndexError: No suitable port found for ids [31, 32, 33, 34, 35, 36, 37, 41, 42, 43, 44, 51, 52, 53, 54]. These ids are missing [32, 33, 34, 35, 36, 37, 41, 42, 43, 44, 51, 52, 53, 54, 31] !

I use spyder from anaconda to compile my script.

Do you know how to fix it ?

Thanks,
Roxane

Remove logs into np arrays
Adapt signature to be called by Agent -> compute x_card from bounds

1. Each environment should provide a method to create its specific test cases (depending on the config).
2. When creating an experiment pool with the form_settings factory you can provide a test cases which will be used for all experiments
3. If you provide a test cases with different environment (or config) should raise an Error

Also warns (ValueError?) user when they want to use the same test cases with different environments.

Hi,
I have tried to install explauto from pip / easy_install and source but have this trouble ::
/usr/lib/python3.4/distutils/dist.py:260: UserWarning: Unknown distribution option: 'extra_require'
I have tried # pip install --upgrade setuptools
Requirement already up-to-date: setuptools in /usr/lib/python3.4/site-packages
Cleaning up...
but the trouble continue
Thanks for your help.