deal with nans about hypertools

[jeremymanning]

is this still an issue? in the MATLAB version i dealt with nans as follows:

1.) remove all nans
2.) compute PCA transforms
3.) apply PCA transforms to the original (nan-included) data
4.) now the transformed data has nans in the same observations as the original data, and is the same size as the original data

another (fancier! better!) option is to use ppca (probabilistic pca): https://github.com/allentran/pca-magic

ppca is robust to nans and can even fill in missing data.

[andrewheusser]

not resolved yet! happy to implement either one/both. PPCA sounds neat but not confident I understand how it works entirely..

how about any([type(i,np.nan) for i in data]) use PPCA, else use PCA?

[andrewheusser]

with a warning message

[jeremymanning]

sure...that'll be faster than always using ppca. (are you thinking the warning will go with ppca-- i.e. something like "missing data: inexact solution"?)

the ppca paper is here: https://www.microsoft.com/en-us/research/publication/probabilistic-principal-component-analysis/

we can also talk about it with a whiteboard sometime and/or go through the paper together. the basic idea is:

1.) principle components (the "feature space") come from a unit Gaussian: p(z) = N(z | 0, I)
2.) the data come from weighted combinations of features, plus some noise: p(x | z) = N(x | Wz + mu, S), where S is a diagonal covariance matrix-- some constant, sigma^2 times the identify matrix

Then, given x (the data), we have to estimate z (the components) and W (the weights) using Bayes' rule. (For details, see above paper.)

[andrewheusser]

OK cool, thanks for the explanation. and yes, something like that for a warning