openidea-yunanuniversity / ycimpute Goto Github PK

AttributeError: 'Solver' object has no attribute 'masker'

感觉代码好多地方逻辑不通啊同学

伟大光荣，万能的hub主，能帮忙实现这篇论文吗？
A novel purity-based k nearest neighbors imputation method and its application in financial distress prediction
Adaptive multiple imputations of missing values using the class center
感激不尽！！

My data is stadarized numeric and the target is binary.
Before trying a logistic regression on my data, I want to choose the best imputation method, then I filtered on rows with no NaNs, made another one with the same shape with NaNs in random positions, and proceeded with show.analysiser
Data sampling:
`train_data_without_na = train_data_without_na.head(10000)
train_data_without_na_ = train_data_without_na.copy(deep=True)
home_services_full = train_data_without_na.as_matrix()

np.random.seed(100)
import random

def add_random_na(row):
vals = row.values
for _ in range(random.randint(0,len(vals)-15)):
i = random.randint(0,len(vals)-1)
vals[i] = np.nan
return vals

home_services_missing = train_data_without_na_.apply(add_random_na,axis=1)
home_services_missing = home_services_missing.as_matrix()`

the problem raises when running:
`import pandas as pd
from ycimpute.utils.shower import show

_result = show.analysiser(missing_X=sample_slice_missing, original_X=sample_slice_full)
pd.DataFrame.from_dict(_result,orient='index')`

`---------------------------------------------------------------------------
LinAlgError Traceback (most recent call last)
in ()
2 from ycimpute.utils.shower import show
3
----> 4 _result = show.analysiser(missing_X=sample_slice_missing, original_X=sample_slice_full)
5 pd.DataFrame.from_dict(_result,orient='index')

X:\ProgramData\Anaconda3\lib\site-packages\ycimpute\utils\shower\show.py in analysiser(missing_X, original_X)
62
63 #######################################################
---> 64 em_X_filled = EM().complete(copy.copy(missing_X))
65 em_filled_arr = em_X_filled[missing_index]
66 rmse_em_score = evaluate.RMSE(original_arr,em_filled_arr)

X:\ProgramData\Anaconda3\lib\site-packages\ycimpute\utils\tools.py in complete(self, X)
246 self._check_missing_value_mask(np.isnan(X))
247 col_type_dict = self._judge_type(X)
--> 248 X = self.solve(X)
249 return self.clip(X, col_type_dict)
250

X:\ProgramData\Anaconda3\lib\site-packages\ycimpute\unsupervised\expectation_maximization.py in solve(self, X)
60 theta = -np.inf
61 for iter in range(self.max_iter):
---> 62 updated_X = self._e_step(mu=mu, sigma=sigma, X=copy.copy(X))
63 mu, sigma, tmp_theta = self._m_step(updated_X)
64 for i in range(rows):

X:\ProgramData\Anaconda3\lib\site-packages\ycimpute\unsupervised\expectation_maximization.py in _e_step(self, mu, sigma, X)
36 loc_nan = np.isnan(X[sample,:])
37 new_mu = np.dot(sigma[loc_nan, :][:, ~loc_nan],
---> 38 np.dot(np.linalg.inv(sigma[~loc_nan, :][:, ~loc_nan]),
39 (X[sample, ~loc_nan] - mu[~loc_nan])[:,np.newaxis]))
40 nan_count = np.sum(loc_nan)

X:\ProgramData\Anaconda3\lib\site-packages\numpy\linalg\linalg.py in inv(a)
526 signature = 'D->D' if isComplexType(t) else 'd->d'
527 extobj = get_linalg_error_extobj(_raise_linalgerror_singular)
--> 528 ainv = _umath_linalg.inv(a, signature=signature, extobj=extobj)
529 return wrap(ainv.astype(result_t, copy=False))
530

X:\ProgramData\Anaconda3\lib\site-packages\numpy\linalg\linalg.py in _raise_linalgerror_singular(err, flag)
87
88 def _raise_linalgerror_singular(err, flag):
---> 89 raise LinAlgError("Singular matrix")
90
91 def _raise_linalgerror_nonposdef(err, flag):

LinAlgError: Singular matrix`

How can I proceed please?

I see RandomForestClassifier in the iterforest module, but I think there is no imputer algorithm for the classification problem. Are you planning to integrate?

您好，我使用您提工的方法导入鸢尾花数据集使用mida出现报错：
X_filled = MIDA().complete(iris_miss)
File "F:\mini_conda\envs\GAT-TSP\lib\site-packages\ycimpute-0.2-py3.6.egg\ycimpute\utils\tools.py", line 86, in complete
File "F:\mini_conda\envs\GAT-TSP\lib\site-packages\ycimpute-0.2-py3.6.egg\ycimpute\imputer\mida.py", line 83, in solve
File "F:\mini_conda\envs\GAT-TSP\lib\site-packages\ycimpute-0.2-py3.6.egg\ycimpute\imputer\mida.py", line 60, in training
ZeroDivisionError: integer division or modulo by zero
使用MissForest出现报错：
X_filled =iterforest.MissForest().complete(iris_miss)
File "F:\mini_conda\envs\GAT-TSP\lib\site-packages\ycimpute-0.2-py3.6.egg\ycimpute\utils\tools.py", line 86, in complete
File "F:\mini_conda\envs\GAT-TSP\lib\site-packages\ycimpute-0.2-py3.6.egg\ycimpute\imputer\iterforest.py", line 106, in solve
File "F:\mini_conda\envs\GAT-TSP\lib\site-packages\ycimpute-0.2-py3.6.egg\ycimpute\imputer\iterforest.py", line 121, in _lose_func
IndexError: index 69 is out of bounds for axis 0 with size 60
使用其他的数据能正常运行，应该对于一些特定数据无法处理导致的报错

Hi. I have setup the library using pip and have managed to write down a simple test for MICE imputation as you have suggested in your docs. But I can not seem to get it to run. I am running the code in Pycharm 3.3 IDE in Ubuntu 17.10.

Here is my console output:

<Connected to pydev debugger (build 173.4301.16)
/home/sundus/.local/lib/python2.7/site-packages/pandas/_libs/init.py:4: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from .tslib import iNaT, NaT, Timestamp, Timedelta, OutOfBoundsDatetime
/home/sundus/.local/lib/python2.7/site-packages/pandas/init.py:26: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from pandas._libs import (hashtable as _hashtable,
/home/sundus/.local/lib/python2.7/site-packages/pandas/core/dtypes/common.py:6: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from pandas._libs import algos, lib
/home/sundus/.local/lib/python2.7/site-packages/pandas/core/util/hashing.py:7: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from pandas._libs import hashing, tslib
/home/sundus/.local/lib/python2.7/site-packages/pandas/core/indexes/base.py:6: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from pandas._libs import (lib, index as libindex, tslib as libts,
/home/sundus/.local/lib/python2.7/site-packages/pandas/core/tools/datetimes.py:6: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from pandas._libs.tslibs.strptime import array_strptime
/home/sundus/.local/lib/python2.7/site-packages/pandas/tseries/frequencies.py:24: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from pandas._libs.tslibs.frequencies import ( # noqa
/home/sundus/.local/lib/python2.7/site-packages/pandas/core/indexes/datetimelike.py:28: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from pandas._libs.period import Period
/home/sundus/.local/lib/python2.7/site-packages/pandas/core/sparse/array.py:33: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
import pandas._libs.sparse as splib
/home/sundus/.local/lib/python2.7/site-packages/pandas/core/window.py:36: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
import pandas._libs.window as _window
/home/sundus/.local/lib/python2.7/site-packages/pandas/core/groupby.py:68: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from pandas._libs import lib, groupby as libgroupby, Timestamp, NaT, iNaT
/home/sundus/.local/lib/python2.7/site-packages/pandas/core/reshape/reshape.py:31: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from pandas._libs import algos as _algos, reshape as _reshape
/home/sundus/.local/lib/python2.7/site-packages/pandas/io/parsers.py:45: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
import pandas._libs.parsers as parsers
/usr/local/lib/python2.7/dist-packages/h5py/init.py:36: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from ._conv import register_converters as _register_converters
/usr/local/lib/python2.7/dist-packages/h5py/init.py:45: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from . import h5a, h5d, h5ds, h5f, h5fd, h5g, h5r, h5s, h5t, h5p, h5z
/usr/local/lib/python2.7/dist-packages/h5py/_hl/group.py:22: RuntimeWarning: numpy.dtype size changed, may indicate binary incompatibility. Expected 96, got 88
from .. import h5g, h5i, h5o, h5r, h5t, h5l, h5p
Traceback (most recent call last):
File "/home/sundus/Documents/pycharm-professional-2017.3.3/pycharm-2017.3.3/helpers/pydev/pydevd.py", line 1668, in
main()
File "/home/sundus/Documents/pycharm-professional-2017.3.3/pycharm-2017.3.3/helpers/pydev/pydevd.py", line 1662, in main
globals = debugger.run(setup['file'], None, None, is_module)
File "/home/sundus/Documents/pycharm-professional-2017.3.3/pycharm-2017.3.3/helpers/pydev/pydevd.py", line 1072, in run
pydev_imports.execfile(file, globals, locals) # execute the script
File "/home/sundus/PycharmProjects/untitled/test_ycimpute.py", line 6, in
from ycimpute.imputer.mice import MICE
File "/home/sundus/.local/lib/python2.7/site-packages/ycimpute/imputer/init.py", line 2, in
from ..unsupervised.expectation_maximization import EM
File "/home/sundus/.local/lib/python2.7/site-packages/ycimpute/unsupervised/init.py", line 2, in
from .expectation_maximization import EM
File "/home/sundus/.local/lib/python2.7/site-packages/ycimpute/unsupervised/expectation_maximization.py", line 2, in
from ..utils.tools import Solver
File "/home/sundus/.local/lib/python2.7/site-packages/ycimpute/utils/tools.py", line 12
fill_method: object = "zero",
^
SyntaxError: invalid syntax

Process finished with exit code 1>

请问EM算法如何使用
solve函数里面的missing_mask是什么

The calculate principle of 'mu' should be improved

I've added some codes about categorical prediction in recent months. :) But of course I can't use it in the package right now. Will you get a new release? I would love to install the package with pip install ycimpute and use for categorical prediction.

Thank you for sharing the implementation. How to use the algorithms if we have categorical (nominal) variables in our data? Do we need to encode them first?