Hello, thank you for the KSG estimator implementation. I ran the estimator for a high-dimensional normal distribution as follows:

SIZE = 32
HALF_SIZE = int(SIZE / 2)
cov = np.eye(SIZE)
cov[:HALF_SIZE, HALF_SIZE:] = 0.5 * np.eye(HALF_SIZE)
cov[HALF_SIZE:, :HALF_SIZE] = 0.5 * np.eye(HALF_SIZE)

print("True:", 0.5 * log(1 / np.linalg.det(cov)))

points = np.random.multivariate_normal(np.zeros(SIZE), cov, size=5000)
X, Y = points[:, :HALF_SIZE], points[:, HALF_SIZE:]

print("Estimated:", kraskov_mi(X, Y, k=5))

The result was

True: 2.3014565796142468
Estimated: 0.7142441789816445

This result is consistent over multiple random samples. Can you provide any intuition on why the estimated mutual information is substantially lower than the true mutual information? Is the estimate normalized in some way I'm not seeing?

I'm using the natural logarithm, which was also used in the kraskov_mi estimator. I'm computing the true MI using the formula here.

Any help is appreciated 😄.

Problem

Thank you for your implementation of the seminal 2004 paper. Your code is pretty neat and well-documented, together with your 2016 contribution to the original paper.

However, when I tried to run your demo, I encountered some incorrect results, where the mutual information should have been non-negative (even not they are negative from a not very large margin from 0).

I(X;Y) =  -0.0004731760591245582
I(X;Y) =  -0.00016452501890240612
I(X;Y;Z) =  0.009085351036950406
I(X;Y;Z) =  0.013315288591181584

I am not sure whether this is the limitation of the originally proposed method. Further, is there any way to remedy this incorrectness?