ggiecold-zz / pyrmt

Python for Random Matrix Theory: cleaning schemes for noisy correlation matrices.

pyRMT

Python for Random Matrix Theory. Implements several cleaning schemes for noisy correlation matrices, including the optimal shrinkage, rotationally-invariant estimator to an underlying correlation matrix (as proposed by Joel Bun, Jean-Philippe Bouchaud, Marc Potters and colleagues).

Such cleaned correlation matrix are known to improve factor-decomposition via Principal Component Analysis (PCA) and could be of relevance in a variety of contexts, including computational biology.

Cleaning schemes also result in much improved out-of-sample risk of Markowitz optimal portfolios, as established over the years in several papers by Jean-Philippe Bouchaud, Marc Potters and collaborators.

Some cleaning schemes can be easily adapted from the various shrinkage estimators implemented in the sklearn.covariance module (see the various publications by O. Ledoit and M. Wolf listed below).

In addition, it might make sense to perform an empirical estimate of a correlation matrix robust to outliers before proceeding with the cleaning schemes of the present module. Some of those robust estimates have been implemented in the sklearn.covariance module as well.

References

• "DISTRIBUTION OF EIGENVALUES FOR SOME SETS OF RANDOM MATRICES", V. A. Marcenko and L. A. Pastur, Mathematics of the USSR-Sbornik, Vol. 1 (4), pp 457-483
• "A well-conditioned estimator for large-dimensional covariance matrices", O. Ledoit and M. Wolf, Journal of Multivariate Analysis, Vol. 88 (2), pp 365-411
• "Improved estimation of the covariance matrix of stock returns with " "an application to portfolio selection", O. Ledoit and M. Wolf, Journal of Empirical Finance, Vol. 10 (5), pp 603-621
• "Financial Applications of Random Matrix Theory: a short review", J.-P. Bouchaud and M. Potters, arXiv: 0910.1205 [q-fin.ST]
• "Eigenvectors of some large sample covariance matrix ensembles", O. Ledoit and S. Peche, Probability Theory and Related Fields, Vol. 151 (1), pp 233-264
• "NONLINEAR SHRINKAGE ESTIMATION OF LARGE-DIMENSIONAL COVARIANCE MATRICES", O. Ledoit and M. Wolf, The Annals of Statistics, Vol. 40 (2), pp 1024-1060
• "Rotational invariant estimator for general noisy matrices", J. Bun, R. Allez, J.-P. Bouchaud and M. Potters, arXiv: 1502.06736 [cond-mat.stat-mech]
• "Cleaning large Correlation Matrices: tools from Random Matrix Theory", J. Bun, J.-P. Bouchaud and M. Potters, arXiv: 1610.08104 [cond-mat.stat-mech]
• "Direct Nonlinear Shrinkage Estimation of Large-Dimensional Covariance Matrices (September 2017)", O. Ledoit and M. Wolf https://ssrn.com/abstract=3047302 or http://dx.doi.org/10.2139/ssrn.3047302

Installing from PyPI

Try

pip install pyRMT

To install manually from the git repo, try this:

python setup.py install

The pyRMT codebase supports Python 2 and 3.

Usage

For more information on cleaning your correlation matrices, please refer to the docstrings to the various schemes implemented within pyRMT.

Attribution

If you happen to use pyRMT in your work or research, please cite its GitHub repository:

G. Giecold and L. Ouaknin, pyRMT, (2017), GitHub repository, https://github.com/GGiecold/pyRMT

The respective BibTex entry is

@misc{GiecoldOuaknin2017,
  author = {G. Giecold and L. Ouaknin},
  title = {pyRMT},
  year = {2017},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/GGiecold/pyRMT}}
}

License

Copyright 2017-2022 Gregory Giecold and contributors.

pyRMT is free software made available under the MIT License. For details see the LICENSE file.

pyrmt's Issues

Incorrect formulation of Stieltjes transform of re-scaled MP density

I might be mistaken, but I think the code on https://github.com/GGiecold/pyRMT/blob/master/pyRMT.py#L452-L455 does not reflect what is written in the green Box 1. on the last page of one of the referenced papers: https://www.cfm.fr/assets/ResearchPapers/2016-Cleaning-Correlation-Matrices.pdf

I think that
gmp = z + sigma_2 * (q - 1) - np.sqrt((z - lambda_N) * (z - lambda_plus))
should be
gmp = z + sigma_2 * (q - 1) - np.sqrt(z - lambda_N) * np.sqrt(z - lambda_plus)

Inverse Wishart regularisation

After applying the algorithm to very small portfolio of ~15 assets, I noticed that the covariance is very underestimated. For instance, correlation between MSCI WLD and S&P 500 was negative (!!)

Mean absolute error on cross validation on my data was 0.41.

After reviewing the literature, most notably https://arxiv.org/pdf/1610.08104.pdf : see 8.1.2. Regularizing the empirical RIE,There exists a regularization technique called Invesrse Wishart, that would correct estimation error on the smallest eigenvalues:
kappa=2*lambda_N/((1-q-lambda_N)**2-4*q*lambda_N) alpha_s=1/(1+2*q*kappa) denom=x/(1+alpha_s*(x-1.)) Gamma /= denom

After applying this technique, MAE on my sample fell to 0.381, and correlation between WLD and S&P went back to 0.73. Markowitz optimization was also improved.

Errors are still much worse than a regular empirical or scikit MinCovDet estimator.

Please advise.