2-2000x faster ML algos, 50% less memory usage, works on all hardware - new and old.
Home Page: https://hyperlearn.readthedocs.io/en/latest/index.html
License: Apache License 2.0
At the end of python setup.py install, it says to run
#### If you want to compile Numba code, please run:
>>>> python numba_compile.py
to compile Numba. Doing this gives me
Progress: |||||||||||||||Traceback (most recent call last):
File "numba_compile.py", line 6, in <module>
from hyperlearn.numba.funcs import *
File "/home/share/software/sources/hyperlearn/hyperlearn/numba/funcs.py", line 4, in <module>
from ..cfuncs import uinteger
File "/home/share/software/sources/hyperlearn/hyperlearn/cfuncs.py", line 3, in <module>
from .cython.base import isComplex, available_memory, isList
ModuleNotFoundError: No module named 'hyperlearn.cython.base'
do you mind to pass to a MIT/BSD license instead of GPL?
This is just a post to highlight how the problem has been solved. Building locally works, however within RTD, the error showcases WARNING: autodoc: failed to import module 'hyperlearn.base'; the following exception was raised:
No module named 'hyperlearn.cython.base'
I've checked: https://stackoverflow.com/questions/13238736/how-to-document-cython-function-on-readthedocs/13364247, https://stackoverflow.com/questions/13372068/importerror-for-my-code-on-readthedocs, https://stackoverflow.com/questions/10324393/sphinx-build-fail-autodoc-cant-import-find-module, readthedocs/readthedocs.org#3634, readthedocs/readthedocs.org#2511, https://docs.readthedocs.io/en/latest/faq.html, astropy/astropy-healpix#4, and mozilla/treeherder@f17bcf8.
All didn't solve the problem. To solve it, in conf.py change
sys.path.insert(0, os.path.abspath('.')) sys.path.insert(0, os.path.abspath('..')) sys.path.insert(0, os.path.abspath('../../'))
to sys.path.insert(0, os.path.abspath('.hyperlearn'))
Only the one line is enough to fix the problem!
Hey! I started documentation @ https://hyperlearn.readthedocs.io/en/latest/index.html
Hello all. I was a bit busy during December - May due to uni (I'm a 4th year student :) ) + I'm part of NVIDIA's cuML team (check out cuML's algos! Theyre super fast on GPUs!)
In terms of plans, I have a 3 week "holiday", so will maintain HyperLearn more. I've had many things I wanted to do, and plan to make things runnable without compiling via C, C++ etc.
My main aims are:
Delete all C/C++ dependencies, and rely solely on Numba + Scipy LAPACK.
Focus on specific algos that are extremely relevant.
a. Randomized + Dense PCA on Sparse Matrices (without converting sparse to dense). decomposition.PCA
b. Porting Eigendecomposition via MRRR for the top K eigenvectors. [Scipy still hasn't..] decomposition.EIGH
c. Fixing all memory copies for SVD, and support Randomized SVD on sparse matrices. decomposition.SVD
d. Fix up LinearSolve and place it into 1 module with LSMR (super fast sparse solve), combine with Cholesky, SVD, Eig, etc solving + Ridge. linear_model.solve
e. And finally, introduce a Python only modified version of Spotify's ANNOY library (a limited nearest neighbor KD-Tree based on other heuristics I found to be useful).
If anyone has other suggestions, please mention them! If anyone wants to help, please msg / email me @ [email protected]!
Thanks for all the support!
Hi there,
Thanks for the package! It looks very cool and seems like you put alot of work into it. Kudos!!! Sadly I can't find installation instructions anywhere. Can you point me to them?
Thanks,
Found some really good material at some site (will add here later).
Shows how to calculate using IRLS and using the Hessian for Newton-Raphson's method.
Hey Contributor!
Thanks for checking out HyperLearn!! Super appreciate it.
Since the package is new (only started like August 27th)..., Issues are the best place to start helping out, and or check out the Projects tab. There's a whole list of stuff I envisioned to complete.
Also, if you have a NEW idea: please post an issue and label it new enhancement.
In terms of priorities, I wanted to start from the bottom up, as to make all functions faster; that means:
Since Singular Value Decomp is the backbone for nearly all Linear algos (PCA, Linear, Ridge reg, LDA, QDA, LSI, Partial LS, etc etc...), we need to focus on making SVD faster! (Also Linear solvers).
When SVD optimization is OK, then slowly creep into Least Squares / L1 solvers. These need to be done before other algorithms in order for speedups to be apparent.
If NUMBA code is used, it needs to be PRE-COMPILED in order to save time, or else we need to wait a whopping 2-3 seconds before each call...
Then, focus on Linear Algorithms, including but not limited to:
numba.errors.LoweringError: Failed at nopython (nopython mode backend)
Buffer dtype cannot be buffer
File "hyperlearn/numba.py", line 126:
def squaresum(v):
s = v[0]**2
for i in prange(1,len(v)):
^
[1] During: lowering "id=6[LoopNest(index_variable = parfor_index.395, range = ($const28.2, $28.5, 1))]{44: <ir.Block at /home/lee/hyperlearn/hyperlearn/numba.py (126)>}Var(parfor_index.395, /home/lee/hyperlearn/hyperlearn/numba.py (126))" at /home/lee/hyperlearn/hyperlearn/numba.py (126)
It seems like the RAM consumption refers to the CPU RAM, right?
However, GPU computation is highly limited to its memory capacity.
Computation over super large dataset such as K-Means/t-SNE tends to cause out-of-memory w/o careful breakdown.
Does HyperLean specifically deal with super large input (Hosted on CPU memory but cannot reside on GPU memory all at once) regardless GPU memory size limit?
After python setup.py install I get
ModuleNotFoundError: No module named 'hyperlearn.cython'
when trying to import hyperlearn.(random|linalg|…). What could the problem be?
At Moonshot, our mission is to predict the future of everything quickly and make JARVIS a reality!
I was on hiatus in late 2018 to early 2020 since I was at NVIDIA helping make GPU ML algos faster!
I incorporated Hyperlearn's methods to make TSNE 2000x faster and various other ML algos faster!
From 2020 to now, I've been working on Moonshot with my brother, so I didn't have too much time to maintain Hyperlearn!
But things been catching up so a polished version will be published mid next year!!
Did you know Hyperlearn's methods has been used / mentioned in 5 research papers from Microsoft, UW, Greece, UC Berkeley and NVIDIA? Also Hyperlearn's methods and effort were incorporated into Cupy, Pytorch, Scipy, UNSW and GCC?
Around mid 2022, Hyperlearn will evolve to GreenAI and aims to incorporate:
It would be nice to have PyPI package, so it can be installed using pip.
Hope this works!
I installed the hpyerlearn successfully usingpython setup.py install. It shows:
#### Welcome to Umbra's HyperLearn! ####
#### During installation, code will be compiled down to C / LLVM via Numba. ####
#### This could mean you have to wait...... ####
#### You MUST have a C compiler AND MKL/LAPACK enabled Scipy. ####
#### If you have Anaconda, then you are set to go! ####
running install
running bdist_egg
running egg_info
writing manifest file 'hyperlearn.egg-info/SOURCES.txt'
running install_lib
running build_py
creating build/bdist.linux-x86_64/egg
creating build/bdist.linux-x86_64/egg/hyperlearn
byte-compiling build/bdist.linux-x86_64/egg/hyperlearn/__init__.py to __init__.cpython-37.pyc
creating build/bdist.linux-x86_64/egg/EGG-INFO
copying hyperlearn.egg-info/PKG-INFO -> build/bdist.linux-x86_64/egg/EGG-INFO
copying hyperlearn.egg-info/SOURCES.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying hyperlearn.egg-info/dependency_links.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying hyperlearn.egg-info/not-zip-safe -> build/bdist.linux-x86_64/egg/EGG-INFO
copying hyperlearn.egg-info/requires.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
copying hyperlearn.egg-info/top_level.txt -> build/bdist.linux-x86_64/egg/EGG-INFO
removing 'build/bdist.linux-x86_64/egg' (and everything under it)
creating /home/lksgcc/.pyenv/versions/anaconda3-5.0.1/envs/sgcc36/lib/python3.7/site-packages/hyperlearn-0.0.1-py3.7.egg
Extracting hyperlearn-0.0.1-py3.7.egg to /home/lksgcc/.pyenv/versions/anaconda3-5.0.1/envs/sgcc36/lib/python3.7/site-packages
byte-compiling /home/lksgcc/.pyenv/versions/anaconda3-5.0.1/envs/sgcc36/lib/python3.7/site-packages/hyperlearn-0.0.1-py3.7.egg/hyperlearn/__init__.py to __init__.cpython-37.pyc
Adding hyperlearn 0.0.1 to easy-install.pth file
Installed /home/lksgcc/.pyenv/versions/anaconda3-5.0.1/envs/sgcc36/lib/python3.7/site-packages/hyperlearn-0.0.1-py3.7.egg
Processing dependencies for hyperlearn==0.0.1
Searching for Cython==0.29
Best match: Cython 0.29
Adding Cython 0.29 to easy-install.pth file
Installing cygdb script to /home/lksgcc/.pyenv/versions/anaconda3-5.0.1/envs/sgcc36/bin
Installing cython script to /home/lksgcc/.pyenv/versions/anaconda3-5.0.1/envs/sgcc36/bin
Installing cythonize script to /home/lksgcc/.pyenv/versions/anaconda3-5.0.1/envs/sgcc36/bin
Using /home/lksgcc/.pyenv/versions/anaconda3-5.0.1/envs/sgcc36/lib/python3.7/site-packages
Searching for psutil==5.7.0
Best match: psutil 5.7.0
Adding psutil 5.7.0 to easy-install.pth file
Using /home/lksgcc/.pyenv/versions/anaconda3-5.0.1/envs/sgcc36/lib/python3.7/site-packages
Searching for numba==0.55.2
Best match: numba 0.55.2
Adding numba 0.55.2 to easy-install.pth file
Using /home/lksgcc/.pyenv/versions/anaconda3-5.0.1/envs/sgcc36/lib/python3.7/site-packages
Searching for torch==1.6.0
Best match: torch 1.6.0
Adding torch 1.6.0 to easy-install.pth file
Installing convert-caffe2-to-onnx script to /home/lksgcc/.pyenv/versions/anaconda3-5.0.1/envs/sgcc36/bin
Installing convert-onnx-to-caffe2 script to /home/lksgcc/.pyenv/versions/anaconda3-5.0.1/envs/sgcc36/bin
Using /home/lksgcc/.pyenv/versions/anaconda3-5.0.1/envs/sgcc36/lib/python3.7/site-packages
Searching for pandas==1.3.5
Best match: pandas 1.3.5
Adding pandas 1.3.5 to easy-install.pth file
Using /home/lksgcc/.pyenv/versions/anaconda3-5.0.1/envs/sgcc36/lib/python3.7/site-packages
Searching for scipy==1.7.3
Best match: scipy 1.7.3
Adding scipy 1.7.3 to easy-install.pth file
Using /home/lksgcc/.pyenv/versions/anaconda3-5.0.1/envs/sgcc36/lib/python3.7/site-packages
Searching for scikit-learn==1.0.2
Best match: scikit-learn 1.0.2
Adding scikit-learn 1.0.2 to easy-install.pth file
Using /home/lksgcc/.pyenv/versions/anaconda3-5.0.1/envs/sgcc36/lib/python3.7/site-packages
Searching for torchvision==0.7.0+cu101
Best match: torchvision 0.7.0+cu101
Adding torchvision 0.7.0+cu101 to easy-install.pth file
Using /home/lksgcc/.pyenv/versions/anaconda3-5.0.1/envs/sgcc36/lib/python3.7/site-packages
Searching for numpy==1.21.6
Best match: numpy 1.21.6
Adding numpy 1.21.6 to easy-install.pth file
Installing f2py script to /home/lksgcc/.pyenv/versions/anaconda3-5.0.1/envs/sgcc36/bin
Installing f2py3 script to /home/lksgcc/.pyenv/versions/anaconda3-5.0.1/envs/sgcc36/bin
Installing f2py3.7 script to /home/lksgcc/.pyenv/versions/anaconda3-5.0.1/envs/sgcc36/bin
Using /home/lksgcc/.pyenv/versions/anaconda3-5.0.1/envs/sgcc36/lib/python3.7/site-packages
Searching for llvmlite==0.38.1
Best match: llvmlite 0.38.1
Adding llvmlite 0.38.1 to easy-install.pth file
Using /home/lksgcc/.pyenv/versions/anaconda3-5.0.1/envs/sgcc36/lib/python3.7/site-packages
Searching for setuptools==62.3.2
Best match: setuptools 62.3.2
Adding setuptools 62.3.2 to easy-install.pth file
Using /home/lksgcc/.pyenv/versions/anaconda3-5.0.1/envs/sgcc36/lib/python3.7/site-packages
Searching for future==0.18.2
Best match: future 0.18.2
Adding future 0.18.2 to easy-install.pth file
Installing futurize script to /home/lksgcc/.pyenv/versions/anaconda3-5.0.1/envs/sgcc36/bin
Installing pasteurize script to /home/lksgcc/.pyenv/versions/anaconda3-5.0.1/envs/sgcc36/bin
Using /home/lksgcc/.pyenv/versions/anaconda3-5.0.1/envs/sgcc36/lib/python3.7/site-packages
Searching for python-dateutil==2.8.2
Best match: python-dateutil 2.8.2
Adding python-dateutil 2.8.2 to easy-install.pth file
Using /home/lksgcc/.pyenv/versions/anaconda3-5.0.1/envs/sgcc36/lib/python3.7/site-packages
Searching for pytz==2022.1
Best match: pytz 2022.1
Adding pytz 2022.1 to easy-install.pth file
Using /home/lksgcc/.pyenv/versions/anaconda3-5.0.1/envs/sgcc36/lib/python3.7/site-packages
Searching for joblib==1.1.0
Best match: joblib 1.1.0
Adding joblib 1.1.0 to easy-install.pth file
Using /home/lksgcc/.pyenv/versions/anaconda3-5.0.1/envs/sgcc36/lib/python3.7/site-packages
Searching for threadpoolctl==3.1.0
Best match: threadpoolctl 3.1.0
Adding threadpoolctl 3.1.0 to easy-install.pth file
Using /home/lksgcc/.pyenv/versions/anaconda3-5.0.1/envs/sgcc36/lib/python3.7/site-packages
Searching for Pillow==7.2.0
Best match: Pillow 7.2.0
Adding Pillow 7.2.0 to easy-install.pth file
Using /home/lksgcc/.pyenv/versions/anaconda3-5.0.1/envs/sgcc36/lib/python3.7/site-packages
Searching for six==1.16.0
Best match: six 1.16.0
Adding six 1.16.0 to easy-install.pth file
Using /home/lksgcc/.pyenv/versions/anaconda3-5.0.1/envs/sgcc36/lib/python3.7/site-packages
Finished processing dependencies for hyperlearn==0.0.1
#### HyperLearn has been installed! ####
#### If you want to compile Numba code, please run:
>>>> python numba_compile.py
However, when I import hyperleran, I found there is not any sub method can be found and there is only a hyperlearn-0.0.1-py3.7.egg file in the site-package file. Is there anything wrong during my installation?
Thanks for your reply.
Hi everyone!!!
This package will be revamped ASAP to make it a functional beast!!!
Join our Discord server: https://discord.gg/QPaNysgA to chat on everything about fast algorithms, AI and more!!!
Logistic Regression's Standard Errors for the coefficients can be easily computed via the Hessian Matrix.
In fact, Var(theta) = diag(-H^-1)^2, and Hessian H ==> X.TX * f(x,theta)(1-f(x,theta))
Where f = sigmoid function.
Now, need to figure out from statsmodels code how they computed for MLR. Is it also Hessian based? Can each matrix of coefficients be computed separately?
Also, should Wald test be applied on N(0,1) or squared applied to Chi2?
The name _min is not defined. Seems like that existed in hyperlearn_old/numba but not the current one.
Anyways, I didn't update the code a lot, but that's because I was busily testing and finding out which algos were the most stable and best.
Key findings for N = 5,000 P = 6,000 [more features than N near square matrix]
For pseudoinverse, (used in Linear Reg, Ridge Reg, lots of other algos), JIT, Scipy MKL, PinvH, Pinv2 and HyperLearn's Pinv are very similar. PyTorch's is clearly problematic, having close to over x4 slower than Scipy MKL.
For Eigh (used in PCA, LDA, QDA, other algos), Sklearn's PCA utilises SVD. Clearly, not a good idea, since it is much better to compute the eigenvec / eigenval on XTX. JIT Eigh is the clear winner at 14.5 seconds on XTX, whilst Numpy is 2x slower. Torch likewise is slower once again...
So, for PCA, a speedup of 3 times is seem if using JIT compiled Eigh when compared to Sklearn's PCA
To solve X*theta = y, Torch GELS is super unstable. Like really. If you use Torch GELS, don't forget to call theta_hat[np.isnan(theta_hat) | np.isinf(theta_hat)] = 0, or else results are problematic. All other algos have very similar MSEs, and HyperLearn's Regularized Cholesky Solve takes a mere 0.635 seconds when compared to say using Sklearn's next fastest Ridge Solve (via cholesky) by over 100% (after considering matrix multiplication time) --> HyperLearn 2.89s vs 4.53s Sklearn.
So to conclude:
HyperLearn's Pseudoinverse has no speed improvement
HyperLearn's PCA will have over 2 times speed boost. (200% improvement)
HyperLearn's Linear Solvers will be over 1 times faster. (100% improvement)
Help make HyperLearn better! All contributors are welcome, as this is truly an overwhelming project...
This is what I see when running python setup.py install (gcc8, python 3.6):
running install
running build
running build_py
file hyperlearn.py (for module hyperlearn) not found
file hyperlearn.py (for module hyperlearn) not found
running install_lib
warning: install_lib: 'build/lib' does not exist -- no Python modules to install
running install_egg_info
running egg_info
writing hyperlearn.egg-info/PKG-INFO
writing dependency_links to hyperlearn.egg-info/dependency_links.txt
writing requirements to hyperlearn.egg-info/requires.txt
writing top-level names to hyperlearn.egg-info/top_level.txt
file hyperlearn.py (for module hyperlearn) not found
reading manifest file 'hyperlearn.egg-info/SOURCES.txt'
writing manifest file 'hyperlearn.egg-info/SOURCES.txt'
Copying hyperlearn.egg-info to /home/share/miniconda3/envs/torch/lib/python3.6/site-packages/hyperlearn-0.0.1-py3.6.egg-info
running install_scripts
###########
>>> Building HyperLearn C extensions. <<<
running build_ext
building 'DEFINE' extension
creating build/temp.linux-x86_64-3.6
gcc -pthread -B /home/share/miniconda3/envs/torch/compiler_compat -Wl,--sysroot=/ -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -march=native -fPIC -I/home/share/miniconda3/envs/torch/lib/python3.6/site-packages/numpy/core/include -I/home/share/miniconda3/envs/torch/include/python3.6m -c DEFINE.c -o build/temp.linux-x86_64-3.6/DEFINE.o
In file included from /home/share/miniconda3/envs/torch/lib/python3.6/site-packages/numpy/core/include/numpy/ndarraytypes.h:1823:0,
from /home/share/miniconda3/envs/torch/lib/python3.6/site-packages/numpy/core/include/numpy/ndarrayobject.h:18,
from /home/share/miniconda3/envs/torch/lib/python3.6/site-packages/numpy/core/include/numpy/arrayobject.h:4,
from DEFINE.c:580:
/home/share/miniconda3/envs/torch/lib/python3.6/site-packages/numpy/core/include/numpy/npy_1_7_deprecated_api.h:15:2: warning: #warning "Using deprecated NumPy API, disable it by " "#defining NPY_NO_DEPRECATED_API NPY_1_7_API_VERSION" [-Wcpp]
#warning "Using deprecated NumPy API, disable it by " \
^
gcc -pthread -shared -B /home/share/miniconda3/envs/torch/compiler_compat -L/home/share/miniconda3/envs/torch/lib -Wl,-rpath=/home/share/miniconda3/envs/torch/lib -Wl,--no-as-needed -Wl,--sysroot=/ -O3 -march=native build/temp.linux-x86_64-3.6/DEFINE.o -o /home/share/software/sources/hyperlearn/hyperlearn/cython/DEFINE.cpython-36m-x86_64-linux-gnu.so
building '__temp__' extension
gcc -pthread -B /home/share/miniconda3/envs/torch/compiler_compat -Wl,--sysroot=/ -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -O3 -march=native -fPIC -I/home/share/miniconda3/envs/torch/lib/python3.6/site-packages/numpy/core/include -I/home/share/miniconda3/envs/torch/include/python3.6m -c __temp__.c -o build/temp.linux-x86_64-3.6/__temp__.o
In file included from /usr/lib/gcc/x86_64-linux-gnu/5/include/immintrin.h:79:0,
from __temp__.c:578:
__temp__.c: In function ‘__pyx_f_8__temp____mult_add’:
/usr/lib/gcc/x86_64-linux-gnu/5/include/fmaintrin.h:55:1: error: inlining failed in call to always_inline ‘_mm_fmadd_ps’: target specific option mismatch
_mm_fmadd_ps (__m128 __A, __m128 __B, __m128 __C)
^
__temp__.c:1084:3: error: called from here
_mm_store_ps1(__pyx_v_x, _mm_fmadd_ps(__pyx_v_mult, _mm_load_ps1(__pyx_v_x), __pyx_v_shift));
^
In file included from /usr/lib/gcc/x86_64-linux-gnu/5/include/immintrin.h:79:0,
from __temp__.c:578:
/usr/lib/gcc/x86_64-linux-gnu/5/include/fmaintrin.h:55:1: error: inlining failed in call to always_inline ‘_mm_fmadd_ps’: target specific option mismatch
_mm_fmadd_ps (__m128 __A, __m128 __B, __m128 __C)
^
__temp__.c:1084:3: error: called from here
_mm_store_ps1(__pyx_v_x, _mm_fmadd_ps(__pyx_v_mult, _mm_load_ps1(__pyx_v_x), __pyx_v_shift));
^
error: command 'gcc' failed with exit status 1
###########
>>> Successfully built C extensions. <<<
Not sure what to make of this.
https://en.wikipedia.org/wiki/Least_absolute_deviations
Planning to use IRLS to compute LAD. Since Cholesky is nearly complete, I'll try focus on this.
https://en.wikipedia.org/wiki/Iteratively_re-weighted_least_squares
Was going to name it LeastAbsoluteRegression
I tried N = 1,000,000 and P = 100, and Cholskey blew me away!!! (it took a whopping 400ms... YES milli seconds) to fit.
Goodness. PyTorch / Numba SVD takes at least 2 seconds!
The issue with Cholskey is STABILITY. If a matrix is near singular, the results will be horrible. So, need to do the following:
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.