leferrad / pyoselm

A Python implementation of Online Sequential Extreme Machine Learning (OS-ELM) for online machine learning

A Python implementation of Online Sequential Extreme Machine Learning (OS-ELM) for online machine learning

Description

pyoselm is a Python library for machine learning models with Extreme Machine Learning (ELM) and Online Sequential Machine Learning (OS-ELM). It allows to fit models for regression and classification tasks, both in batch and online learning (either row-by-row or chunk-by-chunk).

This library offers a scikit-learn like API for easy usage. For more details about setup and usage, check the documentation.

IMPORTANT: This library was developed as a research project. It may not be production-ready, so please be aware of that.

Setup

The easiest way to install this library is using pip:

$ pip install pyoselm

Usage

Here a simple but complete example of usage.

from pyoselm import OSELMRegressor, OSELMClassifier
from sklearn.datasets import load_digits, make_regression 
from sklearn.model_selection import train_test_split

print("Regression task")
# Model
oselmr = OSELMRegressor(n_hidden=20, activation_func='sigmoid', random_state=123)
# Data
X, y = make_regression(n_samples=1000, n_targets=1, n_features=10, random_state=123)   
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=123)
n_batch = 40

# Fit model with chunks of data
for i in range(20):
    X_batch = X_train[i*n_batch:(i+1)*n_batch]
    y_batch = y_train[i*n_batch:(i+1)*n_batch]
    oselmr.fit(X_batch, y_batch)
    print("Train score for batch %i: %s" % (i+1, str(oselmr.score(X_batch, y_batch))))

# Results
print("Train score of total: %s" % str(oselmr.score(X_train, y_train)))
print("Test score of total: %s" % str(oselmr.score(X_test, y_test)))  
print("")


print("Classification task")
# Model 
oselmc = OSELMClassifier(n_hidden=20, activation_func='sigmoid', random_state=123)
# Data
X, y = load_digits(n_class=5, return_X_y=True) 
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=123)

# Sequential learning
# The first batch of data must have the same size as n_hidden to achieve the first phase (boosting)
batches_x = [X_train[:oselmc.n_hidden]] + [[x_i] for x_i in X_train[oselmc.n_hidden:]]
batches_y = [y_train[:oselmc.n_hidden]] + [[y_i] for y_i in y_train[oselmc.n_hidden:]]

for b_x, b_y in zip(batches_x, batches_y):
    oselmc.fit(b_x, b_y)

print("Train score of total: %s" % str(oselmc.score(X_train, y_train)))
print("Test score of total: %s" % str(oselmc.score(X_test, y_test)))

pyoselm's Issues

Assert correct behavior of OSELMClassifier in sequential learning scenarios with feed of new classes

Let's say the model was trained with classes 1-N, and then a batch of data comes to a fit() call with a new class N+1. The model should incorporate this new class, without losing information of the previous ones

Accept multiple layers

In order to increase the learning capacity of a model, it would be convenient to think in architectures of multiple layers.
Check this paper: https://www.hindawi.com/journals/mpe/2017/4670187/

Handle cases of singular matrices when calling pinv2

All the calls to pinv2 function could raise errors if a singular matrix is operated. We could either identify these cases or just try/catch them to raise a proper error / warning

Add API reference docs

We need that to quickly inspect signatures and examples

Examples of README.md are not working

They should be updated to make them work. In particular, the sequential learning example may need some fixes to work as expected

Add partial_fit() methods as alias for fit() in pyoselm.oselm classes

To allow better compatibility with sklearn API

Ideas for layers

A BatchNormalization layer could be a good idea to avoid overflows. Also, Dropout could be something useful (to be checked)

pinv2 error

---

ImportError Traceback (most recent call last)
F:\TEMP\ipykernel_234664\765891537.py in <cell line: 1>()
----> 1 from pyoselm import OSELMRegressor, OSELMClassifier
2 from sklearn.datasets import load_digits, make_regression
3 from sklearn.model_selection import train_test_split
4
5 print("Regression task")

d:\env\conda\envs\money\lib\site-packages\pyoselm_init_.py in
1 from pyoselm.elm import *
2 from pyoselm.layer import *
----> 3 from pyoselm.oselm import *
4
5 version = "1.0.1"

d:\env\conda\envs\money\lib\site-packages\pyoselm\oselm.py in
10
11 import numpy as np
---> 12 from scipy.linalg import pinv2
13 from scipy.sparse import eye
14 from scipy.special import softmax

ImportError: cannot import name 'pinv2' from 'scipy.linalg' (d:\env\conda\envs\money\lib\site-packages\scipy\linalg_init_.py)

Use a sparse matrix of np.eye in fit_woodbury

Otherwise, with large input (e.g. +10k rows) it would involve expensive algebraic operations

Add functional tests

This library needs testing regarding behavior to ensure the models are working fine in some given scenarios. A BDD testing approach is suggested to cover that with some functional tests.

Add a tutorial

A good option would be a simple Jupyter notebook, where it could show the main features to let the user understand how to completely use this library

How to make to run os-elm classification with iris database

Hello, i would like to learn, how to make to run os-elm classification with iris database.

Is it possible ?

Optimization module to tune hyper-parameters

This library could have a new feature: a module optimization with methods to try different hyper-parameters settings to get the best configuration of a model for a given dataset X, y.

Remove not used class GenELMClassifier

ELMClassifier inherits from ELMRegressor, so GenELMClassifier should not be needed

Error when installing it via pip

If I try to install that package via pip.

pip install pyoselm

I get the following error:

ERROR: Command errored out with exit status 1:
     command: ...\python.exe' -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"....\\pyoselm\\setup.py'"'"'; __file__='
"'"'....\\pyoselm\\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"
'"'exec'"'"'))' egg_info --egg-base '....\pyoselm\
    Complete output (5 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "....\setup.py", line 18, in <module>
        install_requires=open("requirements.txt").read().split()
    FileNotFoundError: [Errno 2] No such file or directory: 'requirements.txt'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Use scipy functions in activation funcs for layers

e.g. replace "sigmoid" with this one: https://docs.scipy.org/doc/scipy/reference/generated/scipy.special.expit.html