wsmorgan / analyzefit

analyzefit

Analyze fit is a python package that performs standard analysis on the fit of a regression model. The analysis class validate method will create a residuals vs fitted plot, a quantile plot, a spread location plot, and a leverage plot for the model provided as well as print the accuracy scores for any metric the user likes. For example:

If a detailed plot is desired then the plots can also be generated individually using the methods res_vs_fit, quantile, spread_loc, and leverage respectively. By default when the plots are created individually they are rendered in an interactive inverontment using the bokeh plotting package. For example:

This allows the user to determine which points the model is failing to predict.

Full API Documentation available at: github pages.

Installing the code

To install analyzefit you may either pip install:

pip install analyzefit

or clone this repository and install manually:

python setup.py install

Validating a Model

To use analyze fit simply pass the feature matrix, target values, and the model to the analysis class then call the validate method, (or any other plotting method). For example:

import pandas as pd
import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score
from sklearn.model_selection import train_test_split
from analyzefit import Analysis

df = pd.read_csv('https://archive.ics.uci.edu/ml/machine-learning-databases/housing/housing.data', header=None,sep="\s+")
df.columns = ["CRIM","ZN","INDUS","CHAS","NOX","RM","AGE","DIS","RAD","TAX","PTRATIO","B","LSTAT","MEDV"]
X = df.iloc[:,:-1].values
y = df[["MEDV"]].values
X_train, X_test,y_train,y_test = train_test_split(X,y, test_size=0.3,random_state=0)
slr = LinearRegression()
slr.fit(X_train,y_train)

an = Analysis(X_train, y_train, slr)
an.validate()

an.validate(X=X_test, y=y_test, metric=[mean_squared_error, r2_score])

an.res_vs_fit()

an.quantile()

an.spread_loc()

an.leverage()

Python Packages Used

• numpy

• matplotlib

• bokeh

• sklearn

analyzefit's Issues

The quantile plot should be extended to be able to plot distributions of different lengths.

At present the quantile plot option can only display quantile plots of data against a distribution if the length of the data and distribution arrays are of the some length. This should be extended so that when the distribution and data have different lengths the plot is still generated using an interpolated quantile estimate. See https://en.wikipedia.org/wiki/Q%E2%80%93Q_plot

Need to check if the number of true values and the number of predictions are the same.

At present the user can pass in a feature matrix or a number of predictions that would disagree with the number of true values being used. We should check for this and throw an error when it happens.

Finding the hat matrix fails when a singular design is in use.

If the design matrix is singular then the matrix inverse fails. A check should be implemented that would allow for the pseudo-inverse to be used in these cases.

It is possible to hand the plotting scripts target y values without updating either the predictions.

At present the user can specify their own y values (true target values for the model) without handing the plotting routines either the features that correspond to those y values or the predictions made by the model for those values. This should produce an error since plots produced in this manner would be incorrect.

A plot of the predicted vs actual values should be output.

Since the first thing that virtually everyone does when validating a fit is look at a plot of the predicted vs the actual values the validate method should also produce that plot.

The Spread-location plot should include a 'line' that splints the data 50/50.

To see how the data is distributed in the Spread-Location plots it is useful to know where the 50/50 data split happens. We need to implement such a line. For example see:
http://data.library.virginia.edu/diagnostic-plots/

An idea of how to do this it to plot the change in the means of the standardized residuals and fitted data as we include more and more data points.