title: Timing Comparisons for Supervised Learning of a Classifier author:
- name: Joshua Cook affiliation: Udacity, Machine Learning Nanodegree
- name: Matthew Zhou affiliation: Udacity, Machine Learning Nanodegree
- name: Other Authors
affiliation: Udacity, Machine Learning Nanodegree
abstract:
- This paper describes
$n$ methods for fitting a binary supervised learning classifier on a single large dataset with multiple-typed features. Timing and accuracy metrics are presented for each method, with analysis on the results in terms of the structure of the data set. Fits were performed using the popular open-source machine learning libraryscikit-learn
. Additionally, a code repository including all necessary infrastructure has been developed and shared for reproducibility of results.
- This paper describes
For portability and reproducibility of results, we have elected to use the Docker system and its Dockerfile
syntax to prepare. As this work is done using Python and its scikit-learn
libraries we have elected to use a system built via the Anaconda package manager. Furthermore, leveraging images designed by and for using the Jupyter system, which is built via Anaconda, allows a single container to be used both for running the analysis script and for interactive analysis of the data via Jupyter. The following Dockerfile
completely describes the system used for this work. Note that it inherits from a Docker image designed and maintained by the Jupyter team.
FROM jupyter/scipy-notebook
VOLUMES .:/home/jovyan/work
Via the above, fit analysis can be run on a single classifier,
$ docker run -e CLASSIFIER='decision tree' mlnd/tcsl python project.py
all classifiers,
$ docker run mlnd/tcsl python project.py
or via an interactive notebook server
$ docker run mlnd/tcsl
Note that the last leverages a built-in launch script inherited from the original notebook definition, in that no explicit command was passed to the container.
Select a dataset Proposed requirements:
- Large but not too large i.e. can fit on a single system running Docker
- lends itself to binary classification
- many different types of feature parameters
- from UCI Machine Learning Dataset Library
one-hot encode classification parameters convert all booleans to numeric values
- training
- test
- use seed for reproducibility
For each model complete the following: Copy and paste this template to add a new model. name: brief description: time complexity, training: time complexity, prediction: strengths: Weaknesses:
What metrics should be used for timing, for accuracy, others?
- raw fit of classifier
- raw prediction of classifier
- gridsearchCV fit
- prediction on tuned model
Highest performing model What this says about the data set chosen
List of Supervised Learning Models: PUT YOUR NAME NEXT TO ONE YOU WOULD LIKE TO IMPLEMENT http://scikit-learn.org/stable/supervised_learning.html Decision Trees
- Support Vector Machines (Matt)
- Naive Bayes
- Ridge Regression
- Stochastic Gradient Descent (Joshua)
- Adaptive Moment Estimation (ADAM)
- Linear/Logistic Regression
- K-nearest Neighbors (Matt)
- Random Forests
- XGBoost (may require additional lib) (Matt)
- Linear Discriminant Analysis
- Quadratic Discriminant Analysis
- Gaussian Processes
- Elastic Lasso
- AdaBoost
- Gradient Tree Boost
- Perceptron