paudelsagar / multiscorer

A module for allowing the use of multiple metric functions in scikit's cross_val_score

multiscorer

A module for allowing the use of multiple metric functions in scikit's cross_val_score.

As it has been discussed, Python's SciKit, while it contains a great functionality for computing evaluation metrics of estimators (using cross_val_score), it seems to fail when it comes to computing multiple metrics for the same classifier without trainning it again.

The problem arises because of the scoring parameter of the function which accepts only a single metric name or a single callable.

Module multiscorer of this repo, is a workaround for using any number of metrics in cross_val_score.

Installation

To "install" the module simply download the source code and place it in your project's directory.
(Alternativelly, download and add to your project just multiscorer.py file).

Usage (Quickstart example)

From a Python script, you can write:

from multiscorer import MultiScorer

from sklearn.metrics import accuracy_score, precision_score          # Scikit's libraries for demonstration
from sklearn.model_selection import cross_val_score
from numpy import average

scorer = MultiScorer({                                               # Create a MultiScorer instance
  'accuracy': (accuracy_score, {}),
  'precision': (precision_score, {'average': 'macro'})               # Param 'average' will be passed to precision_score as kwarg 
})

...

cross_val_score(clf, X, target, scoring=scorer, cv=10)               # Use the function with our socrer. Ignore its result 

results = scorer.get_results()                                       # Get a dict of lists containing the scores for each metric

for metric in results.keys():                                        # Iterate and use the results
  print("%s: %.3f" % (metric, average(results[metric])))

Notes

• You can get results only from specific metrics and/or specific folds (see examples ).
• You can also use your own custom metric functions (see how).
• For a full documentation see Wiki.
• Original documentation of SciKit: http://scikit-learn.org/stable/

Development and Contributing

This module was something I had the need for while working with Scikit's libraries and I just thought it might help somebody.
For questions, bugs, suggestions etc, feel free to contact me or submit a Pull Request.