astrazeneca / rexmex

A general purpose recommender metrics library for fair evaluation.

---

reXmeX is recommender system evaluation metric library.

Please look at the Documentation and External Resources.

reXmeX consists of utilities for recommender system evaluation. First, it provides a comprehensive collection of metrics for the evaluation of recommender systems. Second, it includes a variety of methods for reporting and plotting the performance results. Implemented metrics cover a range of well-known metrics and newly proposed metrics from data mining (ICDM, CIKM, KDD) conferences and prominent journals.

Citing

If you find RexMex useful in your research, please consider adding the following citation:

@inproceedings{rexmex,
       title = {{rexmex: A General Purpose Recommender Metrics Library for Fair Evaluation.}},
       author = {Benedek Rozemberczki and Sebastian Nilsson and Piotr Grabowski and Charles Tapley Hoyt and Gavin Edwards},
       year = {2021},
}

---

An introductory example

The following example loads a synthetic dataset which has the mandatory y_true and y_score keys. The dataset has binary labels and predictied probability scores. We read the dataset and define a defult ClassificationMetric instance for the evaluation of the predictions. Using this metric set we create a score card and get the predictive performance metrics.

from rexmex import ClassificationMetricSet, DatasetReader, ScoreCard

reader = DatasetReader()
scores = reader.read_dataset()

metric_set = ClassificationMetricSet()

score_card = ScoreCard(metric_set)

report = score_card.get_performance_metrics(scores["y_true"], scores["y_score"])

---

An advanced example

The following more advanced example loads the same synthetic dataset which has the source_id, target_id, source_group and target group keys besides the mandatory y_true and y_score. Using the source_group key we group the predictions and return a performance metric report.

from rexmex import ClassificationMetricSet, DatasetReader, ScoreCard

reader = DatasetReader()
scores = reader.read_dataset()

metric_set = ClassificationMetricSet()

score_card = ScoreCard(metric_set)

report = score_card.generate_report(scores, grouping=["source_group"])

---

Scorecard

A rexmex score card allows the reporting of recommender system performance metrics, plotting the performance metrics and saving those. Our framework provides 7 rating, 38 classification, 18 ranking, and 2 coverage metrics.

Metric Sets

Metric sets allow the users to calculate a range of evaluation metrics for a label - predicted label vector pair. We provide a general MetricSet class and specialized metric sets with pre-set metrics have the following general categories:

• Ranking
• Rating
• Classification
• Coverage

---

Ranking Metric Set

• Normalized Distance Based Performance Measure (NDPM)
• Discounted Cumulative Gain (DCG)
• Normalized Discounted Cumulative Gain (NDCG)
• Reciprocal Rank

Expand to see all ranking metrics in the metric set.

• Mean Reciprocal Rank (MRR)
• Spearmanns Rho
• Kendall Tau
• HITS@k
• Novelty
• Average Recall @ k
• Mean Average Recall @ k
• Average Precision @ k
• Mean Average Precision @ k
• Personalisation
• Intra List Similarity

---

Rating Metric Set

These metrics assume that items are scored explicitly and ratings are predicted by a regression model.

• Mean Squared Error (MSE)
• Root Mean Squared Error (RMSE)
• Mean Absolute Error (MAE)
• Mean Absolute Percentage Error (MAPE)

Expand to see all rating metrics in the metric set.

• Symmetric Mean Absolute Percentage Error (SMAPE)
• Pearson Correlation
• Coefficient of Determination

---

Classification Metric Set

These metrics assume that the items are scored with raw probabilities (these can be binarized).

• Precision (or Positive Predictive Value)
• Recall (Sensitivity, Hit Rate, or True Positive Rate)
• Area Under the Precision Recall Curve (AUPRC)
• Area Under the Receiver Operating Characteristic (AUROC)

Expand to see all classification metrics in the metric set.

• F-1 Score
• Average Precision
• Specificty (Selectivity or True Negative Rate )
• Matthew's Correlation
• Accuracy
• Balanced Accuracy
• Fowlkes-Mallows Index

---

Coverage Metric Set

These metrics measure how well the recommender system covers the available items in the catalog and possible users. In other words measure the diversity of predictions.

• Item Coverage
• User Coverage

---

Documentation and Reporting Issues

Head over to our documentation to find out more about installation and data handling, a full list of implemented methods, and datasets.

If you notice anything unexpected, please open an issue and let us know. If you are missing a specific method, feel free to open a feature request. We are motivated to constantly make RexMex even better.

---

Installation via the command line

RexMex can be installed with the following command after the repo is cloned.

$ pip install .

Use -e/--editable when developing.

Installation via pip

RexMex can be installed with the following pip command.

$ pip install rexmex

As we create new releases frequently, upgrading the package casually might be beneficial.

$ pip install rexmex --upgrade

---

Running tests

Tests can be run with tox with the following:

$ pip install tox
$ tox -e py

---

Citation

If you use RexMex in a scientific publication, we would appreciate citations. Please see GitHub's built-in citation tool.

---

License

• Apache-2.0 License

rexmex's Issues

Information Retrieval

https://en.wikipedia.org/wiki/Evaluation_measures_(information_retrieval)

Add structured data about metrics

It would be great to include some structured data for each metric to enable other libraries to more easily generally consume implementations provided by rexmex. It would be useful to annotate at a minimum:

1. human-readable name
2. range of values (e.g., [0,1])
3. whether higher is better
4. citation, if available

For example, I've annotated citation information on to all models in PyKEEN using the docdata package (which I wrote for just this purpose): Check how it looks for the CrossE model here. In the accompanying PR, I made a slightly simpler solution that just uses Python decorators and doesn't introduce another dependency to rexmex.

An example downstream consumer of the results from solving this issue can be demonstrated this pull request to PyKEEN, where I would like to ultimately consume metric functions directly from rexmex and not have to annotate this information inside PyKEEN itself.

Pandas versioning issue

Hey rexmex team,

I wanted to ask you if there is a specific reason why your installation requires pandas to be of this version?

Since this package gets installed with the latest version of PyKEEN, I am facing some versioning issues due to the Pandas version restriction of your package.

The related line of code:

rexmex/setup.py

Line 3 in 44f453f

install_requires = ["numpy", "sklearn", "pandas<=1.3.5", "scipy", "scikit-learn"]

Apologies if it's a silly question but: is it indeed a strict requirement for rexmex to have Pandas<1.3.5 or is this something we can modify in rexmex setup.py? ( if so I can open a related PR for it )

Thank you in advance!
Best,
Dimitris

API Suggestions

Right now it's a bit round-about to get a scorecard for a given dataset since it expect a pandas format. I'd suggest exposing ScoreCard._get_performance_metrics as a public user interface and also encourage people to use that directly in case they're generating their own y_true and y_score and don't want to write their own code to generate a pandas dataframe from it, just for rexmex to need to unpack it.

My example is in PyKEEN, where we do just that: https://github.com/pykeen/pykeen/blob/799e224e772176703d796a9247bfcc179d343c6c/src/pykeen/evaluation/sklearn.py#L129-L142

I'd also say it would be worth adding a second introductory example based on this in the README.

Pandas versioning

Hey AstraZeneca,

The related line of code:

rexmex/setup.py

Line 3 in 44f453f

install_requires = ["numpy", "sklearn", "pandas<=1.3.5", "scipy", "scikit-learn"]

Is it a strict requirement for rexmex to have Pandas<1.3.5 or is this something we can modify in rexmex or setup.py? (if so I can open a related PR for it, much like the other question but formatted a bit better and notation to make it clear.)

Thanks,
Montana Mendy

Examples link is broken

Hey, great work. I was looking to try out this repo but the examples link in the readme is broken. Please look into it. Thanks!

Rexmex uses the deprecated sklearn dependency

I recently noticed, that rexmex uses the deprecated sklearn dependency. The correct scikit-learn dependency however is also present in setup.py and removing sklearn should be therefore easy and without issues.

Adjusted mean rank

@mberr's adjusted mean rank address some of the problems with the mean rank, including its size dependence. Reference: https://arxiv.org/abs/2002.06914

Cover metrics from this paper

https://www.researchgate.net/profile/Maik-Thiele-2/publication/268381252_Setting_Goals_and_Choosing_Metrics_for_Recommender_System_Evaluations/links/595e42e1a6fdccc9b17fcffe/Setting-Goals-and-Choosing-Metrics-for-Recommender-System-Evaluations.pdf

Learning to rank

https://en.wikipedia.org/wiki/Learning_to_rank

Cover metrics and typization from this paper

https://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.736.8956&rep=rep1&type=pdf

NDCG

Add NDCG.

Add the DeepCCI model

The original paper:
The reference implementation:

• Write auxiliary layers.
• Write a model architecture layer.
• Write smoke tests with the test data.
• Write an example for the ./examples/ folder.

Improve binning in `binarize()`

The current binarize function uses a cutoff of 0.5 for binarization:

rexmex/rexmex/utils.py

Lines 28 to 34 in 3e26652

	def metric_wrapper(*args, **kwargs):
	# TODO: Move to optimal binning. Youden’s J statistic.
	y_score = args[1]
	y_score[y_score < 0.5] = 0
	y_score[y_score >= 0.5] = 1
	score = metric(*args, **kwargs)
	return score

This is an issue for PyKEEN, where the scores that come from a model could all be on the range of [-5,-2]. The current TODO text says to use https://en.wikipedia.org/wiki/Youden%27s_J_statistic, but it's not clear how that would be used.

As an alternative, the NetMF package implements the following code for constructing an indicator that might be more applicable (though I don't personally recognize what method this is, and unfortunately it's not documented):

def construct_indicator(y_score, y):
    # rank the labels by the scores directly
    num_label = np.sum(y, axis=1, dtype=np.int)
    y_sort = np.fliplr(np.argsort(y_score, axis=1))
    y_pred = np.zeros_like(y, dtype=np.int)
    for i in range(y.shape[0]):
        for j in range(num_label[i]):
            y_pred[i, y_sort[i, j]] = 1
    return y_pred

Running ./dev_setup.sh fails

There is no setup.py file inside the rexmex folder, hence the dev_setup.sh script fails with error message:

ERROR: File "setup.py" or "setup.cfg" not found. Directory cannot be installed in editable mode

A solution could be to update the dev_setup.sh to install the package from the root directory:

pip install -e .

Filtered Metrics

Filtered Metrics

Feature request: add a standard function to calculate filtered metrics. This removes recommendations that appeared in training, validation and test sets.

This could be a pre-made function. E.g

rexmex.filter_scores(scores, training_set, validation_set, testing_set)

As described in this paper (described as triples in knowledge graph recommendation):
https://papers.nips.cc/paper/2013/file/1cecc7a77928ca8133fa24680a88d2f9-Paper.pdf

DCG

• Add DCG.

Why are there duplicate functions?

I noticed that there are duplicate functions such as miss_rate()/false_negative_rate() and fall_out()/false_positive_rate(). What's the reason for the duplication?

Binarizing a metric set

Hi,

is it possible binarize a metric set instead of a single metric?