statisticianinstilettos / recmetrics Goto Github PK

Is there any method for AP@k or AP@K calculation?

In the example bellow, the coverage measured exceeds 100%, which does not make sense.

This happens when items that are not listed on the catalog are recommended.

> from rcmetrics import prediction_coverage
> prediction_coverage([['x', 'y'], ['w', 'z']], catalog=['w', 'x', 'y'])
133.33

Is there any way to integrate this with recommender system frameworks that involve more deep learning-based algorithms such as PyTorch etc.? Sci-Kit Learn's with Surprise doesn't really support such algorithms

First of all: this package looks great! It's exactly what I need for some small projects, so thanks for putting it out there!

I'm looking at the setup.py, and it lists surprise as a requirement. I don't see it imported anywhere in the package though, so I'm wondering if it can be removed? I get that it's useful for the example notebook, but that wouldn't be included in the pip install anyway. (I might suggest making surprise an extras_require if you want to keep it in there for demo purposes.)

If you're open to some packaging changes along these lines, I'd be happy to send a PR your way.

Hi! Have been trying to install recmetrics with "pip install recmetrcis", keep getting an error "ERROR: Could not build wheels for scikit-learn, which is required to install pyproject.toml-based projects". I'm using Windows, Python version 3.9.7, pip all upgraded. pip freeze shows that scikit-learn is actually already installed: "scikit-learn==0.24.2". I've also tried installing with pip from git, same result. Any ideas what I could still try?

This assertion check is incorrect. The actual parameter as used in _apk is expecting a list of true items and the predicted parameter is expecting a list of predicted items that can be true or false. See an example below where only A-C are true items and the prediction can be longer than the true list because it can contain false items.

true_items = ["A","B","C"]
prediction = ["A","Z","B","X"]
metrics.mapk(actual=true_items, predicted=prediction, k = 3)

This is missing a license. You can use https://tldrlegal.com/ for an overview. The top-3 are MIT, BSD and GPL (see my analysis).

The simplest way to add it is in the setup.py as license='MIT' or similar.

Dear @statisticianinstilettos,

Is this library actively maintained?

Bests,

Benedek

Surprise is listed as a module dependency but is not used in metrics or plots. Might be worth removing the dependency - especially since it requires additional built tools (Visual C++) and thus may throw unnecessary errors.

I attached the error below

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-30-05160122655c> in <module>
----> 1 recmetrics.class_separation_plot(pred_df, n_bins=45, class0_label="True class 0", class1_label="True class 1")

TypeError: class_separation_plot() got an unexpected keyword argument 'class0_label'

Importing the repository does not work. I am getting the following error ImportError: cannot import name 'signature'

pip install recmetrics
import recmetrics as re

The problem is this import from sklearn.utils.fixes import signature.

MAP@k implementation linked in the documentation (https://github.com/benhamner/Metrics) has not been updated for 7 years and has bugs in MAP@k implementation (e.g. benhamner/Metrics#51, benhamner/Metrics#57). It would be really useful to have MAP@k implementation in recmetrics. Would it be possible to implement it? It would be almost identical to the existing mark() function.

As described in the title.

Getting following error when importing recmetrics.

There are dead libraries that might have useful features, it would be good to take a peak.

• https://github.com/recommenders/rival/wiki/Evaluation-Metrics
• https://mendeley.github.io/mrec/api.html

First, thank you for providing this great libary! I faced an issue on rather large data sets, in particular when option x_labels is set to False. Suggests to insert an
if x_labels == True: before, similar as it is done on bottom of function. Because, whe I don't want to plot labels, why should plt.xticks(x) be executed?

Hi there
I am trying to run example notebook. But I am getting 'module 'recmetrics' has no attribute 'prediction_coverage'' and "attribute error: module 'recmetrics' has no attribute 'catalog_coverage'"

any pointer or suggestion.

Thanks in advance

On my system (16gb ram), a list of 10k recommendations will run. A list of 50k will crash out. I'd like to try to understand the personalization score across my entire hypothetical customer base 250k+.

Is there a way to chunk the scipy.sparse.csr_matrix and iteratively calculate the cosine similarity to avoid holding the whole thing in memory?

A new metric familiarity would be a nice addition.
Paper:https://link.springer.com/article/10.1007%2Fs11257-011-9115-7

I am working on my thesis and wouldn't mind adding the feature to this project.

Hi @statisticianinstilettos,

kudos for a great tool!
I would like to propose an optimization for calculating Personalization Metric here:

#get indicies for upper right triangle w/o diagonal
upper_right = np.triu_indices(similarity.shape[0], k=1)

#calculate average similarity
personalization = np.mean(similarity[upper_right])
return 1-personalization

There is no need to get the upper triangle indices, as the cosine similarity is a symmetric distance.
I will follow up with a pull request for this.

» poetry add recmetrics   
Using version ^0.1.5 for recmetrics

Updating dependencies
Resolving dependencies... (0.2s)

Because no versions of recmetrics match >0.1.5,<0.2.0
 and recmetrics (0.1.5) depends on pytest-cov (>=2.10.1,<3.0.0), recmetrics (>=0.1.5,<0.2.0) requires pytest-cov (>=2.10.1,<3.0.0).
So, because jewel-ml-models depends on both recmetrics (^0.1.5) and pytest-cov (^4.0.0), version solving failed.

pytest-cov is a development dependency, it shouldn't break like this. You can easily solve this by installing pytest-cov as development dependency. There are others as well... ipython maybe? Jupyter and twine too

I am working on a recommendation engine using collaborative filtering and wanted to try the metrics provided by recmetrics. Here, the error I get trying to import the package (version 0.0.12).

---------------------------------------------------------------------------
ImportError                               Traceback (most recent call last)
<ipython-input-309-301854677c00> in <module>
----> 1 import recmetrics
      2 
      3 recmetrics.long_tail_plot()

~/.virtualenvs/py3/lib/python3.6/site-packages/recmetrics/__init__.py in <module>
----> 1 from .plots import long_tail_plot, mark_plot, mapk_plot, coverage_plot, class_separation_plot, roc_plot, precision_recall_plot
      2 from .metrics import mark, coverage, personalization, intra_list_similarity, rmse, mse, make_confusion_matrix, recommender_precision, recommender_recall

~/.virtualenvs/py3/lib/python3.6/site-packages/recmetrics/plots.py in <module>
      5 from matplotlib.lines import Line2D
      6 from sklearn.metrics import roc_curve, auc, precision_recall_curve, average_precision_score
----> 7 from sklearn.utils.fixes import signature
      8 
      9 

ImportError: cannot import name 'signature'

deprecated

The current version needs to be updated as the packages depends on deprecated/removed functionality from different dependencies.