pointing to issue I mentioned in mabwiser about mab2rec

[bkleyn]

Could you please try setting click_column="score" when defining the evaluation metrics as per below?

# Column names for the response, user, and item id columns
metric_params = {'click_column': 'score', 'user_id_column': 'ID', 'item_id_column':'MailerID'}

[ayush488]

metric_params = {'click_column': 'score', 'user_id_column': 'ID', 'item_id_column':'MailerID'}

Evaluate peformance at different k-recommendations

top_k_list = [4]

List of metrics to benchmark

metrics = []
for k in top_k_list:
metrics.append(BinaryRecoMetrics.AUC(**metric_params, k=k))
metrics.append(BinaryRecoMetrics.CTR(**metric_params, k=k))
metrics.append(RankingRecoMetrics.Precision(**metric_params, k=k))
metrics.append(RankingRecoMetrics.Recall(**metric_params, k=k))
metrics.append(RankingRecoMetrics.NDCG(**metric_params, k=k))
metrics.append(RankingRecoMetrics.MAP(**metric_params, k=k))

from mab2rec.pipeline import benchmark

Benchmark the set of recommenders for the list of metrics

using training data and user features scored on test data

reco_to_results, reco_to_metrics = benchmark(recommenders,
metrics=metrics,
train_data=X_train,
test_data=X_test,
user_features=df_users_X,
user_id_col= 'ID',
item_id_col= 'MailerID',
response_col = 'sales_net')
Traceback (most recent call last):

File "C:\Users\ayush\AppData\Local\Temp\2\ipykernel_5524\2509256242.py", line 27, in
response_col = 'sales_net')

File "C:\ProgramData\Anaconda3\envs\test_env\lib\site-packages\mab2rec\pipeline.py", line 417, in benchmark
return _bench(**args)

File "C:\ProgramData\Anaconda3\envs\test_env\lib\site-packages\mab2rec\pipeline.py", line 531, in _bench
recommendations[name])

File "C:\ProgramData\Anaconda3\envs\test_env\lib\site-packages\jurity\recommenders\combined.py", line 121, in get_score
return_extended_results)

File "C:\ProgramData\Anaconda3\envs\test_env\lib\site-packages\jurity\recommenders\auc.py", line 140, in get_score
return self._accumulate_and_return(results, batch_accumulate, return_extended_results)

File "C:\ProgramData\Anaconda3\envs\test_env\lib\site-packages\jurity\recommenders\base.py", line 121, in _accumulate_and_return
cur_result = self._get_results([results])

File "C:\ProgramData\Anaconda3\envs\test_env\lib\site-packages\jurity\recommenders\auc.py", line 146, in _get_results
return roc_auc_score(results[:, 0], results[:, 1])

File "C:\ProgramData\Anaconda3\envs\test_env\lib\site-packages\sklearn\metrics_ranking.py", line 560, in roc_auc_score
raise ValueError("multi_class must be in ('ovo', 'ovr')")

ValueError: multi_class must be in ('ovo', 'ovr')

[ayush488]

getting this error now?

[ayush488]

seems like the evaluation can only be done with click 0 or 1?

[ayush488]

Could you please try setting click_column="score" when defining the evaluation metrics as per below?

# Column names for the response, user, and item id columns
metric_params = {'click_column': 'score', 'user_id_column': 'ID', 'item_id_column':'MailerID'}

now it is giving different error. Could you please help?

[bkleyn]

Yes, that's correct. All the recommendation metrics above are only well defined for binary (0 or 1) responses.

[ayush488]

so there is no way to evaluate the non binary responses?

[bkleyn]

Unfortunately, we don't have non-binary metrics implemented in the Jurity library. Common metrics include Mean Absolute Error (MAE) and Mean Squared Error (MSE) and are straightforward to compute.

An alternative strategy that is frequently used would be to binarize your response by setting the response to 1 if the continuous value is above some threshold and 0 otherwise.

[ayush488]

But the recommendation system as such works for non binary responses. Is that correct? Only evaluation is more geared towards binary responses.??

…

On Jul 6, 2022 at 6:29 AM, <Bernard Kleynhans ***@***.***)> wrote: Closed #14 (#14) as completed. — Reply to this email directly, view it on GitHub (#14 (comment)), or unsubscribe (https://github.com/notifications/unsubscribe-auth/ABDPIETAPAPSCVINWJNOGD3VSVU3BANCNFSM52WXHAWQ). You are receiving this because you authored the thread.Message ID: ***@***.***>

[bkleyn]

Yes, that's correct.

[ayush488]

 File "C:\Users\ayush\AppData\Local\Temp\2\ipykernel_5524\2998262892.py", line 13, in <module>
    response_col = 'response')

  File "C:\ProgramData\Anaconda3\envs\test_env\lib\site-packages\mab2rec\pipeline.py", line 417, in benchmark
    return _bench(**args)

  File "C:\ProgramData\Anaconda3\envs\test_env\lib\site-packages\mab2rec\pipeline.py", line 526, in _bench
    save_file=None)

  File "C:\ProgramData\Anaconda3\envs\test_env\lib\site-packages\mab2rec\pipeline.py", line 286, in score
    recs_of_batch, scores_of_batch = recommender.recommend(contexts, excluded_arms_batch, return_scores=True)

  File "C:\ProgramData\Anaconda3\envs\test_env\lib\site-packages\mab2rec\rec.py", line 323, in recommend
    expectations = self.mab.predict_expectations(contexts)

  File "C:\ProgramData\Anaconda3\envs\test_env\lib\site-packages\mabwiser\mab.py", line 1229, in predict_expectations
    return self._imp.predict_expectations(contexts)

  File "C:\ProgramData\Anaconda3\envs\test_env\lib\site-packages\mabwiser\linear.py", line 151, in predict_expectations
    return self._parallel_predict(contexts, is_predict=False)

  File "C:\ProgramData\Anaconda3\envs\test_env\lib\site-packages\mabwiser\base_mab.py", line 227, in _parallel_predict
    for i in range(n_jobs))

  File "C:\ProgramData\Anaconda3\envs\test_env\lib\site-packages\joblib\parallel.py", line 1043, in __call__
    if self.dispatch_one_batch(iterator):

  File "C:\ProgramData\Anaconda3\envs\test_env\lib\site-packages\joblib\parallel.py", line 861, in dispatch_one_batch
    self._dispatch(tasks)

  File "C:\ProgramData\Anaconda3\envs\test_env\lib\site-packages\joblib\parallel.py", line 779, in _dispatch
    job = self._backend.apply_async(batch, callback=cb)

  File "C:\ProgramData\Anaconda3\envs\test_env\lib\site-packages\joblib\_parallel_backends.py", line 208, in apply_async
    result = ImmediateResult(func)

  File "C:\ProgramData\Anaconda3\envs\test_env\lib\site-packages\joblib\_parallel_backends.py", line 572, in __init__
    self.results = batch()

  File "C:\ProgramData\Anaconda3\envs\test_env\lib\site-packages\joblib\parallel.py", line 263, in __call__
    for func, args, kwargs in self.items]

  File "C:\ProgramData\Anaconda3\envs\test_env\lib\site-packages\joblib\parallel.py", line 263, in <listcomp>
    for func, args, kwargs in self.items]

  File "C:\ProgramData\Anaconda3\envs\test_env\lib\site-packages\mabwiser\linear.py", line 212, in _predict_contexts
    arm_to_expectation[arm] = arm_to_model[arm].predict(row)

  File "C:\ProgramData\Anaconda3\envs\test_env\lib\site-packages\mabwiser\linear.py", line 93, in predict
    beta_sampled = self.rng.multivariate_normal(self.beta, np.square(self.alpha) * self.A_inv)

  File "C:\ProgramData\Anaconda3\envs\test_env\lib\site-packages\mabwiser\utils.py", line 258, in multivariate_normal
    return np.squeeze(self.rng.multivariate_normal(mean, covariance, size=size, method='cholesky'))

  File "_generator.pyx", line 3625, in numpy.random._generator.Generator.multivariate_normal

  File "<__array_function__ internals>", line 6, in cholesky

  File "C:\ProgramData\Anaconda3\envs\test_env\lib\site-packages\numpy\linalg\linalg.py", line 763, in cholesky
    r = gufunc(a, signature=signature, extobj=extobj)

  File "C:\ProgramData\Anaconda3\envs\test_env\lib\site-packages\numpy\linalg\linalg.py", line 91, in _raise_linalgerror_nonposdef
    raise LinAlgError("Matrix is not positive definite")

LinAlgError: Matrix is not positive definite

Still getting error after binarizing....