Comments (8)
Hello,
Have you tried standardising the input features first?
from allrank.
Hi, thank you for the answer and sorry for the late reply. By standardizing the features I got closer to your results, with an ndcg@10 of 0.5388 on the test set. The results of the paper are still slightly higher. Maybe my architecture is different from yours. Did you use layer normalization in the input layer?
from allrank.
You should be able to replicate the result if you turn off the normalization at the input layer and turn off early stopping (e.g. by setting it to 100). If there are any further problems, we will investigate it.
Also, please remember that the reported results are the average (+ std.dev) over 5 folds.
from allrank.
Hi, I finally got 0.5208 of ndcg@10 on the Fold1 of MSN30k by removing the early stopping criterion. I think that this result matches the one in your paper.
from allrank.
There was one more issue with reproducibility. In our experiments we used an internal allRank fork and missed one detail in the GitHub version - "filler" NDCG value to use when there are no relevant items in the list. LightGBM and XGBoost (AFAIK) use 1.0 so we used this value in our experiments. However, the released code contained 0.0 filler NDCG.
The filler NDCG has been changed to 1.0, as of version 1.4.1. We are also working on a WEB30K reproducibility guide for both papers.
from allrank.
Thank you for your answer. Anyway, I did change the filler NDCG to 1.0 during my experiments, hence my results shall be compared with the ones in your paper. Can you share with me your NDCG@10 score on the Fold1 of WEB30K?
from allrank.
Thank you for your answer. Anyway, I did change the filler NDCG to 1.0 during my experiments, hence my results shall be compared with the ones in your paper. Can you share with me your NDCG@10 score on the Fold1 of WEB30K?
excuse me , does standardizing the features need to be implemented by myself?
from allrank.
Thank you for your answer. Anyway, I did change the filler NDCG to 1.0 during my experiments, hence my results shall be compared with the ones in your paper. Can you share with me your NDCG@10 score on the Fold1 of WEB30K?
excuse me , does standardizing the features need to be implemented by myself?
I think yes, or at least this is what I did. I standardized the features offline before launching the training script.
from allrank.
Related Issues (20)
- How to run "run_example.sh" HOT 3
- Which loss functions are suitable for sentence ordering? HOT 1
- Transformer implementation seemingly not corresponding to paper HOT 3
- Location of loss functions?? HOT 2
- Forward vs Backward Slash in Path definition HOT 3
- AssertionError: val ndcg_5 got 0.7586661577224731. It was expected to be at least 0.76 HOT 2
- Getting started without Docker HOT 1
- How to get predictions from the model? HOT 5
- How to use your own dataset? HOT 2
- How to get predictions for each observation or row in my data. HOT 7
- Whats the expected input to the losses and metrics? HOT 1
- How to produce predictions? HOT 9
- How to produce predictions?
- How to get prediction score HOT 3
- My experiments consistently underperform in comparison to the paper's reported results. HOT 2
- Any sources for the click model based relabeling mentioned in the readme HOT 2
- About the padding value HOT 2
- RuntimeError Numpy not available HOT 1
- Embeddings
- Question about positional encoding HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from allrank.