Comments (3)
I think this line of code need to be removed:
As solution just use the mask_label from
predict_all()
above.
And I think the reason why current code somehow works is because of this part:
Transformers4Rec/transformers4rec/torch/masking.py
Lines 318 to 337 in d0cce61
Given input sequence without padding
[1,2,3]
, the mask schema generated by current code during evaluation will be [True, True, True]
, which exposes the last item. However the apply_mask_to_inputs
will replace the last item with 0
embedding. And since the schema are all True, no mask embedding will be applied on input. I think in this case 0
embedding sort of plays a role as mask.However, when input has padding like
[1,2,3,0,0]
, the current mask schema will be [True, True, True, False, False]
. And because the last item is a padding item, the apply_mask_to_inputs
basically replaces the padding with 0
embedding. Then the mask schema comes in, masks the last 2 padding items, keeping the 1,2,3
visible to transformer.I think thats why people encounter issues testing clm. If there are always paddings in input data, the evaluation metrics would be unrealistically high.
from transformers4rec.
I also noticed this bug as well. After the fix, the recall is down about 20% less
from transformers4rec.
Any further updates? It seems #723 still not solve this bug.
from transformers4rec.
Related Issues (20)
- [QST] ValueError: For masking a categorical_module is required including an item_id.
- [QST] Projecting inputs of NextItemPredictionTask to'64' As weight tying requires the input dimension '320' to be equal to the item-id embedding dimension '64' HOT 4
- [QST] Cross-entropy and pairwise losses are supported in Next Item Prediction
- [QST] How to print metrics while training?
- RuntimeError: CUDF failure at: /__w/cudf/cudf/cpp/src/io/parquet/reader_impl_helpers.cpp:379: Invalid rowgroup index[BUG] HOT 10
- Génerating predictions HOT 5
- [BUG] Inconsistent inference and evaluation results of the XLNET-CLM even on the training set! HOT 2
- [BUG] CausalLanguageModeling masking error on last item only condition HOT 1
- [QST] Help with creating two tower model with transformers. HOT 1
- [FEA] Post context fusion using T4rec api HOT 1
- [QST] Extracting User Representation Vectors from Pre-trained Next Item Prediction Model
- [BUG] AttributeError: 'list' object has no attribute 'output_node'" HOT 3
- Model is not generating accurate recommandations [QST]
- [BUG] RuntimeError: PyTorch execute failure: Expected Tensor but got GenericList
- [QST] Problem with defining input module, item embedding table. HOT 4
- [QST] examples/tutorial/02-ETL-with-NVTabular.ipynb
- [BUG] examples/tutorial/01-preprocess.ipynb: Convert timestamp from datetime - NotImplementedError: cuDF does not yet support timezone-aware datetimes
- [QST] Prediction Output Length Not Matching Input Length HOT 1
- Compound Tags.ITEM_ID
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from transformers4rec.