Comments (2)
In the past, reproducibility issues typically arose due to:
- unstable documents to be reviewed. For example, deleted or no longer accessible news articles (FakeNewsNet) or tweets (coinfo250), or linked articles (e.g. articles linked in a tweet). I think we aimed to have hashes of the raw texts as a way to double check this without having to store the text (due to copyright issues), but not sure we published these hashcodes (which could be useful to compare the datasets). E.g. if a tweet linked to a webpage which is no longer accessible, the credibility rating may change, and thus also the overall accuracy.
- backwards compatibility of the various deep learning models. When loading finetuned models (e.g. for semantic similarity or stance-detection), using newer library versions (e.g. numpy, cuda, pytorch) can result in a decay of the model performance which would translate in an overall acred performance decay. There's no fully automated test/validation for this, although there's typically some description for each model specifying the test results that the model should achieve. For example the worthiness model should achieve 95% accuracy, the claimencoder should achieve 83% accuracy on STS-B. You'd need to test each model individually/separately to rule out backwards compatibility issues.
from acred.
Hi @rdenaux,
Thank you for your quick response and help. Regarding the points you discussed:
- We successfully collected the datasets necessary for the evaluation and the number of samples correspond to the number described for each evaluation dataset (clef18, coinfo250 and FakeNewsNet). Unfortunately, it is quite difficult to follow the validity of linked sources present in texts (such as URLs pointing to external tweets or articles).
- We successfully tested the performances of the models. The
claimencoder
achieves 83% accuracy on STS-B and theworthinesschecker
achieves 95% accuracy on its test set. In addition, we also tried to test theclaimneuralindex
with some sentence similarity test call.
The results currently achieved for each dataset are:
- Clef18 accuracy: 0.3669
- Coinform250 accuracy: 0.1774
- FakeNewNet accuracy: 0.4726
As far as you know, is it possible that there is some other point in the pipeline that can cause such errors?
from acred.
Related Issues (1)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from acred.