Comments (7)
StellaAthena
commented on May 19, 2024
It looks like almost all of the eval code is written, we are just missing RTE. Once that's done, we can move this to DONE.
from lm-evaluation-harness.
anishthite
commented on May 19, 2024
RTE is there. https://github.com/EleutherAI/lm_evaluation_harness/blob/master/lm_eval/tasks/superglue.py#L286
from lm-evaluation-harness.
StellaAthena
commented on May 19, 2024
@anishthite Right. But that is missing the evaluation code. The rest of the SuperGLUE task has the evaluation code written.
from lm-evaluation-harness.
anishthite
commented on May 19, 2024
Sorry, I missed the evaluation part
from lm-evaluation-harness.
thefazzer
commented on May 19, 2024
Happy to take the SuperGLUE implementation
from lm-evaluation-harness.
zphang
commented on May 19, 2024
Updated with #91
- BoolQ
- Commitment Bank
- COPA
- MultiRC
- Words in Context
- SG Winograd Schema Challenge (requires free-form generation)
- RTE (GLUE)
- ReCoRD
from lm-evaluation-harness.
StellaAthena
commented on May 19, 2024
Closing for now as free-form generation is a future problem
from lm-evaluation-harness.
Related Issues (20)
- Adding new Task 'EXAMS' HOT 2
- Output format of samples has been changed HOT 1
- Bug in some model init
- lm_eval not found HOT 2
- The document related to the Task Manager is no longer available. HOT 3
- Add New Lambada Translations
- [Bug] UnboundLocalError: lm acessed before assignment HOT 2
- concurrent api request to accelerate evaluation HOT 1
- [Question] Caching the tokenized evaluation sets? HOT 3
- ValueError: Tasks not found: persona_desire-for-acquiring-eval-results. HOT 1
- ValueError: Input length of input_ids is 494, but `max_length` is set to 128. This can lead to unexpected behavior. You should consider increasing `max_length` or, better yet, setting `max_new_tokens`. HOT 2
- Unused tests in test_tasks.py HOT 1
- TypeError: str expected, not bool when setting HF_DATASETS_TRUST_REMOTE_CODE environment variable
- Write tests for calling CLI arguments downstream to ensure correctly-returned types HOT 4
- what does this log mean๏ผ HOT 2
- mmlu final score and per subject score is calculated wrong HOT 6
- humaneval task HOT 1
- [Question] command to run open_llm_leaderboard ? HOT 5
- Multiple Choice Question Standard Deviation HOT 1
- the score of Qwen/Qwen1.5-7B on mmlu is 1.5 point lower than the score reported on leaderboard. HOT 10
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. ๐๐๐
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google โค๏ธ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from lm-evaluation-harness.