Comments (8)
I make a mistake above, the inference speed of EAGLE-2 is 66 and EAGLE-1 is 69. Besides, temperature is 0
from eagle.
It is not normal. Can you provide more details (e.g., if you are running something else on your machine, what is your environment, which codes are your running)? Without them, it is hard to debug your code.
from eagle.
Some details of my environment are listed as follow:
1 GPU V100 32G memory
python 3.10.14
CUDA 11.7
Driver Version: 515.65.01
torch 2.1.0
triton 2.1.0
transformers 4.36.2
I just run the evaluation script gen_ea_answer_vicuna.py as suggested in Readme
batch size= 1
temperature=0
from eagle.
The possible reason is that total_token was not set correctly.
from eagle.
Thanks for the response. Can you provide me some advise to set total_token? Should I set larger or smaller?
from eagle.
HI, the benchmark can't record new token correctly, I don't know if that's normal.
python -m eagle.evaluation.gen_ea_answer_vicuna\
--ea-model-path yuhuili/EAGLE-Vicuna-7B-v1.3\
--base-model-path lmsys/vicuna-7b-v1.3
python -m eagle.evaluation.gen_baseline_answer_vicuna\
--ea-model-path yuhuili/EAGLE-Vicuna-7B-v1.3\
--base-model-path lmsys/vicuna-7b-v1.3
{"question_id": 81, "answer_id": "TP4CRrbLYBqFHdQqoeb7ug", "model_id": "ess-vicuna-70b-fp16-baseline-temperature-1.0", "choices": [{"index": 0, "turns": ["....... "idxs": [603, 603], "new_tokens": [0, 0], "wall_time": [8.09636378288269, 7.946403741836548]}], "tstamp": 1720166094.253764}
from eagle.
Thanks for the response. Can you provide me some advise to set total_token? Should I set larger or smaller?
Overall, the smaller the model and the more powerful the computational capacity, the larger this value should be.
from eagle.
HI, the benchmark can't record new token correctly, I don't know if that's normal.
It is normal for the baseline not to return new tokens.
from eagle.
Related Issues (20)
- Any plan to support Llama 405B (Llama 3.1) HOT 3
- python -m eagle.ge_data.allocation --outdir [path of data] HOT 2
- How to use multi-GPU when gen data? HOT 2
- Any hint to improve average accept length for fine tuned LLAMA 70b? HOT 4
- ge_data_all_llama2chat
- Question about EAGLE-2 HOT 5
- Can Eagle2 use multi-gpus to inference? HOT 1
- Can Ealge2 use multi-gpus to inference? I mean data parallelism HOT 1
- Is this expected behavior? HOT 2
- Support Qwen2 in Eagle's inference framwork HOT 3
- Confusion in Figure 5 (b) of EAGLE2 paper HOT 2
- Will smaller models be supported? HOT 1
- Qwen2 inference HOT 8
- Actual parameter setting for different sizes of the model? HOT 1
- Reproduction Bias HOT 4
- Some questions that arise when studying your code HOT 2
- Can not reproduce speedup results reported in the paper. HOT 2
- When Train the Auto-regression Head, RuntimeError: CUDA error: device-side assert triggered
- what's the difference between ea_generate and eagenerate
- Evaluating Eagle on HumanEval, GSM8K, Alpaca, CNN/DM, and Natural Questions Datasets
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from eagle.