Comments (7)
You can find the performance comparison at subsection "Performance on INT8 without quantizing residual connection" in https://github.com/NVIDIA/DeepLearningExamples/tree/master/FasterTransformer/v3.0#encoder-performance-on-t4-and-tensorflow
from fastertransformer.
This subsection shows the time and speedup, but it doesn't show the exact match / F1 score for INT8 like https://github.com/NVIDIA/DeepLearningExamples/tree/master/FasterTransformer/v3.0#performance-on-application-codes-of-tensorflow. Could you please tell me where to find performance on application code for INT8?
from fastertransformer.
from fastertransformer.
Thanks for your timely reply :)
from fastertransformer.
The quantization in DeepLearningExamples/FasterTransformer/v3.0/bert-tf-quantization is fake quantizaton which uses FP32 to calculate the quantized values. However, the speedup is tested using INT8 which is 8 bits. Are they the same? Or is there something I misunderstand? Looking forward to your reply.
from fastertransformer.
bert-tf-quantization
is only for training. You should train a checkpoint and import it with FT tensorflow op. FasterTransformer op does inference in INT8 precision. The whole workflow is in Evaluate the accuracy of FasterTransformer under INT8
part of README.
from fastertransformer.
Thanks for your reply.
Besides, I would appreciate it if the mechanism and optimizations taken for INT8 will be made more clearly in the README.
from fastertransformer.
Related Issues (20)
- cuSPARSELt is slower? HOT 1
- Whether fastertransformer supports gpt-2 classification model, such as GPT2ForSequenceClassification?
- Supporting for expert parallelism in MoE inference
- Is llama2 70b supported? Do you know minimal configuration? HOT 1
- How to serving multi-gpu inference? HOT 1
- How to get started?
- Sparsity support
- repetition_penalty logic in FT has bug HOT 1
- can support decoder only bart? such as MBartForCausalLM
- error You need C++17 to compile PyTorch
- Does FasterTransformer support multi-stream pipeline parallelism ?
- multi_block_mode performance issue HOT 1
- Confidence is not returned in the decoding example?
- on H800 can not exec nvidia/pytorch:23.09-py3 container success
- Are `fuseQKV masked attention` and Flash Attention the same?
- what is the mean of EFF-FT?
- How to know the correspondence between versions vcr.io/nvidia/pytorch:xx.xx-py3 and pytorch?
- error: ‘CUDNN_DATA_BFLOAT16’ was not declared in this scope; did you mean ‘CUDNN_DATA_FLOAT’
- bug: memory of position_encoding_table is not malloced correctly.
- can be used in diffusion models,like sd and sdxl? how?where is the demos?tks
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fastertransformer.