FlashAttention Bug about vila

[Li-Qingyun]

V100s are Tesla architecture, not Ampere GPUs. You may require A100s.

[rzyfrank]

Can I not use flash-attention in Inference?

[Li-Qingyun]

ok just remove flash-attn-related codes.

[Lyken17]

For llama based models, FlashAttn requires >V100 >A6000 GPUs to run because of SRAM limit.

You may change the flash attn related codes to better adapt your hardware. For inference, our sister project https://github.com/mit-han-lab/llm-awq provides a good support for vila (both quantize and fp16).

[DwanZhang-AI]

I mute all the flash_attention_2, it still tells me that the model is using flash_attention_2

[NuyoaHygge]

I mute all the flash_attention_2, it still tells me that the model is using flash_attention_2

lib/python3.10/site-packages/transformers/models/llama/modeling_llama.py Lines 608-609, swap the annotation.