Android App to run Llama-v2-7B-Chat Quantized INT4 on my Android Device about ai-hub-models

[bhushan23]

@swb1234554321 @taeyeonlee we are aware of this and are actively working on this with other groups within Qualcomm.
We will be able to release sample app soon once "GenAI" dependencies are released in QNN SDK.

We will update on this issue once we can release sample app.

[AndreaChiChengdu]

hi friend, qcom released a tutorial for deploy llama2 on 8gen3 in ai stack, it maybe helpful

[swb1234554321]

hi friend, qcom released a tutorial for deploy llama2 on 8gen3 in ai stack, it maybe helpful

@AndreaChiChengdu I've tried to follow the tutorial but it turns out to need the “GenAI” feature in QNN.
However, there is no guide on how to get the permission....so lots of genAI related executable & libs missing and the tutorial is just failed.

Do you have any clue on this?

[MenghuaZheng]

hi friend, qcom released a tutorial for deploy llama2 on 8gen3 in ai stack, it maybe helpful

Hi，Can you shared this tutorial? I dont find it.

Best regards.

[dirtdust]

@swb1234554321 @taeyeonlee we are aware of this and are actively working on this with other groups within Qualcomm. We will be able to release sample app soon once "GenAI" dependencies are released in QNN SDK.

We will update on this issue once we can release sample app.

@bhushan23 Thanks for your great work, when can you release the sample app? we are all looking forward to it, especially how to run those downloaded files using QNN

[yolanda1224git]

hi , i have a question.
In ai-hub, llama2-7B model is divided into 4 parts, and each part can run inference job seperetedly. We will get 4 bins as a result, which are like job_jogk97en5_optimized_bin_m6qek5zyq.bin.

Can I run this 4 parts as a whole model using QNN?

[yolanda1224git]

hi friend, qcom released a tutorial for deploy llama2 on 8gen3 in ai stack, it maybe helpful

@AndreaChiChengdu I've tried to follow the tutorial but it turns out to need the “GenAI” feature in QNN. However, there is no guide on how to get the permission....so lots of genAI related executable & libs missing and the tutorial is just failed.

Do you have any clue on this?

hi， can you share the tutorial link, thank u so much