Add support for LLaVA model about multimodal

[ebsmothers]

@theadamsabra if not, you are more than welcome to take it up

[theadamsabra]

@ebsmothers thanks! If I don't get a response by tomorrow I'll just pick it up myself

[youssefadr]

Thanks for your answer @ebsmothers, I would like to add the model to torchmultimodal/models first.

[ebsmothers]

That sounds reasonable to me. We already have CLIP visual encoders in the library here, so feel free to reuse those. Then the bulk of the work for the model should be to add the LLM. A couple pointers to help with that: TransformerDecoderLayer, RMSNorm. We also have an open PR for rotary positional embeddings (#450) that might be useful. Let me know if this makes sense, happy to provide more details as needed.

[youssefadr]

Nice ! I'll come back to you with more questions later, not sure I'll start working on it this week.

[ebsmothers]

Hi @youssefadr, thanks for opening this issue. LLaVA is definitely something we're interested in adding and we would be happy to have you contribute. Is there a specific portion of the model you're especially interested in helping out with?

[theadamsabra]

@youssefadr have you worked on this to any capacity? i'm interested in picking this up if not