Topic: multimodal-learning Goto Github

👇 Here are 219 public repositories matching this topic...

ailab-cvc / unireplknet
multimodal-learning,[CVPR'24] UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio, Video, Point Cloud, Time-Series and Image Recognition
Organization: ailab-cvc
Home Page: https://arxiv.org/abs/2311.15599
architecture artificial-intelligence convolutional-neural-networks deep-learning multimodal-learning

antoyang / frozenbilm
multimodal-learning,[NeurIPS 2022] Zero-Shot Video Question Answering via Frozen Bidirectional Language Models
User: antoyang
Home Page: https://arxiv.org/abs/2206.08155
multimodal-learning video-understanding vqa weakly-supervised-learning large-language-models pre-training video-question-answering videoqa vision-and-language visual-question-answering

antoyang / just-ask
multimodal-learning,[ICCV 2021 Oral + TPAMI] Just Ask: Learning to Answer Questions from Millions of Narrated Videos
User: antoyang
Home Page: https://arxiv.org/abs/2012.00451
vqa visual-question-answering videoqa video-question-answering video-understanding question-generation weakly-supervised-learning vision-and-language pre-training multimodal-learning

antoyang / tubedetr
multimodal-learning,[CVPR 2022 Oral] TubeDETR: Spatio-Temporal Video Grounding with Transformers
User: antoyang
spatio-temporal-video-grounding stvg vidstg hc-stvg vision-and-language multimodal-learning video-understanding visual-grounding

antoyang / vidchapters
multimodal-learning,[NeurIPS 2023 D&B] VidChapters-7M: Video Chapters at Scale
User: antoyang
Home Page: http://arxiv.org/abs/2309.13952
dense-video-captioning multimodal-learning pre-training temporal-language-grounding video-captioning video-understanding vision-and-language weakly-supervised-learning vid2seq video-chapter-generation

arrowluo / clip4clip
multimodal-learning,An official implementation for "CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval"
User: arrowluo
Home Page: https://arxiv.org/abs/2104.08860
video-text-retrieval multimodal-learning multimodality multimodal search ranking retrieval-model retrieval msrvtt lsmdc

declare-lab / multimodal-deep-learning
multimodal-learning,This repository contains various models targetting multimodal representation learning, multimodal fusion for downstream tasks such as multimodal sentiment analysis.
Organization: declare-lab
multimodal-deep-learning multimodal-learning multimodal-interactions multimodal-sentiment-analysis

dmitryryumin / icassp-2023-24-papers
multimodal-learning,ICASSP 2023-2024 Papers: A complete collection of influential and exciting research papers from the ICASSP 2023-24 conferences. Explore the latest advancements in acoustics, speech and signal processing. Code included. Star the repository to support the advancement of audio and signal processing!
User: dmitryryumin
asr denoising domain-adaptation face-recognition icassp icassp2023 keyword-spotting language-modeling self-supervised-learning semantic-segmentation

dmitryryumin / iccv-2023-papers
multimodal-learning,ICCV 2023 Papers: Discover cutting-edge research from ICCV 2023, the leading computer vision conference. Stay updated on the latest in computer vision and deep learning, with code included. ⭐ support visual intelligence development!
User: dmitryryumin
Home Page: https://huggingface.co/spaces/DmitryRyumin/NewEraAI-Papers
iccv iccv2023 3d-graphics 3d-reconstruction biometrics computer-vision datasets deep-learning explainable-ai face-recognition

eurus-holmes / awesome-multimodal-research
multimodal-learning,A curated list of Multimodal Related Research.
User: eurus-holmes
awesome multimodal-research multimodal-learning multimodal

georgian-io / multimodal-toolkit
multimodal-learning,Multimodal model for text and tabular data with HuggingFace transformers as building block for text data
Organization: georgian-io
Home Page: https://multimodal-toolkit.readthedocs.io
huggingface-transformers transformer natural-language-processing tabular-data multimodal-learning

haamoon / mmtm
multimodal-learning,Implementation of CVPR 2020 paper "MMTM: Multimodal Transfer Module for CNN Fusion"
User: haamoon
multimodal-deep-learning multimodal-learning cnn-fusion pytorch action-recognition speech-enhancement gesture-recognition

henghuiding / mevis
multimodal-learning,[ICCV 2023] MeViS: A Large-scale Benchmark for Video Segmentation with Motion Expressions
User: henghuiding
Home Page: https://henghuiding.github.io/MeViS/
multimodal-learning referring-expression-comprehension referring-expression-segmentation referring-video-object-segmentation video-understanding mevis-dataset mose-dataset

henghuiding / rela
multimodal-learning,[CVPR2023 Highlight] GRES: Generalized Referring Expression Segmentation
User: henghuiding
Home Page: https://henghuiding.github.io/GRES/
multimodal-learning referring-expression-comprehension referring-expression-segmentation referring-image-segmentation vision-language-transformer cvpr2023

henryhzy / awesome-multimodal-llm
multimodal-learning,Research Trends in LLM-guided Multimodal Learning.
User: henryhzy
llm multimodal large-language-models multimodal-learning in-context-learning instruction-tuning multimodal-large-language-models parameter-efficient-learning parameter-efficient-tuning

huaizhengzhang / awsome-deep-learning-for-video-analysis
multimodal-learning,Papers, code and datasets about deep learning and multi-modal learning for video analysis
User: huaizhengzhang
deep-learning machine-learning multimodal-learning paper video-analysis video-classification video-dataset

huanglizi / lvit
multimodal-learning,[IEEE Transactions on Medical Imaging/TMI] This repo is the official implementation of "LViT: Language meets Vision Transformer in Medical Image Segmentation"
User: huanglizi
medical-image-analysis pytorch segmentation vision-language multimodal-learning

ilaria-manco / multimodal-ml-music
multimodal-learning,List of academic resources on Multimodal ML for Music
User: ilaria-manco
academic-publications awesome-list multimodal-data multimodal-learning music-ai music-information-retrieval resources music-research multimodal-deep-learning

kyegomez / cm3leon
multimodal-learning,An open source implementation of "Scaling Autoregressive Multi-Modal Models: Pretraining and Instruction Tuning", an all-new multi modal AI that uses just a decoder to generate both text and images
User: kyegomez
Home Page: https://discord.gg/qUtxnK2NMf
attention attention-is-all-you-need dalle imagegeneration multimodal multimodal-learning multimodality

kyegomez / navit
multimodal-learning,My implementation of "Patch n’ Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolution"
User: kyegomez
Home Page: https://discord.gg/qUtxnK2NMf
vit attention-mechanism clip gpt4 multimodal multimodal-deep-learning multimodal-learning multimodality

kyegomez / pali3
multimodal-learning,Implementation of PALI3 from the paper PALI-3 VISION LANGUAGE MODELS: SMALLER, FASTER, STRONGER"
User: kyegomez
Home Page: https://discord.gg/qUtxnK2NMf
artificial-intelligence autogpt gpt4 machine-learning multimodal multimodal-deep-learning multimodal-learning multimodality

machine-intelligence-laboratory / topicnet
multimodal-learning,Interface for easier topic modelling.
Organization: machine-intelligence-laboratory
Home Page: https://machine-intelligence-laboratory.github.io/TopicNet
bigartm-library topic-modelling custom-score modalities pypi topic-modeling multimodal-learning multimodal-data document-representation

mhw32 / multimodal-vae-public
multimodal-learning,A PyTorch implementation of "Multimodal Generative Models for Scalable Weakly-Supervised Learning" (https://arxiv.org/abs/1802.05335)
User: mhw32
machine-learning generative-models variational-autoencoder multimodal-learning

miccunifi / searle
multimodal-learning,[ICCV 2023] - Zero-shot Composed Image Retrieval with Textual Inversion
Organization: miccunifi
cirr composed-image-retrieval fashion-iq multimodal-learning textual-inversion knowledge-distillation circo pytorch clip

microsoft / xpretrain
multimodal-learning,Multi-modality pre-training
Organization: microsoft
multimodal-learning pre-training multimedia computer-vision nlp

mlfoundations / open_flamingo
multimodal-learning,An open-source framework for training large multimodal models.
Organization: mlfoundations
computer-vision deep-learning in-context-learning language-model multimodal-learning pytorch flamingo

mmaaz60 / mvits_for_class_agnostic_od
multimodal-learning,[ECCV'22] Official repository of paper titled "Class-agnostic Object Detection with Multi-modal Transformer".
User: mmaaz60
class-agnostic-detection multimodal-learning open-world-detection object-detection pytorch psuedo-labels

mmmu-benchmark / mmmu
multimodal-learning,This repo contains evaluation code for the paper "MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGI"
Organization: mmmu-benchmark
Home Page: https://mmmu-benchmark.github.io/
computer-vision deep-learning deep-neural-networks evaluation foundation-models large-language-models large-multimodal-models llm llms machine-learning multimodal multimodal-deep-learning multimodal-learning multimodality natural-language-processing question-answering stem visual-question-answering

mmstar-benchmark / mmstar
multimodal-learning,This repo contains evaluation code for the paper "Are We on the Right Way for Evaluating Large Vision-Language Models"
Organization: mmstar-benchmark
Home Page: https://mmstar-benchmark.github.io
evaluation large-language-models large-multimodal-models large-vision-language-model large-vision-language-models llm llms lvlm lvlms multimodal multimodal-learning multimodality visual-question-answering

njustkmg / omml
multimodal-learning,Multi-Modal learning toolkit based on PaddlePaddle and PyTorch, supporting multiple applications such as multi-modal classification, cross-modal retrieval and image caption.
User: njustkmg
multimodal multimodal-learning python paddlepaddle pytorch crossmodal-retrieval imagecaptioning classification

ofa-sys / ofasys
multimodal-learning,OFASys: A Multi-Modal Multi-Task Learning System for Building Generalist Models
Organization: ofa-sys
audio computer-vision deep-learning motion multimodal-learning multitask-learning nlp pretrained-models pytorch transformers

pliang279 / awesome-multimodal-ml
multimodal-learning,Reading list for research topics in multimodal machine learning
User: pliang279
multimodal-learning machine-learning representation-learning natural-language-processing computer-vision speech-processing robotics healthcare reading-list deep-learning

pliang279 / mfn
multimodal-learning,[AAAI 2018] Memory Fusion Network for Multi-view Sequential Learning
User: pliang279
machine-learning multimodal-learning

pliang279 / multibench
multimodal-learning,[NeurIPS 2021] Multiscale Benchmarks for Multimodal Representation Learning
User: pliang279
machine-learning multimodal-learning robotics natural-language-processing computer-vision deep-learning healthcare representation-learning speech-processing

pliang279 / multiviz
multimodal-learning,[ICLR 2023] MultiViz: Towards Visualizing and Understanding Multimodal Models
User: pliang279
computer-vision machine-learning multimodal-learning natural-language-processing

pointcept / gpt4point
multimodal-learning,[CVPR'24 Highlight] GPT4Point: A Unified Framework for Point-Language Understanding and Generation.
Organization: pointcept
Home Page: https://gpt4point.github.io/
3d-generation llm multimodal-learning

preferredai / cornac
multimodal-learning,A Comparative Framework for Multimodal Recommender Systems
Organization: preferredai
Home Page: https://cornac.preferred.ai
recommender-system recommendation-algorithms recommendation-engine matrix-factorization collaborative-filtering multimodal-learning recommendation-system multimodality

preferredai / vista-net
multimodal-learning,Code for the paper "VistaNet: Visual Aspect Attention Network for Multimodal Sentiment Analysis", AAAI'19
Organization: preferredai
sentiment-analysis multimodal-sentiment-analysis attention-mechanism multimodal-learning

pykale / pykale
multimodal-learning,Knowledge-Aware machine LEarning (KALE): accessible machine learning from multiple sources for interdisciplinary research, part of the 🔥PyTorch ecosystem. ⭐ Star to support our work!
Organization: pykale
Home Page: https://pykale.github.io/
machine-learning knowledge-aware-learning computer-vision graph-analysis pytorch medical-image-analysis multimodal-learning transfer-learning domain-adaptation data-science

richard-peng-xia / awesome-multimodal-in-medical-imaging
multimodal-learning,A collection of resources on applications of multi-modal learning in medical imaging.
User: richard-peng-xia
medical-imaging medical-report-generation multimodal-deep-learning multimodal-learning visual-question-answering large-language-models large-multimodal-models multimodal-large-language-models

sangminwoo / awesome-vision-and-language
multimodal-learning,A curated list of awesome vision and language resources (still under construction... stay tuned!)
User: sangminwoo
awesome awesome-list vision-and-language multimodal-learning

sdc17 / upop
multimodal-learning,[ICML 2023] UPop: Unified and Progressive Pruning for Compressing Vision-Language Transformers.
User: sdc17
Home Page: https://dachuanshi.com/UPop-Project/
efficient-deep-learning model-compression multimodal-learning vision-language-transformer image-captioning image-text-retrieval visual-question-answering visual-reasoning text-image-retrieval framework

snap-research / mmvid
multimodal-learning,[CVPR 2022] Show Me What and Tell Me How: Video Synthesis via Multimodal Conditioning
Organization: snap-research
Home Page: https://snap-research.github.io/MMVID/
deep-learning video-generation multimodal-learning text-to-video multimodal-video-generation transformer bert

subho406 / omninet
multimodal-learning,Official Pytorch implementation of "OmniNet: A unified architecture for multi-modal multi-task learning" | Authors: Subhojeet Pramanik, Priyanka Agrawal, Aman Hussain
User: subho406
Home Page: https://arxiv.org/abs/1907.07804
machine-learning deep-learning neural-network artificial-intelligence transformer nlp image-captioning video-recognition multitask-learning multimodal-learning

tencentarc / vit-lens
multimodal-learning,[CVPR 2024] ViT-Lens: Towards Omni-modal Representations
Organization: tencentarc
Home Page: https://ailab-cvc.github.io/seed/vitlens/
multimodal-learning

ucsc-vlaa / clipa
multimodal-learning,[NeurIPS 2023] This repository includes the official implementation of our paper "An Inverse Scaling Law for CLIP Training"
Organization: ucsc-vlaa
contrastive-learning foundation-models multimodal-learning zero-shot-classification zero-shot-learning deep-learning neurips-2023 pytorch

verlab / learning2dance_cag_2020
multimodal-learning,PyTorch implementation of our graph convolutional network (GCN) for human motion generation from music. Also with paired dance-music data for training!
Organization: verlab
computer-vision motion-synthesis motion-analysis motion-animation multimodal-learning sound-processing graph-convolutional-networks gcn graph-adversarial-learning human-motion

yilunlee / missing_aware_prompts
multimodal-learning,Multimodal Prompting with Missing Modalities for Visual Recognition, CVPR'23
User: yilunlee
Home Page: https://yilunlee.github.io/missing_aware_prompts/
transformer computer-vision visual-recognition cvpr missing-modality multimodal-learning

ys-zong / awesome-self-supervised-multimodal-learning
multimodal-learning,A curated list of self-supervised multimodal learning resources.
User: ys-zong
awesome-list machine-learning multimodal-learning self-supervised-learning

zjunlp / hvpnet
multimodal-learning,[NAACL 2022 Findings] Good Visual Guidance Makes A Better Extractor: Hierarchical Visual Prefix for Multimodal Entity and Relation Extraction
Organization: zjunlp
entity-extraction relation-extraction multimodal kg prefix hvpnet ner re naacl dataset