rasbt/LLMs-from-scratch
Implement a ChatGPT-like LLM in PyTorch from scratch, step by step
Topic: pretraining
424 repositories
LlamaChinese/Llama-Chinese
Llama中文社区，实时汇总最新Llama学习资料，构建最好的中文Llama大模型开源生态，完全开源可商用
microsoft/LMOps
General technology for enabling AI capabilities w/ LLMs and MLLMs
OFA-Sys/OFA
Official repository of OFA (ICML 2022). Paper: OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Framework
X-PLUG/mPLUG-Owl
mPLUG-Owl: The Powerful Multi-modal Large Language Model Family
sapientinc/HRM-Text
HRM-Text is a 1B text generation model based on the HRM architecture, strengthened by task completion and latent space reasoning.
ChandlerBang/awesome-self-supervised-gnn
Papers about pretraining and self-supervised learning on Graph Neural Networks (GNN).
keyu-tian/SparK
[ICLR'23 Spotlight🔥] The first successful BERT/MAE-style pretraining on any convolutional network; Pytorch impl. of "Designing BERT for Convolutional Networks: Sparse and Hierarchical Masked Modeling"
InternScience/GraphGen
GraphGen: Enhancing Supervised Fine-Tuning for LLMs with Knowledge-Driven Synthetic Data Generation
deepmodeling/Uni-Mol
Official Repository for the Uni-Mol Series Methods
yuewang-cuhk/awesome-vision-language-pretraining-papers
Recent Advances in Vision and Language PreTrained Models (VL-PTMs)
qqlu/Entity
EntitySeg Toolbox: Towards Open-World and High-Quality Image Segmentation
OpenWAM-Official/OpenWAM
Official repository for "OpenWAM: An Open, Modular Exploration Towards Systematic World–Action Model Pretraining".
seal-rg/recurrent-pretraining
Pretraining and inference code for a large-scale depth-recurrent language model
YehLi/xmodaler
X-modaler is a versatile and high-performance codebase for cross-modal analytics(e.g., image captioning, video captioning, vision-language pre-training, visual question answering, visual commonsense reasoning, and cross-modal retrieval).
PKU-YuanGroup/LanguageBind
【ICLR 2024🔥】 Extending Video-Language Pretraining to N-modality by Language-based Semantic Alignment
zubair-irshad/Awesome-Robotics-3D
A curated list of 3D Vision papers relating to Robotics domain in the era of large models i.e. LLMs/VLMs, inspired by awesome-computer-vision, including papers, codes, and related websites
Alibaba-MIIL/ImageNet21K
Official Pytorch Implementation of: "ImageNet-21K Pretraining for the Masses"(NeurIPS, 2021) paper
AGI-Arena/MARS
The official implementation of MARS: Unleashing the Power of Variance Reduction for Training Large Models
Coobiw/MPP-LLaVA
Personal Project: MPP-Qwen14B & MPP-Qwen-Next(Multimodal Pipeline Parallel based on Qwen-LM). Support [video/image/multi-image] {sft/conversations}. Don't let the poverty limit your imagination! Train your own 8B/14B LLaVA-training-like MLLM on RTX3090/4090 24GB.
baidu-baige/LoongForge
A high-performance framework for training LLMs, VLMs, diffusion, and embodied models on NVIDIA GPUs and Kunlun XPUs. Supports Pi0.5, GR00T-N1.7, FastWAM, Cosmos3, DreamZero, DeepSeek-V4, GLM-5.3, Kimi-K3 and more.
cxcscmu/Craw4LLM
Official repository for "Craw4LLM: Efficient Web Crawling for LLM Pretraining"
westlake-repl/SaProt
Saprot: Protein Language Model with Structural Alphabet (AA+3Di)
PITI-Synthesis/PITI
PITI: Pretraining is All You Need for Image-to-Image Translation
PaddlePaddle/PaddleFleetX
飞桨大模型开发套件，提供大语言模型、跨模态大模型、生物计算大模型等领域的全流程开发工具链。
tensorgi/TPA
[NeurIPS 2025 Spotlight] TPA: Tensor ProducT ATTenTion Transformer (https://arxiv.org/abs/2501.06425)
michiyasunaga/LinkBERT
[ACL 2022] LinkBERT: A Knowledgeable Language Model 😎 Pretrained with Document Links
microsoft/AzureML-BERT
End-to-End recipes for pre-training and fine-tuning BERT using Azure Machine Learning Service
amazon-science/bigdetection
BigDetection: A Large-scale Benchmark for Improved Object Detector Pre-training
showlab/UniVTG
[ICCV 2023] UniVTG: Towards Unified Video-Language Temporal Grounding