strivin0311 / long-llms-learning

A repository sharing the literatures about long-context large language models, including the methodologies and the evaluation benchmarks

long-llms-learning

A repository sharing the panorama of the methodology literature on Transformer architecture upgrades in Large Language Models for handling extensive context windows, with real-time updating the newest published works.

Overview

Survey

For a clear taxonomy and more insights about the methodology, you can refer to our survey: Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Survey with a overview shown below

Flash-ReRoPE

We have augmented the great work rerope by Su with flash-attn kernel to combine rerope's infinite postional extrapolation capability with flash-attn's efficience, named as flash-rerope.

You can find and use the implementation as a flash-attn-like interface function here, with a simple precision and flops test script here.

Or you can further see how to implement llama attention module with flash-rerope here.

Latest News

Latest Works

• [2024.07]

  • FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision, located here in this repo.

  • MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attention, located here in this repo.

• [2024.04] Linear Attention Sequence Parallelism, located here in this repo.

• [2024.02] Data Engineering for Scaling Language Models to 128K Context, located here in this repo.

• [2024.01] Lightning Attention-2: A Free Lunch for Handling Unlimited Sequence Lengths in Large Language Models, located here in this repo.

Latest Baselines

• [2024.3.18] Longer than long, the Kimi AI assistant launches 200w lossless context window, located here in this repo.

• [2024.01.30] chatglm-6b-128k with $L_{max}$ 128k, located here in this repo.

• [2024.01.25] gpt-4-turbo-preview with $L_{max}$ 128k, located here in this repo.

Latest Benchmarks

• [2023.12.19] InfiniteBench, located here in this repo.
• [2023.08.29] LongBench, located here in this repo.

More to Learn

Long-LLMs-Evals

• We've also released a building repo long-llms-evals as a pipeline to evaluate various methods designed for general / specific LLMs to enhance their long-context capabilities on well-known long-context benchmarks.

LLMs-Learning

• This repo is also a sub-track for another repo llms-learning, where you can learn more technologies and applicated tasks about the full-stack of Large Language Models.

Table of Contents

• Methodology
  • Efficient Attention
    • Local Attention
    • Sparse Attention
    • Hierarchical Attention
    • Approximated Attention
    • IO-Aware Attention
  • Long-Term Memory
    • Internal MemoryCache
    • External MemoryBank
  • Extrapolative PEs
    • Attention Bias
    • Extended RoPE
  • Context Processing
    • Context Selection
    • Context Aggregation
    • Context Compression
  • Miscellaneous
    • Specific Objectives
    • Specific Activation Functions
    • Mixture of Experts
    • Parallelism
    • Weight Compression
    • Long-contenxt Training
• Evaluation
  • Datasets
  • Metrics
  • Baselines
• Tookits
• Empirical Study & Survey

Contribution

If you want to make contribution to this repo, you can just make a pr / email us with the link to the paper(s) or use the format as below:

• (un)read paper format:

#### <paper title> [(UN)READ]

paper link: [here](<link address>)

xxx link: [here](<link address>)

citation:
<bibtex citation>

Citation

If you find the survey or this repo helpful in your research or work, you can cite our paper as below:

@misc{huang2024advancing,
      title={Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Survey}, 
      author={Yunpeng Huang and Jingwei Xu and Junyu Lai and Zixu Jiang and Taolue Chen and Zenan Li and Yuan Yao and Xiaoxing Ma and Lijuan Yang and Hao Chen and Shupeng Li and Penghao Zhao},
      year={2024},
      eprint={2311.12351},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

long-llms-learning's Issues

Our work for long-llms in a new perspective, thanks if studied

CoCA: Fusing position embedding with Collinear Constrained Attention for fine-tuning free context window extending

https://arxiv.org/abs/2309.08646

Thanks and new papers

Thanks a lot for the survey, very helpful for me to get started with the literature! There are recent relevant papers I read about but not included in the paper: https://arxiv.org/abs/2305.19370, https://arxiv.org/abs/2310.08560, https://arxiv.org/abs/2310.01889.

It might be useful to add a note about best ways to contribute to this repo such as via pull request.