weishengying Goto Github PK

autoawq

AutoAWQ implements the AWQ algorithm for 4-bit quantization with a 2x speedup during inference. Documentation:

autogptq

An easy-to-use LLMs quantization package with user-friendly apis, based on GPTQ algorithm.

cub

[ARCHIVED] Cooperative primitives for CUDA C++. See https://github.com/NVIDIA/cccl

cuda-learn-notes

🎉CUDA 笔记 / 大模型手撕CUDA / C++笔记，更新随缘: flash_attn、sgemm、sgemv、warp reduce、block reduce、dot product、elementwise、softmax、layernorm、rmsnorm、hist etc.

cute_exercise

Cute_exercise

cutlass

CUDA Templates for Linear Algebra Subroutines

cutlass-kernels

deepspeed

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

fastertransformer

Transformer related optimization, including BERT, GPT

flash-attention

Fast and memory-efficient exact attention

hands-on-gemm

llm-awq

AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration

llmspeculativesampling

Fast inference from large lauguage models via speculative decoding

megatron-lm

Ongoing research training transformer models at scale

moe

MoE layer for pytorch

nvidia_sgemm_practice

Step-by-step optimization of CUDA SGEMM

omniquant

[ICLR2024 spotlight] OmniQuant is a simple and powerful quantization technique for LLMs.

sgemm_cuda

Fast CUDA matrix multiplication from scratch

smoothquant

[ICML 2023] SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models

tensorrt-llm

TensorRT-LLM provides users with an easy-to-use Python API to define Large Language Models (LLMs) and build TensorRT engines that contain state-of-the-art optimizations to perform inference efficiently on NVIDIA GPUs. TensorRT-LLM also contains components to create Python and C++ runtimes that execute those TensorRT engines.

tiny-flash-attention

flash attention tutorial written in python, triton, cuda, cutlass

torch-int

This repository contains integer operators on GPUs for PyTorch.

vllm

A high-throughput and memory-efficient inference and serving engine for LLMs