ifromeast / cuda_learning

learning how CUDA works

cuda_learning

learning how CUDA works

project list:

• custom op [Done]

  • CUDA 编程基础

• memory & reduction [Done]

  • GPU的内存体系及其优化指南

• Gemm [Done]

  • 通用矩阵乘法：从入门到熟练

• Transformer [Done]

  • 基础算子：
    • LayerNorm 算子的 CUDA 实现与优化
    • SoftMax 算子的 CUDA 实现与优化
    • Cross Entropy 的 CUDA 实现
    • AdamW 优化器的 CUDA 实现
    • 激活函数与残差连接的 CUDA 实现
    • embedding 层与 LM head 层的 CUDA 实现
  • 核心模块
    • self-attention 的 CUDA 实现及优化 (上)
    • self-attention 的 CUDA 实现及优化 (下)

• CUDA mode lectures [WIP]

cuda_learning's Issues

transformer项目中的tokenizer.h不存在？

导致make train_gpt2cu编译失败

softmax_forward_kernel2精度不对齐

softmax_forward_kernel2我本地运行为什么精度不对呀
是因为分段求和导致的嘛？
将这一行改为double还是不行

cuda_learning/04_transformer/ops/softmax_forward.cu

Line 155 in a5def4e

float sumval = 0.0f;

报错如下：

Using kernel 2
Largest output is: 1.000000
Checking block size 32.
0.000006 0.000006
0.000009 0.000009
0.000021 0.000021
0.000007 0.000007
0.000010 0.000010
Mismatch of out at 82072: CPU_ref: 0.758795 vs GPU: 0.758635
Mismatch of out at 102789: CPU_ref: 0.997274 vs GPU: 0.996691
Mismatch of out at 254777: CPU_ref: 0.999091 vs GPU: 0.998002
Mismatch of out at 520595: CPU_ref: 0.999906 vs GPU: 0.999745
Mismatch of out at 565770: CPU_ref: 0.998760 vs GPU: 0.998198
Mismatch of out at 655980: CPU_ref: 0.998224 vs GPU: 0.997693
Mismatch of out at 969710: CPU_ref: 0.989513 vs GPU: 0.989766
Mismatch of out at 1086046: CPU_ref: 0.989720 vs GPU: 0.989867
Mismatch of out at 1112891: CPU_ref: 0.999857 vs GPU: 0.999680
Mismatch of out at 1256552: CPU_ref: 0.999830 vs GPU: 0.999130

Process finished with exit code 1

这是我的设备信息：

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Tue_May__3_19:00:59_Pacific_Daylight_Time_2022
Cuda compilation tools, release 11.7, V11.7.64
Build cuda_11.7.r11.7/compiler.31294372_0