Hi, I'm Vachan. I like to build and train Deep Neural Networks from scratch.
- Implemented Neural Network (Forward and Backward Propagation), Batchnorm and Layernorm, Dropout from scratch without using
torch.nn
, used onlytorch.tensor
- Neural Networks => nn.ipynb
- Batch-Normalization and Layer-Normalization: Why When Where & How? => batchnorm.ipynb, layernorm.ipynb
- Dropout: Why When Where & How? => dropout.ipynb, dropout_scale.ipynb
- Adam and AdamW
- GPT written in
jax
, trained ontiny shakespeare dataset (1.1 MB text data)
and scaled it on thetiny stories dataset (~2 GB text data)
Model-Params d_model
n_heads
maximum_context_length
num_layers
vocab_size
Estimated Validation Loss on tiny stories dataset 280K 64 8 512 5 512 1.33 15M 288 6 256 6 32000 1.19 45M 512 8 1024 8 32000 TODO 110M 768 12 2048 12 32000 TODO
- Vision Transformers in
jax
, trained onMNIST
dataset
- Going to make a website for music generation completely from scratch using Pytorch. On-going project...
-
-
- Non Autoregressive Transformer
-
- Autoregressive Transformer
-
- Diffusion Transformer
-