Hi, I'm Vachan. I like to build and train Deep Neural Networks from scratch.

• Implemented Neural Network (Forward and Backward Propagation), Batchnorm and Layernorm, Dropout from scratch without using torch.nn, used only torch.tensor
• Neural Networks => nn.ipynb
  • Logistic Regression
  • MLP
    • Forward Propagation (Explained on Pen and Paper)
    • Back Propagation (Equations Derived on Pen and Paper)
    • Gradient Descent
    • Train Loop
    • Results
• Batch-Normalization and Layer-Normalization: Why When Where & How? => batchnorm.ipynb, layernorm.ipynb
  • Batch-Normalization
  • Layer-Normalization
  • Comparision
• Dropout: Why When Where & How? => dropout.ipynb, dropout_scale.ipynb
  • Comparision before and after scaling the model => dropout_scale.ipynb, nn_scale.ipynb
• Adam and AdamW
  • Adam
  • AdamW

• GPT written in jax, trained on tiny shakespeare dataset (1.1 MB text data) and scaled it on the tiny stories dataset (~2 GB text data)

  Model-Params	d_model	n_heads	maximum_context_length	num_layers	vocab_size	Estimated Validation Loss on tiny stories dataset
  280K	64	8	512	5	512	1.33
  15M	288	6	256	6	32000	1.19
  45M	512	8	1024	8	32000	TODO
  110M	768	12	2048	12	32000	TODO

• Vision Transformers in jax, trained on MNIST dataset

• Going to make a website for music generation completely from scratch using Pytorch. On-going project...

• Models for this Project

  • • Non Autoregressive Transformer
  • • Autoregressive Transformer
  • • Diffusion Transformer