maximevandegar / papers-in-100-lines-of-code

Implementation of papers in 100 lines of code.

Papers in 100 Lines of Code

Implementation of papers in 100 lines of code.

Implemented papers

[Maxout Networks]

• Maxout Networks [arXiv]
• Ian J. Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron Courville, Yoshua Bengio
• 2013-02-18

[Playing Atari with Deep Reinforcement Learning]

• Playing Atari with Deep Reinforcement Learning [arXiv]
• Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Alex Graves, Ioannis Antonoglou, Daan Wierstra, Martin Riedmiller
• 2013-12-19

[Auto-Encoding Variational Bayes]

• Auto-Encoding Variational Bayes [arXiv]
• Diederik P Kingma, Max Welling
• 2013-12-20

[Generative Adversarial Networks]

• Generative Adversarial Networks [arXiv]
• Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, Yoshua Bengio
• 2014-06-10

[Conditional Generative Adversarial Nets]

• Conditional Generative Adversarial Nets [arXiv]
• Mehdi Mirza, Simon Osindero
• 2014-11-06

[Adam: A Method for Stochastic Optimization]

• Adam: A Method for Stochastic Optimization [arXiv]
• Diederik P. Kingma, Jimmy Ba
• 2014-12-22

[NICE: Non-linear Independent Components Estimation]

• NICE: Non-linear Independent Components Estimation [arXiv]
• Laurent Dinh, David Krueger, Yoshua Bengio
• 2014-10-30

[Deep Unsupervised Learning using Nonequilibrium Thermodynamics]

• Deep Unsupervised Learning using Nonequilibrium Thermodynamics [arXiv]
• Jascha Sohl-Dickstein, Eric A. Weiss, Niru Maheswaranathan, Surya Ganguli
• 2015-03-12

[Variational Inference with Normalizing Flows]

• Variational Inference with Normalizing Flows [arXiv]
• Danilo Jimenez Rezende, Shakir Mohamed
• 2015-05-21

[Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks]

• Convolutional Generative Adversarial Networks [arXiv]
• Alec Radford, Luke Metz, Soumith Chintala
• 2015-11-19

[Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)]

• Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) [arXiv]
• Djork-Arné Clevert, Thomas Unterthiner, Sepp Hochreiter
• 2015-11-23

[Adversarially Learned Inference]

• Adversarially Learned Inference [arXiv]
• Vincent Dumoulin, Ishmael Belghazi, Ben Poole, Olivier Mastropietro, Alex Lamb, Martin Arjovsky, Aaron Courville
• 2016-06-02

[Improved Techniques for Training GANs]

• Improved Techniques for Training GANs [arXiv]
• Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, Xi Chen
• 2016-06-10

[Gaussian Error Linear Units (GELUs)]

• Gaussian Error Linear Units (GELUs) [arXiv]
• Dan Hendrycks, Kevin Gimpel
• 2016-06-27

[Least Squares Generative Adversarial Networks]

• Least Squares Generative Adversarial Networks [arXiv]
• Xudong Mao, Qing Li, Haoran Xie, Raymond Y.K. Lau, Zhen Wang, Stephen Paul Smolley
• 2016-11-13

[Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks]

• Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks [arXiv]
• Chelsea Finn, Pieter Abbeel, Sergey Levine
• 2017-03-09

[Adversarial Feature Learning]

• Adversarial Feature Learning [arXiv]
• Jeff Donahue, Philipp Krähenbühl, Trevor Darrell
• 2017-04-03

[Self-Normalizing Neural Networks]

• Self-Normalizing Neural Networks [arXiv]
• Günter Klambauer, Thomas Unterthiner, Andreas Mayr, Sepp Hochreiter
• 2017-06-08

[Deep Image Prior]

• Deep Image Prior [arXiv]
• Dmitry Ulyanov, Andrea Vedaldi, Victor Lempitsky
• 2017-11-29

[On First-Order Meta-Learning Algorithms]

• On First-Order Meta-Learning Algorithms [arXiv]
• Alex Nichol, Joshua Achiam, John Schulman
• 2018-03-08

[Sequential Neural Likelihood]

• Sequential Neural Likelihood: Fast Likelihood-free Inference with Autoregressive Flows [arXiv]
• George Papamakarios, David C. Sterratt, Iain Murray
• 2018-05-18

[On the Variance of the Adaptive Learning Rate and Beyond]

• On the Variance of the Adaptive Learning Rate and Beyond [arXiv]
• Liyuan Liu, Haoming Jiang, Pengcheng He, Weizhu Chen, Xiaodong Liu, Jianfeng Gao, Jiawei Han
• 2019-08-08

[Optimizing Millions of Hyperparameters by Implicit Differentiation]

• Optimizing Millions of Hyperparameters by Implicit Differentiation [PMLR]
• Jonathan Lorraine, Paul Vicol, David Duvenaud
• 2019-10-06

[Implicit Neural Representations with Periodic Activation Functions]

• Implicit Neural Representations with Periodic Activation Functions [arXiv]
• Vincent Sitzmann, Julien N. P. Martel, Alexander W. Bergman, David B. Lindell, Gordon Wetzstein
• 2020-06-17

[Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains]

• Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domains [arXiv]
• Matthew Tancik, Pratul P. Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan T. Barron, Ren Ng
• 2020-06-18

[Denoising Diffusion Probabilistic Models]

• Denoising Diffusion Probabilistic Models [arXiv]
• Jonathan Ho, Ajay Jain, Pieter Abbeel
• 2020-06-19

[Likelihood-free MCMC with Amortized Approximate Ratio Estimators]

• Likelihood-free MCMC with Amortized Approximate Ratio Estimators [PMLR]
• Joeri Hermans, Volodimir Begy, Gilles Louppe
• 2020-06-26

[NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis]

• NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis [arXiv]
• Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, Ren Ng
• 2020-08-03

[Multiplicative Filter Networks]

• Multiplicative Filter Networks [OpenReview]
• Rizal Fathony, Anit Kumar Sahu, Devin Willmott, J Zico Kolter
• 2020-09-28

[Learned Initializations for Optimizing Coordinate-Based Neural Representations]

• Learned Initializations for Optimizing Coordinate-Based Neural Representations [arXiv]
• Matthew Tancik, Ben Mildenhall, Terrance Wang, Divi Schmidt, Pratul P. Srinivasan, Jonathan T. Barron, Ren Ng
• 2020-12-03

[FastNeRF: High-Fidelity Neural Rendering at 200FPS]

• FastNeRF: High-Fidelity Neural Rendering at 200FPS [arXiv]
• Stephan J. Garbin, Marek Kowalski, Matthew Johnson, Jamie Shotton, Julien Valentin
• 2021-03-18

[KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs]

• KiloNeRF: Speeding up Neural Radiance Fields with Thousands of Tiny MLPs [arXiv]
• Christian Reiser, Songyou Peng, Yiyi Liao, Andreas Geiger
• 2021-03-25

[PlenOctrees for Real-time Rendering of Neural Radiance Fields]

• PlenOctrees for Real-time Rendering of Neural Radiance Fields [arXiv]
• Alex Yu, Ruilong Li, Matthew Tancik, Hao Li, Ren Ng, Angjoo Kanazawa
• 2021-03-25

[NeRF--: Neural Radiance Fields Without Known Camera Parameters]

• NeRF--: Neural Radiance Fields Without Known Camera Parameters [arXiv]
• Zirui Wang, Shangzhe Wu, Weidi Xie, Min Chen, Victor Adrian Prisacariu
• 2021-02-14

[Gromov-Wasserstein Distances between Gaussian Distributions]

• Gromov-Wasserstein Distances between Gaussian Distributions [arXiv]
• Antoine Salmona, Julie Delon, Agnès Desolneux
• 2021-08-16

[Plenoxels: Radiance Fields without Neural Networks]

• Plenoxels: Radiance Fields without Neural Networks [arXiv]
• Alex Yu, Sara Fridovich-Keil, Matthew Tancik, Qinhong Chen, Benjamin Recht, Angjoo Kanazawa
• 2021-12-09

[InfoNeRF: Ray Entropy Minimization for Few-Shot Neural Volume Rendering]

• InfoNeRF: Ray Entropy Minimization for Few-Shot Neural Volume Rendering [arXiv]
• Mijeong Kim, Seonguk Seo, Bohyung Han
• 2021-12-31

[K-Planes: Explicit Radiance Fields in Space, Time, and Appearance]

• K-Planes: Explicit Radiance Fields in Space, Time, and Appearance [arXiv]
• Sara Fridovich-Keil, Giacomo Meanti, Frederik Warburg, Benjamin Recht, Angjoo Kanazawa
• 2023-01-24

[FreeNeRF: Improving Few-shot Neural Rendering with Free Frequency Regularization]

• FreeNeRF: Improving Few-shot Neural Rendering with Free Frequency Regularization [arXiv]
• Jiawei Yang, Marco Pavone, Yue Wang
• 2023-03-13

papers-in-100-lines-of-code's Issues

shoudl be if __name__ == '__main__':

Papers-in-100-Lines-of-Code/Neural_Radiance_Fields_Without_Known_Camera_Parameters/nerfmm.py

Line 180 in d8747ad

if __name__ == 'main':

if __name__ == '__main__' in kilo_nerf.py

There are missing underscores in the file in line number 138. I have created a pr for the same #14

Regrading to "NeRF_Representing_Scenes_as_Neural_Radiance_Fields_for_View_Synthesis"

Hi Max,

Thanks very much for your detailed tutorial on Youtube. But I have one thing pretty confused about. Taking the training dataset as an example, where the shape is 16,000,000 by 9, I see the first three numbers are the ray_origins, the mid three are the ray_directions, and the last three are the RGB colors. I am wondering how the ray origins and ray directions are computed. The problem may be tedious, but I think it would very helpful if it could be included in the video or the code.

DefaultCPUAllocator

Hi Maxime

I tried to run the repo as it is. But I am getting the below error. Is there any resolution to resolve this issue. I am using my CPU instead of my GPU is that the cause for this issue.

" RuntimeError: [enforce fail at C:\b\abs_bao0hdcrdh\croot\pytorch_1675190257512\work\c10\core\impl\alloc_cpu.cpp:81] data. DefaultCPUAllocator: not enough memory: you tried to allocate 1981808640 bytes."

The error is while running the test function(below 2 lines of the code).

100-lines K-planes is NOT equals to official K-planes?

e.g.

1. with 'Varying Appearance'
2. with 'Dynamic Volums'

this 100-lines version just implemented: Static-Volums? 3-planes + Hadamard-Product + 2 encoders (spatially-localized to density and ...; ... and direction to RGB)

How about trying this: https://github.com/google-research/nerf-from-image

Hey man,
Very nice work.
Any interests to replicate this work ? I found it very large and difficult.
Never mind if not interested :-)

Scaling value

Hi! Thank you very much for sharing this repo! It is very helpful for understanding all the different models out there!

Now I'm trying to understand DDPMs and found that in the attention block, you set the score value of the NIN block to 0.0. Since this NIN block is applied to the last value of the attention block, I think it makes the whole computation equal to 0.0, making the whole attention block equivalent to the identity. Maybe a scaling factor different from zero should be used here?

And why do you prefer applying the NIN block compared to 2D convolutions with kernel size (1,1), stride (1,1), and zero padding?

Thank you!

Papers-in-100-Lines-of-Code/Denoising_Diffusion_Probabilistic_Models/unet.py

Line 125 in 2decf0d

self.nin = Nin(ch, ch, scale=0.)

Question on Using Custom Datasets with FastNeRF

Hi,

I've been working with FastNeRF and I am interested in extending its functionality to work with custom datasets. I've noticed that the code currently works with specific pickle files, and I'm looking for guidance on how to adapt it for different data formats or sources.

Could you provide some insights or documentation on how to go about this? Are there specific requirements or constraints for the data that I should be aware of when trying to use my own datasets?

Any help or pointers you could provide would be greatly appreciated.

Thanks for your great work on this project!

Kamran

hi, bro, i want to contribute the smallest GPT, just 88 lines. logic correct, with train.

import torch

class CausalSelfAttention(torch.nn.Module):
    def __init__(self, n_embd, n_head, block_size, use_bias):
        super().__init__()
        self.n_embd = n_embd
        self.n_head = n_head
        self.c_attn = torch.nn.Linear(n_embd, 3 * n_embd, bias=use_bias)
        self.c_proj = torch.nn.Linear(n_embd, n_embd, bias=use_bias)
        self.register_buffer("bias", torch.tril(torch.ones(block_size, block_size)).view(1, 1, block_size, block_size))

    def forward(self, x):
        batch, sequence, embed = x.size()
        q, k ,v = self.c_attn(x).split(self.n_embd, dim=2)
        q = q.view(batch, sequence, self.n_head, embed // self.n_head).transpose(1, 2)
        k = k.view(batch, sequence, self.n_head, embed // self.n_head).transpose(1, 2)
        v = v.view(batch, sequence, self.n_head, embed // self.n_head).transpose(1, 2)
        a = (q @ k.transpose(-2, -1)) * (1.0 / torch.sqrt(torch.tensor(k.size(-1))))
        a = a.masked_fill(self.bias[:,:,:sequence,:sequence] == 0, float('-inf'))
        a = torch.nn.functional.softmax(a, dim=-1)
        y = a @ v
        y = y.transpose(1, 2).contiguous().view(batch, sequence, embed)
        y = self.c_proj(y)
        return y

class ResidualBlock(torch.nn.Module):
    def __init__(self, n_embd, n_head, block_size, use_bias):
        super().__init__()
        self.lna = torch.nn.LayerNorm(n_embd)
        self.att = CausalSelfAttention(n_embd, n_head, block_size, use_bias)
        self.lnb = torch.nn.LayerNorm(n_embd)
        self.mlp = torch.nn.Sequential(torch.nn.Linear(n_embd, 4 * n_embd, bias=use_bias), torch.nn.GELU(), torch.nn.Linear(4 * n_embd, n_embd, bias=use_bias))

    def forward(self, x):
        x = x + self.att(self.lna(x))
        x = x + self.mlp(self.lnb(x))
        return x

class GPT(torch.nn.Module):
    def __init__(self, vocab_size, block_size, n_embd, n_head, n_layer, use_bias):
        super().__init__()
        self.te = torch.nn.Embedding(vocab_size, n_embd)
        self.pe = torch.nn.Embedding(block_size, n_embd)
        self.hi = torch.nn.Sequential(*[ResidualBlock(n_embd, n_head, block_size, use_bias) for _ in range(n_layer)])
        self.ln = torch.nn.LayerNorm(n_embd)
        self.lm = torch.nn.Linear(n_embd, vocab_size, bias=False)

    def forward(self, idx):
        tok_emb = self.te(idx)
        pos_emb = self.pe(torch.arange(0, idx.size()[1], dtype=torch.long, device=idx.device).unsqueeze(0))
        x = tok_emb + pos_emb
        x = self.hi(x)
        logits = self.lm(self.ln(x)[:, -1, :]) 
        return logits

def main(seq=list(map(int, "11100110110")), vocab_size=2, block_size=3, n_embd=16, n_head=4, n_layer=4, ):
    x, y = [], []
    for i in range(len(seq)-block_size):
        x.append(seq[i:i+block_size])
        y.append(seq[i+block_size])
        print('sample:', x[-1], '-->', y[-1])
    X, Y = torch.tensor(x, dtype=torch.long), torch.tensor(y, dtype=torch.long)

    network = GPT(vocab_size, block_size, n_embd, n_head, n_layer, use_bias=False)
    optimizer = torch.optim.AdamW(network.parameters(), lr=0.001, weight_decay=0.1)
    for epoch in range(1, 100+1):
        logits = network(X)
        loss = torch.nn.functional.cross_entropy(logits, Y)
        loss.backward()
        optimizer.step()
        optimizer.zero_grad()
        if epoch%10==0: print('train:', 'epoch=%03d'%(epoch), 'loss=%.4f'%(loss.item()))

    def all_possible_full_permutation(n, k):
        if k == 0:
            yield []
        else:
            for i in range(n): 
                for c in all_possible_full_permutation(n, k-1): yield [i] + c
    for x in all_possible_full_permutation(vocab_size, block_size):       
        logits = network(torch.tensor(x, dtype=torch.long)[None, ...])
        probs = torch.nn.functional.softmax(logits, dim=-1)
        print('infer:', x, '-->', torch.argmax(probs[0]).tolist(), '    ', [round(p,2) for p in probs[0].tolist()])

if __name__ == '__main__':
    torch.manual_seed(333)
    main()