weidler / semlc

Biologically Inspired Lateral Connectivity in Convoluiontal Neural Networks

Hello there 👋 I hope you like what you find here 🤓

class Researcher(CuriousHuman):

    def __init__(self):
        self.name = "Tonio Weidler"
        self.role = "PhD Candidate"
        self.affiliation = "Maastricht University"
        self.residence = "The Netherlands"

        self.code = [
            "Python",
            "JavaScript",
            "Java",
            "PHP"
        ]

        self.research_field = "Neuroscience"
        self.research_topics = [
            "Sensorimotor Control",
            "Human Dexterity",
            "Goal-Driven Models",
            "Deep Learning"
        ] 

        self.language_spoken = ["de_DE", "en_US"]

    def greet(self, name):
        print(f"Hello there, {name}! I hope you like what you find here :)")

semlc's Issues

AlexNet for ImageNet + Inhibition

Clean up experiment folder

either delete or move some files to meaningful folders / subfolders to create a better overview of our experiments

Refactor HP-Opt script

so that it uses the train function from utils.

speed comparison scripts

• for different input sizes and scopes

Implement SSI with Toeplitz

Toeplitz proved to be a lot faster, so we should reimplement Toeplitz (and Recurrent?) using it.

make it work for batch

[CIFAR10] AlexNet-like Baseline (fully convolutional)

Develop baseline based on AlexNet that achieves ~90% accuracy without Locally Connected Layers and instead is fully convolutional. To match parameter count we can increase the number of layers, e.g. to 6 as this mathces the number of areas in the visual cortex. Need to HP otpim this. Potentially the literature can give hints too.

checkpoint saving

visualisation of hp opt for both ordering and accuracy

"stack"-like visualisation

allow training with augmentation

Some transormations like RandomCrop return PIL Images rather than Tensors, that our current framework does not allow. Probably not hard or time consuming but needs to be done

Implement Toeplitz Inversion with O(nlogn)

HP Testing Script

Implement Ricker Wavelet based on PyTorch s.t. we can backpropagate through the equation.

Refactor ordering visualization

I only hacked something together the night before the deadline and it is really bad code. Refactor in general.

[CIFAR10] SotA Baseline

Need current state of the art baseline model that also allows for integration of our layers.

Significance testing

Allow for Zero-Padding in all Strategies

Cleanup old branches and create static report state branch

Logging refactoring

Generate unique IDs for each model and loss files and save a lookup table in a different file for more generic saving and loading of experiments

Capsule Net + First Layer Inhibition on MNIST

Implement Ricker Wavelet with self-inhibition

Center of wavelet (self connection) should be negative.

[MNIST] SotA Baseline

Make damping a network parameter

Refactor description of old Strategies

Add description of Parametric Inhibition

Loss analysis

Analyse loss for ConvNet11 and -18 and find heuristic (e.g. hard coded epoch, loss change) for dynamic learning rate adaption to create good baseline

(maybe) training for 11% error net

11% error on CIFAR-10 - layer parameter file
Methodology:

1. Train on batches 1-4, use batch 5 for validation.
2. After about 350 epochs, validation error no longer making improvements.
3. Fold in batch 5.
4. Train on batches 1-5 for about 150 more epochs, until the batch 5 error is near the errors for batches 1-4. It takes forever to actually get there but after 150 epochs it's close enough.
5. Lower learning rates (epsW) by a factor of 10 to 0.0001, train for 10 more epochs.
6. Lower learning rates (epsW) by another factor of 10 to 0.00001, train for 10 more epochs.
7. Stop. Test on batch 6 with --test-range=6 --multiview-test=1 --logreg-name=logprob (read more about what this does here: http://code.google.com/p/cuda-convnet/wiki/TrainingNet#Training_on_image_translations )

More details about methodology: http://code.google.com/p/cuda-convnet/wiki/Methodology

reimplement conv2nets for freezed setting

for the speeds

Abstract for End of January Submission

modify logging

change our logging such that the filename can contain all possible permutations of configurations without overriding previous produced files. (e.g. inhib_depth, learn_weights=true/false, strategy etc.)

[IMAGENET] SotA Baseline

Current state of the art model for ImageNet.

Set up all HPs for AlexNet

Inhibition with Width and Damping as Learned Parameters

Implement inhibition where width and damping of the wavelet are the learned parameters. We may benchmark to versions: Firstly we can simply use the Ricker wavelet equation to obtain the filter values and add its parameters to the module. For this we need to implement our own Ricker wavelet function in PyTorch. Secondly we can approximate the Wavelet by the difference of two Gaussians, which might be faster during backpropagation. For the latter, to still only have two parameters, use the difference of standard deviations as parameter, rather than two stds.