es-ude / elastic-ai.creator

elastic ai.creator

ElasticAI.creator

Design, train and compile neural networks optimized specifically for FPGAs. Obtaining a final model is typically a three stage process.

• design and train it using the layers provided in the elasticai.creator.nn package.
• translate the model to a target representation, e.g. VHDL
• compile the intermediate representation with a third party tool, e.g. Xilinx Vivado (TM)

This version currently only supports parts of VHDL as target representations.

The project is part of the elastic ai ecosystem developed by the Embedded Systems Department of the University Duisburg-Essen. For more details checkout the slides at researchgate.

Table of contents

• Users Guide
  • Install
  • Minimal Example
  • General Limitations
• Structure of the Project
• Developers Guide
  • Install Dev Dependencies
  • Conventional Commit Rules
  • Adding new translation targets
  • Syntax Checking
  • Tests
    • Test Style Guidelines
      • File IO
      • Directory structure and file names
      • Unit tests
      • Integration tests
      • System tests
      • Adding new functionalities and tests required
      • Updating tests

Users Guide

Install

You can install the ElasticAI.creator as a dependency using pip:

python3 -m pip install elasticai-creator

Minimal Example

The following example shows how to use the ElasticAI.creator to define and translate a machine learning model to VHDL. It will save the generated VHDL code to a directory called build_dir.

from elasticai.creator.nn import Sequential
from elasticai.creator.nn.fixed_point import Linear, HardSigmoid
from elasticai.creator.file_generation.on_disk_path import OnDiskPath


def main() -> None:
    # Define a model
    model = Sequential(
        Linear(in_features=10, out_features=2, bias=True, total_bits=16, frac_bits=8),
        HardSigmoid(total_bits=16, frac_bits=8),
    )

    # Train the model
    run_training(model)

    # Save the VHDL code of the trained model
    destination = OnDiskPath("build_dir")
    design = model.create_design("my_model")
    design.save_to(destination)


if __name__ == "__main__":
    main()

General Limitations

By now we only support Sequential models for our translations.

Structure of the Project

The structure of the project is as follows. The creator folder includes all main concepts of our project, especially the qtorch implementation which is our implementation of quantized PyTorch layer. It also includes the supported target representations, like the subfolder nn is for the translation to vhdl. Additionally, we have unit and integration tests in the tests folder.

Developers Guide

Install Dev Dependencies

• poetry
• recommended:
  • pre-commit
  • commitlint to help following our conventional commit guidelines poetry can be installed in the following way:

pip install poetry
poetry install
poetry shell
pre-commit install
npm install --save-dev @commitlint/{config-conventional,cli}

# Optional:
sudo apt install ghdl

Project Structure

All packages and modules fall into one of five main categories and can thus be found in the corresponding package

• nn: trainable modules that can be translated to vhdl to build a hardware accelerator
• base_modules: (non-public) shared functionality and data structures, that we use to create our neural network software modules
• vhdl: (non-public) shared code that we use to represent and generate vhdl code
• file_generation: provide a very restricted api to generate files or file subtrees under a given root node and defines a basic template api and datastructure, compatible with file_generation

Glossary

• fxp/Fxp: prefix for fixed point
• flp/Flp: prefix for floating point
• x: parameter input tensor for layer with single input tensor
• y: output value/tensor for layer with single output
• _bits: suffix to denote the number of bits, e.g. total_bits, frac_bits, in python context
• _width: suffix to denote the number of bits used for a data bus in vhdl, e.g. total_width, frac_width
• MathOperations/operations: definition of how to perform mathematical operations (quantization, addition, matrix multiplication, ...)

Conventional Commit Rules

We use conventional commits (see here). The following commit types are allowed. The message scope is optional.

Commit Types
feat
fix
docs
style
refactor
revert
chore
wip
perf

Tests

Our implementation is tested with unit and integration. You can run one explicit test with the following statement:

python3 -m pytest ./tests/path/to/specific/test.py

If you want to run all tests, give the path to the tests:

python3 -m pytest ./tests

If you want to add more tests please refer to the Test Guidelines in the following.

Test Style Guidelines

File IO

In general try to avoid interaction with the filesystem. In most cases instead of writing to or reading from a file you can use a StringIO object or a StringReader. If you absolutely have to create files, be sure to use pythons tempfile module and cleanup after the tests. In most cases you can use the InMemoryPath class to write files to the RAM instead of writing them to the hard disc (especially for testing the generated VHDL files of a certain layer).

Directory structure and file names

Files containing tests for a python module should be located in a test directory for the sake of separation of concerns. Each file in the test directory should contain tests for one and only one class/function defined in the module. Files containing tests should be named according to the rubric test_<class_name>.py. Next, if needed for more specific tests define a class. Then subclass it. It avoids introducing the category of bugs associated with copying and pasting code for reuse. This class should be named similarly to the file name. There's a category of bugs that appear if the initialization parameters defined at the top of the test file are directly used: some tests require the initialization parameters to be changed slightly. Its possible to define a parameter and have it change in memory as a result of a test. Subsequent tests will therefore throw errors. Each class contains methods that implement a test. These methods are named according to the rubric test_<name>_<condition>

Unit tests

In those tests each functionality of each function in the module is tested, it is the entry point when adding new functions. It assures that the function behaves correctly independently of others. Each test has to be fast, so use of heavier libraries is discouraged. The input used is the minimal one needed to obtain a reproducible output. Dependencies should be replaced with mocks as needed.

Integration Tests

Here the functions' behaviour with other modules is tested. In this repository each integration function is in the correspondent folder. Then the integration with a single class of the target, or the minimum amount of classes for a functionality, is tested in each separated file.

System tests

Those tests will use every component of the system, comprising multiple classes. Those tests include expected use cases and unexpected or stress tests.

Adding new functionalities and tests required

When adding new functions to an existing module, add unit tests in the correspondent file in the same order of the module, if a new module is created a new file should be created. When a bug is solved created the respective regression test to ensure that it will not return. Proceed similarly with integration tests. Creating a new file if a functionality completely different from the others is created e.g. support for a new layer. System tests are added if support for a new library is added.

Updating tests

If new functionalities are changed or removed the tests are expected to reflect that, generally the ordering is unit tests -> integration tests-> system tests. Also, unit tests that change the dependencies should be checked, since this system is fairly small the internal dependencies are not always mocked.

references: https://jrsmith3.github.io/python-testing-style-guidelines.html

elastic-ai.creator's Issues

pytorch api for input_domain.py

The input_domain module heavily relies on python standard lib and numpy, since we seem to mostly deal with tensors anyway we should try to stick to the pytorch api as much as possible.

Can we just set the random seeds instead of explicitly setting weights in tests?

couldn't we just set the random seeds accordingly?

Originally posted by @glencoe in #13 (comment)

build and train one lstm cell in pytorch/qtorch

convert lstm cell to vhdl

convert mac_async to it's own code generator class

Use \t instead of spaces for the vhdl strings

should be consistent with spaces or \t, also in the following tests + remove for loop in tests for the lines

Originally posted by @Annika10 in #13 (comment)

get rid of QConv1d and co.

With pytorch 3.10 the concept of parametrization was introduced... Now pytorch provides a native way for us to inject and apply arbitrary modules to other modules parameters before they are used... E.g. A module behaving equivalently to QConv1d with Binarize can be built like this

import torch
from torch.nn import Conv1d
from torch.nn.utils.parametrize import register_parametrization
from elasticai.creator.layers import Binarize

layer = Conv1d(in_channels=2,
               out_channels=3,
               kernel_size=(1,),
               bias=False)

register_parametrization(layer, "weight", Binarize())

Therefore neither the implementations of our quantizable convolutions nor of our qlstm cells should [not] be needed anymore.
If we decide to still keep the implementations we should implement them with the help of parametrization.

Also see: https://pytorch.org/tutorials/intermediate/parametrizations.html

Refactor code and create a demo for a single working example

precomputable as decorator for classes

currently the precomputable function in elasticai.creator.precomputation takes an instance of a module... This allows us to modify instances of classes to attach to them the information we need to perform precomputations. However i guess it would be easier to read if we tagged classes as precomputable like so:

@precomputable(items=[-1, 1], input_generator=my_input_generator)
class MyCustomModule(torch.nn.Module):
  ...

Tests fail in CI/CD Pipeline due to IO access

see https://github.com/es-ude/elastic-ai.creator/runs/4764855994?check_suite_focus=true#step:10:147

Test Coverage Tool

Look for a test coverage tool for our repo

use the ghdl syntax checker

ghdl has a syntax checker which is useful for us for files that do not have a testbench. Check which command is needed for this and add it in the readme. Chao already mentioned elaborate could be the command for this

convert rom_generator to it's own code generator class

generate lstm testbench file

Express how inputs and output tables of consecutive layers are connected

The next thing we need is a way to express which outputs comin from layer l_i flow into which inputs of layer l_(i+1) in the context of precomputed convolutions

Originally posted by @glencoe in #14 (comment)

Example Case 8 or 16bit LSTM

Provide a simple example case for an LSTM with fixed point weights of 8 or 16 bit. This should allow us to get the fixed point weights from the model for verifying the hardware implementation in progress. Maybe we can use easily accessible training data from torchaudio? In a later step we could generate the corresponding testbench from that LSTM example already, but that's for a future issue.

Serialize Precomputations to JSON

Currently Precomputation has a dump_to(f) method. To encode as JSON and dump the result to a file-like object f. Instead we should see if there is a way to make the Precomputation JSON-serializable instead, so the following becomes possible:

with open("precomputations.json", "w") as f:
  for precomputation in precomputations:
    json.dump(precomputation)

or even

with open("precomputations.json", "w") as f:
  json.dump(precomputations)

Use CodeGenerators for LSTMs

#13 (comment)

fix the poetry installation guide in the readme.md

py running the command

poetry install 

ONNX and Brevitas are not getting installed.
make the poetry command maybe like the one in workflow/unit_tests.yml ?

Test if inputted x values of the precomputed function are synthesizable

We want to check if the list of x values, we put into a precomputed function is synthesizable. For this we generate the vhdl file for different lists of x values and synthesize them with Vivado

Fix resource management when using project as dependency

When importing the elastic ai creator as a dependency of a different project, resources like the static vhdl files are missing.

Refactoring after finishing code generators for mac_async and lstm

I think lots of functions in form of write_... aren't need anymore when using code generators. We should check that and remove the outdated functions

add counter when generating files for multiple lstm cells

When we have multiple lstm cells, we also have multiple weights and bias vhdl files. The idea is to add an counter at the file names or create a dictionary for each of them.

can the _prefix_use function be used for all libraries?

I'm just not sure about the prefix_use function because the lstm_cell.vhd has something like LIBRARY work; without use in before

Originally posted by @Annika10 in #31 (comment)

run doctests automatically

run the python doctests automatically before we merge back into main to make sure documentation is mostly up to date.

remove the literal line breaks from precomputed_scalar_functions

What i meant is to remove the literal line breaks '\n' and instead have the code variable be a list or iterable of strings. In the style of

code = chain(library, entity)
process = Process(...)
code = chain(code, process)
# and so on
return code

Originally posted by @glencoe in #31 (comment)

Generating BRAMs to store weights and bias for lstm cell

Storing weight and bias on FPGA with BRAM is quite usual. And I want a feature from our creator tool that can generate a rom_name.vhd file by sending a floating array to it. Sure I also will specify the bit with for a fixed point and the bit with for its fractional part.

I made a Jupyter-notebook and an HDL template. In the template, all arguments that can be modified are with a prefix PARAM_, e.g, PARAM_ROM_NAME, PARAM_ADDR_WIDTH, PARAM_DATA_WITH. The Jupyter-notebook also gives an example of how these arguments are defined.

Use Code Generators for Testbench vhdl creation

started with this in pull request: #51

Implement precomputed scalar generation completly with code generators

The idea is to refactor the __call__ function for the precomputed scalar class to only use code generators

Get rid of indentation in code generators

We're gonna use an automatic formatter for this so there's no need to control indentation levels of generated vhdl code ourselves. #47

Improve documentation of the VHDL CodeGenerators

I think the idea is nice but I wouldn't say I get everything in this implementation. A documentation would be nice, also about the usage

Originally posted by @Annika10 in #13 (comment)

LSTMCell parameters effectively hard coded

elastic-ai.creator/elasticai/creator/vhdl/generator/generator_functions.py

Line 244 in 28ca7d8

_ensure_reproducibility()

and

elastic-ai.creator/elasticai/creator/vhdl/generator/generator_functions.py

Line 245 in 28ca7d8

lstm_single_cell = _elastic_ai_creator_lstm()

cannot be correct. This seems like we are always generating the exact same LSTM Cell every time.

To fix this create two different LSTM Cells in tests and make the tests pass.
Additionally the module that generates the vhdl implementation should not depend directly on the QLSTM cell.

get rid of the file creation in the other tests

before merging back we might want to get rid of the file creation in the other tests as well.

Originally posted by @glencoe in #13 (comment)

Check if simple templates can be used for code generation

There are still quite some vhdl components that are built from scratch by using a representation of the formal grammar of vhdl. That grammar (in elasticai.creator.vhdl.language) was originally started to generate the vhdl implementations for precomputed components like the Sigmoid and Tanh functions, since more than 80% of the contents for these files need to be generated.
However for many other components we can just use a template and fill in a few fields, e.g., just the field for the bitwidths and entity_name. See https://github.com/es-ude/elastic-ai.creator/pull/83/files#diff-0365f931810180507db0470eda0743afec6363dc6a4fc8f95f58b5793ac177f6 for an example.

An additional advantage of templates is that they are easier to understand for hardware designers.

The following code generators should be checked as well to see if using simple templates makes more sense for them. For each of them, check whether or not to use a template. In case we should rather use a template, create a new issue for changing that code generator.

• elastic-ai.creator/elasticai/creator/vhdl/generator/rom.py

  Line 17 in c9d97fa

  class Rom:

• elastic-ai.creator/elasticai/creator/vhdl/generator/precomputed_scalar_function.py

  Line 142 in c9d97fa

  class PrecomputedScalarTestBench:

use and update numerical represenation module

merge tanh and sigmoid process

For now sigmoid and tanh generation has lots of duplication's which can be merged and outsourced in a general precomputed scalar function

convert the current LSTM definition to it's own code generator class

Get rid of lines only containing white spaces

guess we should get rid of the line only containing white spaces in the future

Originally posted by @glencoe in #31 (comment)

Shape Inference Warnings for ONNX exports

Currently we're producing the following warning

The shape inference of elasticai.creator::Wrapper type is missing, so it may result in wrong shape inference for the exported graph. Please consider adding it in symbolic function.

when exporting to onnx. I do not know atm how to add the shape inference to the symbolic function, but i think we should really do it.

automatic update pypi by new release

The package on pypi should be updated when a new release is released. You need a key from Lukas for this

Check if there exist automatic formatter for vhdl

The idea is to not indent our generated vhdl code by ourselves, but just put the code into an automatic formatter and this indents the code correctly. Is there such a formatter?

ghdl syntax check as a python integration test

Use VHDL Formatter for indentation of generated files

see #47
note: this should be solved in very naive, straight-forward way, e.g. calling the tool from the main module, or even calling it via the operating system, e.g. subprocess.

The intended behaviour is to change the exisiting file not create a new one

change folder structure for vhd files

put vhd files in highest directory, add static files and copy them, when generating files, give user the possibility to define where generated files should be located

batch_first parameter of the QLSTM layer not working as expected

When setting the batch_first parameter to True it seems to have no effect. It needs to be checked.

IOTableBuilder

Currently the created IOTables exactly match the data that was passed to the corresponding pytorch module. E.g.:

>> class Simple(torch.nn.Module):
>>   def forward(x):
>>     return x*torch.tensor([2, 1])
>>
>> x = torch.tensor([[1, 1], [0, 0]])
>> p = Precomputation(module, x)
>> p()
>> print(p.output.tolist())
[[2, 1], [0, 0]]

The Precomputation class does not need any information about how inputs and outputs are connected with each other inside the module. However for creating the final IOTable, especially on hardware it makes a massive difference whether we can express the above operation as two tables

input0	input1	output0
1	1	2

input0	input1	output1
1	1	1

or as two tables

input0	output0
1	2

input1	output1
1	1

I don't think we should add functionality to the precomputation class. My current plan is to add write a function or class that is responsible for formatting or splitting a bigger table into smaller ones according to how inputs and outputs are connected. In the old code base we already had code to format the tables according to an implicit contract. However we decide to solve this, the rules that we use to format the tables should be as explicit and obvious as possible.

Use CodeGenerators for MacAsync

#13 (comment)

Basic Fixed Point quantization functions

As an AI designer i want to be able to train my network with a specific fixed point number representation, so the corresponding representation can then be used on hardware.

Comment: I'd propose to implement quantization functions similar to the fake quant in pytorch with a straight-through estimator. But we should also talk about how related publications, e.g. FINN+Brevitas, are handling this. However from what i remember the rough idea is mostly the same.

Merge "Developers Guide" with the `Developers introductory Guide and Glossary`

Maybe change the name of this or merge it with the Developers introductory Guide and Glossary because it is named very similar. Or when not merged than I would propose to order it after each other in the readme

Originally posted by @Annika10 in #13 (comment)

update readme

we have some part in the readme which refers to the gitlab mtl translator. Can this be discarded by having explained how to add our project as a dependency with poetry? @glencoe