antmicro-labs / voice-assistant Goto Github PK

The project is meant to be run in the Zephyr OS in the LiteX/VexRiscv ecosystem on the Digilent Arty board. The board must be properly configured and the binary with Zephyr must be uploaded onto the board.

The goal is to flash the Arty board with example Zephyr binary and run it.

Responsible for this task is @szymon-filipiak.

References: Wiki page

We need to find a way to convert microphone input to data that will be fed to the TFLite model.
Conversion:
(singed 24bit PCM, stereo, little endian, 44100 Hz) => (signed 16bit PCM)

I have a question about creating a configuration for our board.

Do we need to do this?

From what I've learned this step is required to be done once, and then binary can be uploaded onto multiple devices. Or is this a bad assumption?

I wonder if You have any board configurations that we can use?

My reason for asking that is setting up whole litex toolchain with Vivado is time-consuming, not only because of large files that must be downloaded but also someone must get familiar with the tools.
If You think that this step is important for us to learn, please let me know why, because I fear that I'm missing something.
(Also, can we call for attention with @ Antmicro?)

Summon Antmicro! 🛬

I'm now trying to run our application and I'm getting confused with overlays. Do I need them?

I have been provided with modified riscv32-litex-vexriscv.dtsi file and placed it inside my zephyr sources. Do I need to change values in overlay then? @kgugala or @rw1nkler

We are close to hitting a milestone and having the micro_speech example running on Arty. Therefore I would like to ask if Antmicro will provide a driver for the microphone we were talking about at the first meeting. I am also sure that our team will have questions concerning the driver. Will some sort of documentation for the driver be available? If no, is it possible to organize a meeting to discuss our questions?

As can be seen in the TF Lite micro examples that Antmicro ported to the Zephyr OS in Litex/VexRiscv SoC environment (accelerometer_handler.h, accelerometer_handler.cc), includes are from the TF Lite modules and the Zephyr API. I would like to know if these APIs are enough to code for the Arty board in our project.

Also, I would like to ask if samples provided in the Zephyr documentation are a good source of knowledge when it comes to coding for Arty.

We need to find a clever way to use the model's classification results to make our board being a device that could be classified as some sort of assistant. Feel free to list your ideas on this issue.

To start, I suggest creating some kind of menu that could be managed by voice commands. I going to implement this feature.

It seems that sharing a copy of the repository with buildenv would be comfortable. Could you create a fork for us @antmicro?

To increase the accuracy of the device we should try to do some preprocessing on the raw microphone data.
This task includes:

• research on the preprocessing methods,
• choosing an adequate approach,
• implementing the solution,
• documentation.

We have created some custom code in TensorFlow in order to adapt micro_speech example, but for some reason, it's not running correctly. We have trouble seeing what really our software does when it's run in the simulation or on the board (we think that Zephyr is not booting successfully).

Can we maybe schedule a call where you will share with us some tips and tricks for debugging?

@kgugala @rw1nkler What do you think?

Today together with @plonajakub and @Marzecki we tried to debug our application and we approached the problem we don't know how to solve. We set up breakpoints in our code, then we tried to go step by step but it looks like it halts somewhere in zephyrproject files, debugger don't get through to our code.
We use debugger which can be found under zephyr-sdk-0.11.2/riscv64-zephyr-elf/bin/riscv64-zephyr-elf-gdb and our project can be found in micro-speech-build-fix branch.

Loading files to renode

Continue don't seem to do anything

Same here

The last log is being logged endlessly

I think that I've found an error in the LiteX Zephyr tutorial on our wiki.

Installation of xc3sprog requires static library libftdi what is not said explicitly.

Another error is in the cmake command:
cmake ..
should be:
cmake .
or
cd build cmake ..

Lastly link to the repository in the first point:

Download the xc3sprog program from this repository:

is malfunctioning.

Standalone TF example "Micro speech" compiled for the board will have no real use case. There is a need for feeding the example with audio data, taking outputs from the TF model and visualizing them on the board - for example by setting the LEDs.

The goal is to write a small app capable of driving the "Micro speech" TF model.

Investigation of this topic is delegated to @Marzecki and @FilipKadziolka .

Reference: TF Lite for microcontrollers

We have created python application which we are going to use in order to present capabilities of our voice assistant platform. We decided to not add more functionalities to board itself, instead we send instructions over UART port to PC where python script is working.
So far our python script can:

• read user unread gmail mails,
• return information about wheather,
• control music player.

Code can be found in voice-assistant repository.

I did a little research and found useful links that can help us in building well optimized TensorFlow Lite app.

• Converting TF to TFLite
• TFLite converter
• Quantization

Open-source projects feature code of conduct file, that defines rules that project contributors obey.

Please create or include one for our repository.

The good starting point is this page.

Slack notifications are our preferred way of receiving GitHub project status updates. We would like to have it connected with our repository.

Can someone from Antmicro accept the request for adding it to our project?

We are observing weird behaviour when we load a larger tf model. We get false when matching the same values. If you have some time now we are working on it. If not, can you tell us how big can stack on our board be? Our model has approx 50kb @kgugala

CONTRIBUTING file in GitHub projects defines the rules of contributing, creating pull requests, creating issues, community and behavioral expectations.

We would like to add one for consistency in our workflow, and the ability for further growth of our project.

For more information on this topic see this.

Together with @plonajakub we are going to teach our model to recognize more words and we will focus on customizing the model parameters for lower latency while losing as little accuracy as possible. Here and here you can find more information about the model. I have also created a cheat sheet containing a list of parameters we can tweak during training.

Hi,

I have started in the Python apps world recently and I have some questions that you may know the answer :)

I would like to know the best library or package to create the apk file. I saw that one popular is p4a and I have installed it; do you recommend any other way?

Also for the graphic interfaces for Python apps apart of kivi or tkinter or beware is there anything you may recommend that is better or works better with the app creation?

Thank you,

The goal is to properly set up a structure of the repository for current and future use.

This task belongs to @szymon-filipiak.

Reference: minimalistic repository

Good open-source projects have readable and extensive README.md file.

We should create one for our repository, where we will define:

• the goals of our project,
• requirements for application,
• how to install and run the application.

For reference on how to write good README files see this or this.

Please also utilize markdown language capabilities to make it readable and attractive visually.

I have added links to useful resources to the project Wiki. Please, have a look at this.

Since any custom project for the Digilent board will require proper make process there is a need for exploring the process and customize it for our use.

The goal is to have valid binary suitable to be run in the Renode environment and on the real hardware - our Digilent board with Zephyr as the target. The binary must contain the TensorFlow source.

@plonajakub and @zawadzki-dawid will investigate the subject and run through the process.

Reference:

• Page on Wiki
• Zephyr documentation about building an app

In the past week, we encountered some troubles with setting up the necessary dependencies required by Renode and Zephyr. Future expectations are to automate the testing process of our custom voice-assistant app. Therefore it will be helpful to have an integrated docker image not only capable of running simulations but also building our app and testing it.

The docker image will be based on existing renode image.

The goal is to have a docker image with an integrated work environment for our project.

Responsible for this task is @szymon-filipiak.

We would be very gratefull for reinviting Filip Kądziołka to our team. Unfortunetaly he lost his invitation and cannot join us. Thank you for help in advance.

Is it possible to create a TensorFlow submodule in this repository similarly to the demo which in turn would be a fork of the newest version of the main TensorFlow repository?