cd <workspace>
#conda (optional): conda create -n intel_dns Python=3.10.6 && conda activate intel_dns
#venv (optional): python -m venv venv_intel_dns && source venv_intel_dns/bin/activate
wget https://github.com/lava-nc/lava/releases/download/v0.7.0/lava_nc-0.7.0.tar.gz
pip install lava_nc-0.7.0.tar.gz
rm lava_nc-0.7.0.tar.gz
wget https://github.com/lava-nc/lava-dl/releases/download/v0.3.3/lava_dl-0.3.3.tar.gz
pip install lava_dl-0.3.3.tar.gz
rm lava_dl-0.3.3.tar.gz
git clone [email protected]:thebarnable/IntelNeuromorphicDNSChallenge.git
cd IntelNeuromorphicDNSChallenge
rm -r microsoft_dns
git clone microsoft/DNS-Challenge.git microsoft_dns
cd microsoft_dns && git checkout 5582dcf && cd ..
pip install -r requirements.txt
pip install tensorboard
python -c "import os; from distutils.sysconfig import get_python_lib; open(get_python_lib() + os.sep + 'ndns.pth', 'a').write(os.getcwd())"
# start training to check if everything works
python baseline_solution/sdnn_delays/train_sdnn.py -path /mnt/data4tb/stadtmann/dns_challenge_4/datasets_fullband/
The Intel Neuromorphic Deep Noise Suppression Challenge (Intel N-DNS Challenge) is a contest to help neuromorphic and machine learning researchers create high-quality and low-power real-time audio denoising systems. The Intel N-DNS challenge is inspired by the Microsoft DNS Challenge, and it re-uses the Microsoft DNS Challenge noisy and clean speech datasets. This repository contains the challenge information, code, and documentation to get started with Intel N-DNS Challenge.
A solution to the Intel N-DNS Challenge consists of an audio encoder, a neuromorphic denoiser, and an audio decoder. Noisy speech is input to the encoder, which converts the audio waveform into a form suitable for processing in the neuromorphic denoiser. The neuromorphic denoiser takes this input and removes noise from the signal. Finally, the decoder converts the output of the neuromorphic denoiser into a clean output audio waveform. The Intel N-DNS Challenge consists of two tracks:
Track 1 (Algorithmic) aims to encourage algorithmic innovation that leads to a higher denoising performance while being efficient when implemented as a neuromorphic system. The encoder, decoder, and neuromorphic denoiser all run on CPU.
Track 2 (Loihi 2) aims to realize the algorithmic innovation in Track 1 on actual neuromorphic hardware and demonstrate a real-time denoising system. The encoder and decoder run on CPU and the neuromorphic denoiser runs on Loihi 2.
Solutions submitted to the Intel N-DNS challenge are evaluated in terms of an audio quality metric (denoising task performance) and computational resource usage metrics, which measure the efficiency of the solution as a system; submissions also include source code and a short write-up. Solutions will be holistically considered (metrics, write-up, innovativeness, commercial relevance, etc.) by an Intel committee for a monetary prize (details below).
Please see our arXiv paper for a more detailed overview of the challenge.
- How to participate?
- Install instructions
- Dataset
- Dataloader
- Baseline Solution
- Evaluation Metrics
- Metricsboard
- Challenge Rules
Follow the registration instructions below to participate. The overview of the challenge timeline is shown below.
gantt
Title Neuromorphic DNS Challenge
dateFormat MM-DD-YYYY
axisFormat %m-%d-%Y
Challenge start :milestone, s0, 03-16-2023, 0d
section Track 1
Track 1 solution development :active, t0, after s0, 138d
Test Set 1 release :milestone, after t0
Track 1 submission :t2, after t0, 30d
Model freeze :crit, t2, after t0, 30d
Track 1 evaluation :t3, after t2, 15d
Track 1 winner announcement :crit, milestone, aftet t3
section Track 2
Track 2 solution development :tt0, after t0, 182d
Test Set 2 release :milestone, after tt0
Track 2 submission :tt2, after tt0, 30d
Model freeze :crit, tt2, after tt0, 30d
Track 2 evaluation :tt3, after tt2, 15d
Challenge winner announcement :crit, milestone, aftet t3
Phase | Date |
---|---|
Challenge start | Mar 16, 2023 |
Test set 1 release | On or about July 28, 2023 |
Track 1 submission deadline | On or about Aug 28, 2023 |
Track 1 winner announcement | Sep 14, 2023 |
Test set 2 release | On or about Jan 28, 2024 |
Track 2 submission deadline | On or about Feb 28, 2024 |
Track 2 winner announcement | Mar 14, 2024 |
Challenge dates are subject to change. Registered participants shall be notified of any changes in the dates or fixation of on or about dates.
- Create your challenge github repo (public or private) and provide access to
lava-nc-user
user. - Register for the challenge here.
- You will receive a registration confirmation.
Once registered, you will receive updates about different phases of the challenges.
Participation for Track 2 will need Loihi system cloud access which needs an Intel Neuromorphic Research Collaboration agreement. Please see Join the INRC or drop an email to [email protected]. This process might take a while, so it is recommended to initiate this process as early as possible if you want to participate in Track 2.
Once the test set 1 for Track 1 is released, we will enter Track 1 model freeze phase. The details on test set 1 will be updated later.
- Participants shall not change their model during this phase.
- Participants shall evaluate their model on test set 1, measure all the necessary metrics on an Intel Core i5 quad-core machine clocked at 2.4 GHz or weaker, and submit their metrics along with a solution writeup.
Important: At least one valid metric board entry must have been submitted before Track 1 model freeze phase. Metricboard entries will be randomly verified.
A committee of Intel employees will evaluate the Track 1 solutions to decide the winners, making a holistic evaluation including audio quality, computational resource usage, solution write-up quality, innovativeness, and commercial relevance.
Important: Intel reserves the right to consider and evaluate submissions at its discretion. Implementation and management of this challenge and associated prizes are subject to change at any time without notice to contest participants or winners and is at the complete discretion of Intel.
Once the test set 2 for Track 2 is released, we will enter Track 2 model freeze phase. The details on test set 2 will be updated later.
- Participants shall not change their model during this phase.
- Participants shall evaluate their model on test set 2, measure all the necessary metrics on Loihi, and submit their metrics along with a solution writeup.
Important: At least one valid metric board entry must have been submitted before Track 2 model freeze phase. Metricboard entries will be randomly verified.
A committee of Intel employees will evaluate the Track 2 solutions to decide the winners, making a holistic evaluation including audio quality, computational resource usage, solution write-up quality, innovativeness, and commercial relevance.
Important: Intel reserves the right to consider and evaluate submissions at its discretion. Implementation and management of this challenge and associated prizes are subject to change at any time without notice to contest participants or winners and is at the complete discretion of Intel.
There will be two prizes awarded
- Track 1 winner: fifteen thousand dollars (US $15,000.00) or the equivalent in grant money to the winner of Track 1
and six months later,
- Track 2 winner: forty thousand dollars (US $40,000.00) or the equivalent in grant money to the winner of Track 2.
These awards will be made based on the judging of the Intel committee. Where the winner is a resident from one of the named countries in the Intel N-DNS Challenge Rules and not a government employee, Intel can directly award the prize money to the winner. Where the winner is a government employee to which Intel can administer academic grant funding (regardless of whether the winner resides in one of the named countries in the Intel N-DNS Challenge Rules); a research grant in the amount for the appropriate track will be awarded to the university where the researcher/government employee is from, and in the researcher's name. Where the winner does not fall into the above categories, Intel will publicly recognize the winner, but the winner is not eligible to receive a prize. Limit of one prize per submission.
Important:
- Researchers affiliated with universities worldwide, not restricted to the countries listed in the N-DNS Challenge Rules, are also eligible to receive prizes that Intel will administer. This includes, but is not limited to, government employees such as professors, research associates, postdoctoral research fellows, and research scientists employed by a state-funded university or research institution. Where possible, Intel will provide unrestricted gift funding to the awardee's department or group. However, universities in countries under U.S. embargo are not eligible to receive award funding.
- Other individuals that do not fall into the above categories, but wish to enter this Contest, may do so. However, they are not eligible for any prize, but will be publicly recognized if they win. See Prizes under N-DNS Challenge Rules for further details.
- For avoidance of doubt, Intel has the sole discretion to determine the category of the entries to the N-DNS Award contest.
We also ask that challenge participants submit a short (one or two page) write-up that explains the thought process that went into developing their solution. Please include what worked, what did not work, and why certain strategies were chosen versus others. While audio quality and power are key metrics for evaluating solutions, the overarching goal of this challenge is to drive neuromorphic algorithm innovation, and challenge participant learnings are extremely valuable.
This write-up can be submitted directly to Intel to maintain privacy before the track deadline, but for the write-up to be considered in the holistic evaluation of the solution for the monetary prize, we require that it be shared publicly within 14 days after the deadline for each track. Naturally, however, we encourage participants to share their write-ups publicly at any time, to help inspire others' solutions.
Additionally, we plan to invite a select group of challenge participants to present their solutions at a future Intel Neuromorphic Research Community (INRC) forum, based on their algorithmic innovation and metricsboard results as judged by the Intel committee, to share their learnings and participate in a discussion on developing new and improved neuromorphic computing challenges.
For ease of comparison, we ask that your solution write-up include a clear table with the evaluation metrics for your solution akin to the Table in the Metricsboard.
For your writeup, please use a single-column Word document or Latex template with 1-inch margins, single-spacing, reasonable font size (11pt or 12pt; default font like Times New Roman), and up to two US letter-size or A4 pages. Please submit a PDF.
Challenge participants must provide the source code used in the creation of their solution (model definition, final trained model, training scripts, inference scripts, etc.) with MIT or BSD3 license.
Challenge participant source code for Track 1 will be publicly released after the Track 1 winner is announced. Likewise for Track 2.
pip install -r requirements.txt
python -c "import os; from distutils.sysconfig import get_python_lib; open(get_python_lib() + os.sep + 'ndns.pth', 'a').write(os.getcwd())"
python -c "import os; from distutils.sysconfig import get_python_lib; pth = get_python_lib() + os.sep + 'ndns.pth'; os.remove(pth) if os.path.exists(pth) else None;"
- Edit
microsoft_dns/download-dns-challenge-4.sh
to point the desired download location and downloader bash microsoft_dns/download-dns-challenge-4.sh
- Extract all the
*.tar.bz2
files.
- Download SHA2 checksums and extract it.
- Run the following to verify dataset validity.
import pandas as pd import hashlib def sha1_hash(file_name: str) -> str: file_hash = hashlib.sha1() with open(file_name, 'rb') as f: fb = f.read() file_hash.update(fb) return file_hash.hexdigest() sha1sums = pd.read_csv("dns4-datasets-files-sha1.csv.bz2", names=["size", "sha1", "path"]) file_not_found = [] for idx in range(len(sha1sums)): try: if sha1_hash(sha1sums['path'][idx]) != sha1sums['sha1'][idx]: print(sha1sums['path'][idx], 'is corrupted') except FileNotFoundError as e: file_not_found.append(sha1sums['path'][idx]) # 336494 files with open('missing.log', 'wt') as f: f.write('\n'.join(file_not_found))
- Training dataset:
python noisyspeech_synthesizer.cfg -root ./
- Validation dataset:
python noisyspeech_synthesizer.cfg -root ./ -is_validation_set true
- Testing data with similar statistics as the validation dataset generated from the script above will be made available towards the end of each track. We will initiate a model freeze before the release, meaning the participants will not be able to change their trained model after that.
from audio_dataloader import DNSAudio
train_set = DNSAudio(root=<your dataset folder> + 'training_set/')
validation_set = DNSAudio(root=<your dataset folder> + 'validation_set/')
The baseline solution is described in the Intel N-DNS Challenge paper.
The code for training and running the baseline solution can be found in this directory: baseline_solution/sdnn_delays
.
The training script baseline_solution/sdnn_delays/train_sdnn.py
is run as follows:
python train_sdnn.py # + optional arguments
The N-DNS solution will be evaluated based on multiple different metrics.
- SI-SNR of the solution
- SI-SNRi of the solution (improvement against both noisy data and encode+decode processing).
- DNSMOS quality of the solution (overall, signal, background)
- Latency of the solution (encode & decode latency + data buffer latency + DNS network latency)
- Power of the N-DNS network (proxy for Track 1)
- Power Delay Product (PDP) of the N-DNS solution (proxy for Track 1)
This repo provides SI-SNR module which can be used to evaluate SI-SNR and SI-SNRi metrics.
where
$s = \text{zero mean target signal}$
$\hat{s} = \text{zero mean estimate signal}$
$s_\text{target} = \displaystyle\frac{\langle\hat s, s\rangle,s}{\Vert s \Vert ^2}$
$e_\text{noise} = \hat s - s_\text{target}$
- In Code Evaluation
from snr import si_snr score = si_snr(clean, noisy)
This repo provides DNSMOS module which is wrapped from Microsoft DNS challenge. The resulting array is a DNSMOS score (overall, signal, noisy). It also supports batched evaluation.
- In Code Evaluation
from dnsmos import DNSMOS dnsmos = DNSMOS() quality = dnsmos(noisy) # It is in order [ovrl, sig, bak]
Other metrics are specific to the N-DNS solution system. For reference, a detailed walkthrough of the evaluation of the baseline solution is described in baseline_solution/sdnn_delays/evaluate_network.ipynb
.
Please refer to the Intel N-DNS Challenge paper for more details about the metrics.
The evaluation metrics for participant solutions will be listed below at regular intervals. The entries will be updated with the best solution of the team. Please create a table in markdown (.md) akin to the one below in the Github repository that you share with Intel so that we can import your metrics and update them on the public metricsboard. This will help you meaure the progress of your solution against other participating teams. Earlier submissions are encouraged.
Track 1
Entry | ||||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|---|
Team xyz (mm/dd/yyyy) | ||||||||||||
Microsoft NsNet2 (02/20/2023) | 11.89 | 4.26 | 4.26 | 2.95 | 3.27 | 3.94 | 0.024 | 20.024 | 136.13 | 2.72 | 2,681 | 10,500 |
Intel proprietary DNS (02/28/2023) | 12.71 | 5.09 | 5.09 | 3.09 | 3.35 | 4.08 | 0.036 | 8.036 | - | - | 1,901 | 3,802 |
Baseline SDNN solution (02/20/2023) | 12.50 | 4.88 | 4.88 | 2.71 | 3.21 | 3.46 | 0.036 | 8.036 | 11.59 | 0.09 | 525 | 465 |
Validation set | 7.62 | - | - | 2.45 | 3.19 | 2.72 | - | - | - | - | - | - |
Track 2
Entry | |||||||||||
---|---|---|---|---|---|---|---|---|---|---|---|
Team xyz (mm/dd/yyyy) |
Note:
- An Intel committee will determine the challenge winner using a holistic evaluation (not one particular metric). We encourage challenge participants to strive for top performance in all metrics.
- Metrics shall be taken as submitted by the participants. There will be a verification process during the contest winner evaluation.
For any additional clarifications, please refer to the challenge FAQ or Rules or ask questions in the discussions or email us at [email protected].