azure-samples / cognitive-services-voice-assistant

Welcome to the Microsoft Voice Assistant samples repository! Here you will find samples to help you get started building client application for your bot or Custom Command service. You will also be able to easily deploy a working Custom Command based Voice Assistant to your own Azure subscription

page_type	description	languages	products
sample	Sample code for Microsoft Cognitive Services Voice Assistant	csharp c++	dotnet azure-bot-service azure-cognitive-services azure-iot-edge azure-language-understanding azure-speech-text azure-text-speech windows windows-iot windows-wpf windows-uwp Cognitive Services

Microsoft Cognitive Services - Voice Assistant Sample Code

Overview

This repository includes everything needed to build a sample Voice Assistant hosted in your own Azure subscription, with a client application of your choice. Voice Assistants use Microsoft's Speech SDK in the client application to connect to Direct Line Speech Channel and your Bot-Framework bot. Voice Assistants can be keyword activated using keyword model files in this repository or new ones you create.

If you are writing a Bot-Framework bot, read the tutorial Voice-enable your bot using the Speech SDK, and use sample clients in this repo to help you get started.

Or just browse individual components per the repository content below.

Repository Content

Sample Client Applications

Folder	Description	Language/Platform
clients	Landing page for sample clients
clients\csharp-uwp	UWP voice assistant client sample. An example UWP app that uses Window's new voice activation APIs, together with the Voice Assistant platform, to connect to your bot or Custom Commands application.	C#, Universal Windows Platform (UWP)
clients\csharp-wpf	Windows voice assistant client sample. Generic Windows tool to manually test your bot or Custom Commands application	C#, Windows Presentation Foundation (WPF)
clients\cpp-console	Console client sample for Smart Speaker devices, configured via a json file to connect to your bot or Custom Commands application. It supports microphone input and audio playback. Build scripts for Raspberry Pi, GGEC Speaker and Windows are provided	C++, Windows, Linux
clients\java-android	An Android application (phone or tablet) configured to connect to your bot or Custom Commands application	Java Android

Sample Keyword Models

Folder	Description
keyword-models	A few keyword models (.table files) for you to try out. Or you can create your own -- See Create a custom keyword using Speech Studio

Test Code

Folder	Description	Language/Platform
clients\csharp-dotnet-core\voice-assistant-test	Automated, multi-turn, end-to-end regression test for your bot or Custom Commands application. Supports WAV file input, text or Bot-Framework activities	C# .NET Core

Articles

File	Description
docs\CloudConnectionLogic.md	Discusses options for when to connect to Direct Line Speech channel and how long to keep the connection open
docs\LocalBot.md	Running & debugging your Bot locally while using Direct Line Speech Channel

Root Folder Files

File	Description	Language/Platform
.gitignore	Define what to ignore at commit time
CODE_OF_CONDUCT.md	Code of Conduct for all Microsoft repositories
CONTRIBUTING.md	Guidelines for contributing to these samples
README.md	This README file
LICENSE.md	The license for these samples
SECURITY.md	Information about reporting any security vulnerabilities to Microsoft
NOTICE.txt	License of third party software incorporated in these samples

Clone the Repo

To build any of the sample client applications or deploy custom commands to your Azure subscription, you will first need to clone this GitHub repository:

    git clone https://github.com/Azure-Samples/Cognitive-Services-Voice-Assistant.git
    cd Cognitive-Services-Voice-Assistant

Related Microsoft GitHub Repos

• https://github.com/Azure-Samples/cognitive-services-speech-sdk
• https://github.com/Azure-Samples/Cognitive-Services-Speech-Devices-SDK
• https://github.com/microsoft/botframework-solutions
• https://github.com/Microsoft/BotBuilder-Samples

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.opensource.microsoft.com.

When you submit a pull request, a CLA bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., status check, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

Reporting Security Issues

Security issues and bugs should be reported privately, via email, to the Microsoft Security Response Center (MSRC) at [email protected]. You should receive a response within 24 hours. If for some reason you do not, please follow up via email to ensure we received your original message. Further information, including the MSRC PGP key, can be found in the Security TechCenter.

Copyright (c) Microsoft Corporation. All rights reserved.

cognitive-services-voice-assistant's Issues

Fix documentation around running the sample

This issue is for a: (mark with an x)

- [ ] bug report -> please search issues before submitting
- [ ] feature request
- [ x] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Replace the text in the configs/config.json file with your subscription key and key region. If you are using a custom speech commands application or custom speech font you can insert those GUID's there as well.
** Custom Commands is the official name
** Also call out the table file

cd into the scripts directory

run ./buildArm32.sh
** Did not work, had to do sh or could have done ./run
** It should also be indented to make it formatted as a code block

Running the sample

** It was not obvious that sample.exe was created and in the out folder.

** My first attempt **

ubuntu@ubuntu:/cpp-console/out$ export LD_LIBRARY_PATH="../lib/arm32"
ubuntu@ubuntu:/cpp-console/out$ sample.exe ../configs/config.json
sample.exe: command not found
** the command in the instructions didn’t work, it needed to be ./sample.exe

Support bot framework 'inputhint'

• bug report -> please search issues before submitting
• feature request
• documentation issue or request
• regression (a behavior that used to work and stopped in a new release)

Tag bot connection settings with friendly names, for easy selection

When you test multiple bots, it's hard to remember which key, key region (and other configurations) are associated with which bot. It would be nice to have a drop down menu where you have friendly names of several connection settings, such that you can easily switch between them.

Do not show green bubble for empty text in VoiceAssistantClient

Please provide us with the following information:

This issue is for a: (mark with an x)

- [ ] bug report -> please search issues before submitting
- [x] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

Send an activity with empty text (maybe have cards etc.)

Any log messages given by the failure

Expected/desired behavior

Do not show the green bubble

OS and Version?

Windows 7, 8 or 10. Linux (which distribution). macOS (Yosemite? El Capitan? Sierra?)

Windows 10

Versions

Today master

Mention any other details that might be useful

---

Thanks! We'll be in touch soon.

Custom Azure DevOps build agent with pre-release Windows SDK

In order to check-in the UWP Voice Assistant Sample we need a VM running pre-release Windows SDK. We will use that in our Azure build pipeline instead of the standard Microsoft agent.

Tests for KeywordRegistration.cs

Tests for DialogManager.cs

Automatic creation of LUIS and Speech Azure resources

This issue is for a: (mark with an x)

- [ ] bug report -> please search issues before submitting
- [x ] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

click on the new button in the readme and follow the prompts.

Any log messages given by the failure

Expected/desired behavior

azure resources should be created

OS and Version?

Anywhere powershell 6 or greater can run

We need a simple mostly automatic way to create azure resources

• speech resource
• LUIS resource
• LUIS authoring key

One-level at a time expand and shrink JSON in Activity Payload

The expand and shrink JSON in Activity Payload should do expanding/shrinking one level at a time, not all the way.

Make it clear Raspbian is not supported in Docs

This issue is for a: (mark with an x)

- [ ] bug report -> please search issues before submitting
- [ ] feature request
- [x ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Raspbian is not supported and we should be explicit in the docs.

Add "About" box 

With MS copyright, app version, link to GitHub readme.md, link to GitHub issues, link to GitHub releases to check new version,... (see example in Bot-Framework Emulator)

Ramp up on Custom Commands - Darren

Tests for AgentAudioProducer

Add a wiki page to explain how to run & debug your bot locally while using Direct Line Speech Client

This is a common case: you would like to develop your bot while running it locally on your PC. You would like to use Direct Line (DL) Speech Client and Direct Line channel while doing so, as if it was a real deployment. A wiki is needed to show how you do this.

Add drops of executable in the Release tab

Populate the Release tab with builds of major versions so users do not need to build from source code if they just want to run the tool

Halt TTS playback on barge-in

• bug report -> please search issues before submitting
• feature request
• documentation issue or request
• regression (a behavior that used to work and stopped in a new release)

Create a script to automatically create and deploy a custom commands application

This issue is for a: (mark with an x)

- [ ] bug report -> please search issues before submitting
- [ x] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

run the powershell script and see that a custom commands app was created from the json file. at speech.microsoft.com

Expected/desired behavior

this will create and deploy a custom commands resource from the json file specified in the script.

OS and Version?

Anywhere Powershell 6 or greater can be run.

Ramp up on Custom Commands - Fan

Deploy and test the three Custom Commands demos, using Windwos Voice Assistant Client

Document manual tests for Voice Assistant Devices

We need a set of manual tests that we can run to make sure all our Voice Assistant clients behave the same way and work well with voice input/ouput

publish the Windows console executable as an "Artifact" of the pipeline

This issue is for a: (mark with an x)

- [ ] bug report -> please search issues before submitting
- [x ] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

publish the Windows console executable as an "Artifact" of the ADO build pipeline. Not urgent. See for example the "Artifacts" button here: https://msasg.visualstudio.com/Skyman/_build/results?buildId=10643429

Add web resources to azure template

This issue is for a: (mark with an x)

- [ ] bug report -> please search issues before submitting
- [ x] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

Click on the deploy button in a README

Expected/desired behavior

web resources required by a specific demo should be deployed

OS and Version?

Windows 7, 8 or 10. Linux (which distribution). macOS (Yosemite? El Capitan? Sierra?)

The specific resources needed are still being decided, but they should allow the hospitality demo task completion to be shown on a web page.

Update Jot NuGet to version 2.# (breaking change)

Jot NuGet package needs to be updated to version 2.#. It's a breaking change, so some code changes will be needed.

Missing "clear" (or "delete") button to handle Custom Activities

You can create a new Activity, Edit an existing one, but you can't delete an existing one. Missing "Clear" or "Delete" button at the bottom of the main Window.

Add settings page for configuration values

Add check for KWS file and print proper error

This issue is for a: (mark with an x)

- [ ] bug report -> please search issues before submitting
- [ x] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

Currently if the KWS table doesn't exist you get a meaningless message from the SDK. We can do better

Any log messages given by the failure

Expected/desired behavior

Print out if the file doesn't exist

OS and Version?

Windows 7, 8 or 10. Linux (which distribution). macOS (Yosemite? El Capitan? Sierra?)

Add support for barge in on c++ audio player

This issue is for a: (mark with an x)

- [ ] bug report -> please search issues before submitting
- [ x] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

The audio players in the c++ sample don't currently support stopping audio playback.

We should stop playing and clear the queue on a Stop() call on the player.

Web client sample

This issue is for a: (mark with an x)

- [ ] bug report -> please search issues before submitting
- [ ] feature request
- [x] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Any plans on releasing a web client sample so also mac users (who don't want to deploy Java) can use it?

Update sample code to show activity templates

This issue is for a: (mark with an x)

- [ ] bug report -> please search issues before submitting
- [ x] feature request
- [x ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

The Speech SDK now has the ability to use activity templates to "stamp" activities sent to the bot. We should add this to our samples.

Add link on how to install ubuntu in docs

This issue is for a: (mark with an x)

- [ ] bug report -> please search issues before submitting
- [ ] feature request
- [ x] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

For Ubuntu, include this link as it steps through getting Ubuntu working including networking.
https://ubuntu.com/tutorials/how-to-install-ubuntu-on-your-raspberry-pi

Add a disconnect button

Have a button to disconnect from Bot (so you can release files and redeploy).

Support for Prompt Calls

Please provide us with the following information:

This issue is for a: (mark with an x)

- [X] bug report -> please search issues before submitting
- [ ] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

return await stepContext.PromptAsync(nameof(ChoicePrompt),
new PromptOptions
{
Prompt = stepContext.Context.Activity.CreateReply(message),
Choices = new[]
{
new Choice {Value = "Yes", Synonyms = new List {"yeah","ok"}},
new Choice {Value = "No", Synonyms = new List {"never"}}
}.ToList(),
Style = ListStyle.HeroCard
});

Any log messages given by the failure

No.
The Direct Line speech client app just don't display it.
However, the application responds to user inputs regarding the prompt question like 'Yes'

Expected/desired behavior

To see the prompt message e.g. https://binged.it/2FlaHsh

OS and Version?

Windows 10 + Botframework

Versions

Windows: version 1909
Botframework: V4.7.0

Mention any other details that might be useful

---

Thanks! We'll be in touch soon.

Update Android Java client configuration settings

Update to support

• Custom command application
• Custom SR end point
• Voice Deployment ID
  And others as needed for our hero demos

Hero card is not displayed

Please provide us with the following information:

This issue is for a: (mark with an x)

- [ ] bug report -> please search issues before submitting
- [x] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

Send a herocard.

Any log messages given by the failure

No log. Just don't display it.

Expected/desired behavior

Display it correctly.

OS and Version?

Windows 7, 8 or 10. Linux (which distribution). macOS (Yosemite? El Capitan? Sierra?)
Windows 10

Versions

Latest master.

Mention any other details that might be useful

---

Thanks! We'll be in touch soon.

Fix app icon

Needs the have the right color and background to match the app

Tests for MVARegistration.cs

Add a wiki page on how to sniff network traffic

For folks who want to monitor network traffic. Use Fiddler software.

Add script and wget to grab SDK

This issue is for a: (mark with an x)

- [ ] bug report -> please search issues before submitting
- [x ] feature request
- [x ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

This command will get the SDK
wget -c https://aka.ms/csspeech/linuxbinary -O - | tar -xz

We should include that in our build script and set it up for the user.
Documentation should also reflect this.

OAuth card cannot show up in

Please provide us with the following information:

This issue is for a: (mark with an x)

- [x] bug report -> please search issues before submitting
- [ ] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

connect a bot which have an OAuth card, then check the card message

Any log messages given by the failure

the client received the card but didn't render it

Expected/desired behavior

OAuth card shows well

OS and Version?

Windows 7, 8 or 10. Linux (which distribution). macOS (Yosemite? El Capitan? Sierra?)

Windows 10

Versions

latest master build

Mention any other details that might be useful

---

Thanks! We'll be in touch soon.

Ramp up on Custom Commands - Chris

Need to ramp up on Custom Commands. Will specifically need to learn how to export an existing app.

Add Custom Commands feature

• Custom Commands Text Box
• Bot ID Text Box

Action.Submit is not supported in VoiceAssistantClient

Please provide us with the following information:

This issue is for a: (mark with an x)

- [x] bug report -> please search issues before submitting
- [ ] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

send an adaptive card with Action.Submit and click https://adaptivecards.io/explorer/Action.Submit.html

Any log messages given by the failure

It pops up a window

Expected/desired behavior

send back to bot

OS and Version?

Windows 7, 8 or 10. Linux (which distribution). macOS (Yosemite? El Capitan? Sierra?)

Windows 10

Versions

Today's master

Mention any other details that might be useful

---

Thanks! We'll be in touch soon.

Call out we have only tested ARM 32 on pi 3 and pi 4 not other platforms

This issue is for a: (mark with an x)

- [ ] bug report -> please search issues before submitting
- [ ] feature request
- [x ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Add documentation around commands

This issue is for a: (mark with an x)

- [ ] bug report -> please search issues before submitting
- [ ] feature request
- [ x] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

There is default behavior that starts KWS so we should document that. Also there are commands that print out if you have a console interface which show examples of how to use the basic API commands

Button for cancel listening

At the moment there is no way to cancel listening once started. You have to wait until the speech recognizer times out.

Automatically scroll down the "Conversation View" to see latest user & bot live text

For a long conversation, while it's on going, the "Conversation View" window should show the latest user & bot text. As of now you have to manually use the scroll bar to see the latest live text.

Document Manual Tests performed for Java App

Add contributions to https://github.com/Azure-Samples/Cognitive-Services-Voice-Assistant/wiki/Manual-Test-Pass for validating device/app performance.

Add link to PUTTY in docs

Many devs use putty so we should link to it in the docs.

Configure ADO Build Agent for UWP Voice Assistant

Add configuration file feature to UWP Voice Assistant