microsoft / botbuilder-realtimemediacalling

BotBuilder-RealTimeMediaCalling extends the BotBuilder to enable bots to engage in Skype audio-video calling. It provides real-time, programmable access to the voice, video, and screen sharing streams of a Skype call. The bot is a direct participant in a Skype 1:1 call.

Real-time media calling with Skype

The Real-Time Media Platform for Bots adds a new dimension to how bots can interact with users by enabling real-time voice, video and screen sharing modalities. This provides the capability to build compelling and interactive entertainment, educational, and assistance bots. Users communicate with real-time media bots using Skype.

This is an advanced capability which allows the bot to send and receive voice and video content frame by frame. The bot has "raw" access to the voice, video and screen-sharing real-time modalities. For example, as the user speaks, the bot will receive 50 audio frames per second, with each frame containing 20 milliseconds (ms) of audio. The bot can perform real-time speech recognition as the audio frames are received, rather than having to wait for a recording after the user has stopped speaking. The bot can also send high-definition-resolution video to the user at 30 frames per second, along with audio.

The Real-Time Media Platform for Bots works with the Skype Calling Cloud to take care of call setup and media session establishment, enabling the bot to engage in a voice and (optionally) video conversation with a Skype caller. The platform provides a simple "socket"-like API for the bot to send and receive media, and handles the real-time encoding and decoding of media, using codecs such as SILK for audio and H.264 for video.

Currently Real-Time Media Platform supports P2P bots only. Conferencing scenarios are not supported.

Developer resources

SDKs

To develop a real-time media bot, you must install these NuGet packages within your Visual Studio project:

• Bot Builder Real-Time Media Calling for .NET
• Microsoft.Skype.Bots.Media .NET library

Get started

• Review the documentation to understand the concepts behind building real-time media bots.
• Check out the requirements for real-time media bots.
• Get started quickly with our samples.

Contributing

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

Release notes

• Due to privacy concerns with bots, conferencing scenarios are currently not supported. Hence we have removed the MeetingScreenShots sample. We are working on enabling this feature again.

botbuilder-realtimemediacalling's Issues

is this supported by Skype for business?

Hi,

is this supported by Skype for business?

Swati

HueBot -- OnAnswerAppHostedMediaCompleted --> MediaSession.Dispose() ALWAYS

So the bot receives the call, and can receive messages on the text channel.. but when the call comes in and the callback OnAnswerAppHostedMediaCompleted I am winding up here.

AnswerAppHostedMediaOutcome answerAppHostedMediaOutcome = answerAppHostedMediaOutcomeEvent.AnswerAppHostedMediaOutcome;
                if (answerAppHostedMediaOutcome.Outcome == Outcome.Failure)
                {
                    Log.Info(new CallerInfo(), LogContext.FrontEnd, $"[{CallId}] AnswerAppHostedMedia failed with reason: {answerAppHostedMediaOutcome.FailureReason}");
                    //cleanup internal resources
                    MediaSession.Dispose();
                }

So my MediaSession is immediately disposed. Failure Reason in the logs is

EventName="MessageEvent" Message="RealTimeMediaCallingConversation: System.ArgumentException: Empty Actions List not allowed
   at Microsoft.Bot.Builder.Calling.ObjectModel.Misc.Utils.AssertArgument(Boolean condition, String format, Object[] args)
   at Microsoft.Bot.Builder.RealTimeMediaCalling.ObjectModel.Contracts.RealTimeMediaValidActions.Validate(IEnumerable`1 actions)
   at Microsoft.Bot.Builder.RealTimeMediaCalling.ObjectModel.Contracts.RealTimeMediaWorkflow.Validate(Boolean expectEmptyActions)
   at Microsoft.Bot.Builder.RealTimeMediaCalling.RealTimeMediaCallService.<ProcessConversationResult>d__38.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Bot.Builder.RealTimeMediaCalling.RealTimeCallProcessor.<ProcessCallbackAsync>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
   at Microsoft.Bot.Builder.RealTimeMediaCalling.RealTimeMediaCalling.<SendAsync>d__3.MoveNext()" TraceSource="WaWorkerHost.exe"	WorkerRole_IN_0	d40934041d6e4715a67e03035b092799	6/28/2017 12:19 AM

I haven't changed any the OnIncomingCallReceived method where an ActionBase Array is loaded with a single AnswerAppHostedMedia action is added.

Any help appreciated. !!

There was an error completing this request due to an internal channel failure - BOT

Hello
For past 2 weeks i am facing error "There was an error completing this request due to an internal channel failure" with my bot. BOT App id is "cc35e86c-faf5-4032-aef8-448f47bac745".
It used to work fine earlier from different skype ids. Since last week it started throwing above error for [email protected] login. But with other ids there are no issues. Can you please help.

Getting started links are broken

The links are broken for "Review the documentation" and "requirements". Docs links seem to redirect to the same Github page

Get started **Review the documentation** to understand the concepts behind building real-time media bots. Check out the **requirements** for real-time media bots. Get started quickly with our samples.

No more messages after responding to the original invite

Hello,

I deployed the MeetingScreenshots bot on a computer with public IP. Everything seems to work ok up to the point where the application is answering the incoming call. After that there are no more message coming from Skype or bot connector.
NOTE: This happens when adding the bot to a conference call.
If I call it directly, the call gets connected successfully

Incoming:: POST https://avstdevbot.avst.com/api/calling/call
X-Microsoft-Skype-Chain-ID: 1f3dbf1e-4f4c-4830-9403-4941780ae164
X-Microsoft-Skype-Message-ID: 0ea00ea5-2021-400c-b76a-72d224972672
Connection: Keep-Alive
Accept: application/json
Authorization: Bearer ey....
Host: avstdevbot.avst.com
User-Agent: Microsoft-Skype/3.0,(Calling/1.0)
Content-Length: 1315
Content-Type: application/json; charset=utf-8
{
  "id": "7f210f42-dc98-4968-b86b-1a6f88615575",
  "participants": [
    {
      "identity": "29:1EZXCxJvE_KBFR2nVmvx3yrWyryhu184FAri7w8GWBgw9qDv0o0nHvL5eflC0xET-",
      "originator": true,
      "isHidden": false
    },
    {
      "identity": "28:8c7313ab-d97f-43dd-9658-413b6b5073b4",
      "originator": false,
      "isHidden": false
    }
  ],
  "isMultiparty": true,
  "joinToken": "eyJ...",
  "threadId": "19:[email protected]",
  "presentedModalityTypes": [
    "audio",
    "video"
  ],
  "callState": "incoming"
}

Response to incoming:: POST https://avstdevbot.avst.com/api/calling/call
202 Accepted
Connection: close
X-Microsoft-Skype-Chain-ID: 1f3dbf1e-4f4c-4830-9403-4941780ae164
Content-Type: text/plain; charset=utf-8
{
  "notificationSubscriptions": [
    "rosterUpdate",
    "callStateChange"
  ],
  "links": {
    "callback": "https://avstdevbot.avst.com:443/api/calling/callback",
    "notification": "https://avstdevbot.avst.com:443/api/calling/notification"
  },
  "actions": [
    {
      "operationId": "48a6bf9d-9f77-485c-804f-7a11f2bc0c52",
      "action": "answerAppHostedMedia",
      "mediaConfiguration": {
        "mpUri": "net.tcp://www.avstdevbot.avst.com:5070/MediaProcessor",
        "audioRenderContexts": [
          "46b3d4f3-f9b6-4157-9c2d-d165b0d26678"
        ],
        "videoRenderContexts": [
          "ed4f7777-a379-48c2-9464-10a36a03e35d"
        ],
        "audioSourceContexts": [
          null
        ],
        "videoSourceContexts": [
          null
        ],
        "supportedAudioFormat": "Pcm16K",
        "mpMediaSessionId": "e317cea6-70ce-430a-a9b8-57af8b92edb8",
        "regionAffinity": null,
        "skypeMediaBotsVersion": "1.5.0.1177",
        "mediaStackVersion": "6.0.8980.141",
        "mpVersion": "7.0.697.0"
      }
    }
  ],
  "appState": "7f210f42-dc98-4968-b86b-1a6f88615575"
}

Screen sharing

Hi, are there any examples/instructions about how to enable screen-sharing?

Calling to bot - "you can not talk to the bot just yet..."

Hi

I follow all the instrucation and I deploy the Huebot sample

I have no issue with the deployment and I can see the worker running

I set the certificate (not a self sign one)

But when I call the bot I am getting - "you can not talk to the bot just yet...."

You can't talk to this Bot just yet. But we are working on it

Facing the same issue.

How to receive Text Messages during group conversation

Could you please clarify how it is possible to receive group chat messages?
Thanks.

Determine Group Call participants (names)

Could you please clarify how it is possible to determine call participants (Names).
At the moment I look into HueBot example RealTimeMediaCall class.
To be more specific OnIncomingCallReceived() event with RealTimeMediaIncomingCallEvent parameter that have Conversation property with list of Participants. Each participant has internal identity property that is for example : "29:1c2fgntp......".
Is there a way knowing that participant Identity obtain name of Skype account that takes part in a Call?
Maybe there is another way to determine List of Skype Accounts that take part in a Call?

AudioVideoPlayerBot audio and video files not saved.

Hi,

I just want to know how to properly setup the file locations setting. I tried to deploy and test the AudioVideoPlayerBot sample you have and it seems that no audio and video are being saved.

< Setting name="AudioFileLocation" value="$audioFileLocation$" / >
< Setting name="VideoFileLocation" value="$videoFileLocation$" / >

some questions regarding the above setting:

• what should be? directory/folder name or absolute filename with path?
• where should this be located? local from within the app location on the VM or on Azure Storage?

I also noticed that the MediaSession gets disposed immediately after the bot accepts the call, according to the logs

Also on the logs, I'm getting this exception whenever i do video call:

EventName="MessageEvent" Message="[4710da0c-dc55-4966-bcca-b50dfe8b7323 ForgetAndLogException,Utilities.cs(30)] Caught an Exception running the task: The given path's format is not supported. StackTrace: at System.IO.FileStream.Init(String path, FileMode mode, FileAccess access, Int32 rights, Boolean useRights, FileShare share, Int32 bufferSize, FileOptions options, SECURITY_ATTRIBUTES secAttrs, String msgPath, Boolean bFromProxy, Boolean useLongPath, Boolean checkHost) at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share) at FrontEnd.Utilities.CreateVideoMediaBuffers(Int64 currentTick) in C:\AVBot\FrontEnd\Utilities.cs:line 43 at FrontEnd.Call.MediaSession.d__26.MoveNext() in C:\AVBot\FrontEnd\CallLogic\MediaSession.cs:line 306--- End of stack trace from previous location where exception was thrown --- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at FrontEnd.Utilities.d__0.MoveNext() in C:\AVBot\FrontEnd\Utilities.cs:line 25" TraceSource="FrontEnd"

I'm suspecting its the fileLocation settings, or maybe I'm wrong. :-(

Please help.

.

IIS error with Microsoft.Skype.Bots.Media

Hello!
I want to deploy RealTimeMediaCalling bot application for IIS
By default, I created projects (Bot Application, MVC, WebApi, etc.)
All works good.

After adding Microsoft.Skype.Bots.Media to the project, I see an error.
https://www.screencast.com/t/GYhpa10jJ7a

"Could not load file or assembly 'Microsoft.Rtc.Internal.Media' or one of its dependencies. An attempt was made to load a program with an incorrect format."

I tried:

1. Compiled the project for the platforms AnyCPU, x64, x86
2. Changed "Enable 32-bit Application to True" in ApplicationPool(IIS).
3. Installed the latest version VCRedist x86 and x64
   Nothing helps...

If I delete this library, then again the project works well.
Where my mistake? Can you please help.

PS:I have VS2015(14.0.25421.03 Update 3)

Speech Recognition not working

Hi,

in the HueBot I'm getting an exception by starting speech recognition and there is no inner exception or further information. When I look at the code I'm not sure how the audio stream comes in. The recognition stream will only be set as: _recognitionStream = new SpeechRecognitionPcmStream(16000);

the exception occures at:
await _speechClient.RecognizeAsync(new SpeechInput(_recognitionStream, requestMetadata), _recognitionCts.Token);

EventName="MessageEvent" Message="[56f59758-11a8-48d3-971b-464e61b6358a StartSpeechRecognition,MediaSession.cs(254)] [56f59758-11a8-48d3-971b-464e61b6358a:876b4db5-cdba-48c5-b8ab-7e336f503d86]: Speech recognize threw exception System.Threading.Tasks.TaskCanceledException: A task was canceled.   at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)   at Microsoft.Bing.Speech.AudioReader.d__18.MoveNext()....

Is it possible to use Microsoft translator API ?

I am trying to utilize bot to convert the real time audio conversation into translated text of my language. Is it possible to extend the functionality. If yes please inform where do i need to make adjustments in code ?

Exceptions on incoming connections

I'm stuck. I have the sample deployed, but every attempt to communicate with the bot fails with an exception. There are two cases I'm hitting as I attempt to troubleshoot. Note that I attempted to add a messaging endpoint to the sample as I was struggling getting A/V working and I thought it could help me with troubleshooting.

1. I have the Bot messaging endpoint point to my CNAME record (myhuebot.mydomain.org/api/messages) which points to the actual Azure service FQDN of myhuebot.cloudapp.net. In this case, I get the following error from AppInsights- "POST to the bot's endpoint failed with HTTP status 404".

2. I set the Bot messaging endpoint to the Azure cloud service FQDN (https://myhuebot.cloudapp.net/api/messages). In this case, my cert that I uploaded is for my personal domain and as a result I get an error "Bad certificateThe remote certificate is invalid according to the validation procedure." Now in this case, I am assuming the issue is due to the fact that the ServiceDNSName in ServiceConfiguration.Cloud.cscfg doesn't match the subject name of the certificate I used. For ServiceDNSName I was using myhuebot.cloudapp.net and the cert SN is myhuebot.mydomain.org. So I subsequently tried modifying the ServiceDNSName to match the SN (myhuebot.mydomain.org) and that failed when attempting to deploy to Azure from VS seemingly because of this ServiceDNSName change.

I know there's a deployment guide referenced from the readme, but not having every built a bot that requires certificates I seem to be missing the point here. Any help would be greatly appreciated.

Supported Audio Format

Is there any details on the required Audio format requirements? I have a Wav file but its distorted I tried mulaw and PCM16 using ffmpeg cant seem to find the current format.

Passed stream to AudioVideoFramePlayer and video didn't play

Hi
I am passing stream to AudioVideoFramePlayer but I am just getting only one dark frame and below is my code:

var videoBuffer = new VideoSendBuffer(frame.FrameSegment.Array, (uint)frameSize, VideoFormat.H264_320x180_15Fps);
videoMediaBuffers.Add(videoBuffer);
this.audioVideoFramePlayer.EnqueueBuffersAsync(audioMediaBuffers, videoMediaBuffers).Wait();
please help to me fix it.

How to save the audio conversation of the call?

Hi

Do you have sample on how to record the audio of the conversation? Most specially the audio of each participant on the call conversation.

thanks,

Microsoft.Rtc.Internal.Media assembly error

I am getting this weird error when i try to run the application.

I am using Windows 10, VS 2017, Microsoft.Skype.Bots.Media-1.5.0.1177-alpha, .Net 4.6.

Microsoft.Bot.Builder.RealTimeMediaCalling Nuget package not available.

does the Microsoft.Bot.Builder.RealTimeMediaCalling nuget package removed. If yes, any alternative ?

HueBot sample: Skype bot cannot connect to the service using skype

I followed instructions for registering a bot (https://docs.microsoft.com/en-us/bot-framework/portal-register-bot) and deploying a real-time audio-video bot (https://docs.microsoft.com/en-us/bot-framework/dotnet/bot-builder-dotnet-real-time-deploy-visual-studio) to run the HueBot sample (https://github.com/Microsoft/BotBuilder-RealTimeMediaCalling/tree/master/Samples/HueBot).
But after my bot was added to my skype contacts I cannot make a video call to my bot. I always get a response like here: "You can't talk to this bot just yet. But we are working on it." And at the bot framework dashboard, I have the issues: "There was an error completing this request due to an internal channel failure."
I connected the Visual Studio 2017 debugger to the published job worker and saw that the controller didn't receive any requests.
Bot configuration:
Messaging endpoint: https://dataartskypebots.cloudapp.net
App ID: bf0f99b0-1836-4548-b1c6-6540ee25d191
Enable messaging: Yes;
Enable autoplay of media cards: Yes;
Enable calling: Yes;
Real Time Media: Yes;
Video (Audio alway enabled): Yes;
Screen sharing: Yes;
Webhook (for calling): https://dataartskypebots.cloudapp.net/api/calling/call
Enable adding to group: Yes;

Could you help me what is incorrect in the bot configuration or maybe you have any other recommendations?

Recording Audio/Video in a real time call

Is it possible to record Audio and/or Video in a real time call. I am using Skype for Business Channel and creating a group conference. Bot will be part of group conference and it records audio/video and later transcribes that media using Azure Cognitive services. Is this use case supported by current version of RealTimeMediaCalling SDK ?

Huebot Call ends after 11 seconds

I have the following configuration:

ServieDnsName : huebotstk.cloudapp.net
ServiceCNAME : itcanoe.com

the certificate is from itcanoe.com

The bot makes the call but it doesnt do recognition and it closes

Can't respond to bot call

I trying to respond to a call in Teams but actually I'm not getting a respond from the bot.

First I get access_token from Graph API.

Then I have a route that intercept bot calls.

(app.post("/api/call", function(req, res) {
  if (j === 1) {
    j = j + 1;
    res.status(204).send();
  } else {
    var answerbody = {
      callbackUri: "https://8a73b7ad.ngrok.io/api/call",
      acceptedModalities: ["audio"],
      mediaConfig: {
        "@odata.type": "#microsoft.graph.serviceHostedMediaConfig",
        preFetchMedia: [
          {
            uri: "https://cdn.contoso.com/beep.wav",
            resourceId: "1D6DE2D4-CD51-4309-8DAA-70768651088E"
          },
          {
            uri: "https://cdn.contoso.com/cool.wav",
            resourceId: "1D6DE2D4-CD51-4309-8DAA-70768651088F"
          }
        ]
      }
    };
    POST(
      "https://graph.microsoft.com/beta/" + req.body.resource + "/answer",
      answerbody
    )
      .then(
        data => console.log(data) // I get undefined
      )
      .catch(function(err) {
        console.log("err   " + err);
        res.status(200).send();
      });
  }
});)

Here's POST function

function POST(url, BB) {
  return new Promise(function(resolve, reject) {
    var options = {
      url: url,
      method: "POST",
      headers: {
        Accept: "application/json",
        Authorization: "Bearer " + token
      },
      body: BB,
      json: true
    };
    request(options)
      .then(function(body) {
        resolve(body);
      })
      .catch(function(err) {
        reject(err);
      });
  });
}

As mentionned in documentation , Server sould first reply 204 in order to get response in Graph API protocol.

Actually I don't get a response. Bot still ringing until It gets voice message : " You can't talk to the bot just yet , we are working on it".

As mentioned in Teams API documentation, I should get callback with the ressource id and other information to be able to answer to the call.

So I use my POST function to answer. but here I don't get any 202 Accepted response as indicated in docs, instead I get more than one callback with different ressource ids, then after some seconds I get the voice message.

Unable to call Hue Bot

I have deployed the HueBot and added to my Skype contacts , but when I am trying to call the Bot , it says "you can't talk to the bot just yet" .
On remote debugging , I am seeing
var response = await RealTimeMediaCalling.SendAsync(request, RealTimeMediaCallRequestType.IncomingCall).ConfigureAwait(false);

the above response is returned as 400 Bad Request .
What could be the issue ?
Bot Id : d34db906-1778-44fd-ac47-8277459c2d0c

I have verified all the trouble shoot steps provided in the earlier posts .
The messagecontroller returns HttpStatusCode.Accepted.

AudioSocket is either sending or receiving but not both at the same time

Is there a limitation on sending and receiving at the same time, from multiple threads? I have a solution that works if I am only receiving media, but when I start sending, it stops receiving messages.

Unable to use this example

My bot application in visual studio is running successfully and all the bot configurations on azure portal also successfully configured but still my BOT is not able to join any call.

Deploy to another environment outside of Azure

Would it be possible to deploy RealTimeMediaCalling bot application to a VM different than Azure cloud service?

Microsoft.Skype.Bots.Media in .Net Core

How to add a depedency of Microsoft.Skype.Bots.Media in a .Net Core 3.1 application?

Video playback upside down

I am sending video frames in VideoFormat.Rgb24_424x240_15Fps and they are played on the client upside down.

Any idea what could cause this?

HueBot not available in group calls

I have created a bot on basis of Hue bot example, and till last week it was available during group calls. Starting from this week bot is not available during group calls.

Could someone please confirm that access to the group calls was closed in passed 10 days?
If so, any ideas when bots would be available for group cals again?

Thanks in advance!

Conferencing scenarios

Any idea when using bots in conferencing scenarios will be enabled again?

Sample audio and video files for AudioVideoPlayerBot

Can you please provide a working example of an NV12 YUV video file and PCM16K wav file for the AudioVideoPlayerBot Sample? I've not been able to create a video that is not distorted, or audio that is good quality. I'm using the following ffmpeg commands:

ffmpeg -i input.mov -s 640x360 -pix_fmt nv12 -framerate 15 output.yuv
ffmpeg -i input.mov -acodec pcm_s16le -ar 8K -ab 16K output.wav

Is there a recommended tool or approach for converting modern video formats such as mov/mp4 to the required Real Time Media formats? Or is there a way to provide H264 to the AudioVideoFramePlayer?

Determine Participant.Identity (DisplayName) knowing MediaSourceId

I have faced following issue:

In HueBot sample when I get
OnAudioMediaReceived(object sender, AudioMediaReceivedEventArgs e) event fired event argument e.Buffer.ActiveSpeakers contains list of MediaSourceIds of active speakers in the conference.

Could you please clarify how it is possible to determine Participant.Identity or DisplayName knowing that participant MediaSourceId?

HueBot : Error on incoming call

Hello,

I deployed the HueBot Sample on Azure with a cloud service. The bot's configuration seems to be ok because i can send a message, but when i try to call it i got : "you can not talk to the bot yet ...".
With Remote debugger, the event "OnCallback" is raised but i received some errors in the logs :

• EventName="MessageEvent" Message="Exception in conversationResult validate Newtonsoft.Json.JsonSerializationException: Required property 'operationOutcome' not found in JSON. Path '', line 1, position 398. at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.EndProcessProperty(Object newObject, JsonReader reader, JsonObjectContract contract, Int32 initialDepth, JsonProperty property, PropertyPresence presence, Boolean setDefaultValue) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject(Object newObject, JsonReader reader, JsonObjectContract contract, JsonProperty member, String id) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal(JsonReader reader, Type objectType, JsonContract contract, JsonProperty member, JsonContainerContract containerContract, JsonProperty containerMember, Object existingValue) at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(JsonReader reader, Type objectType, Boolean checkAdditionalContent) at Newtonsoft.Json.JsonSerializer.DeserializeInternal(JsonReader reader, Type objectType) at Newtonsoft.Json.JsonSerializer.Deserialize(JsonReader reader, Type objectType) at Newtonsoft.Json.JsonConvert.DeserializeObject(String value, Type type, JsonSerializerSettings settings) at Newtonsoft.Json.JsonConvert.DeserializeObject[T](String value, JsonSerializerSettings settings) at Microsoft.Bot.Builder.RealTimeMediaCalling.ObjectModel.Misc.RealTimeMediaSerializer.DeserializeFromJson[T](String json) at Microsoft.Bot.Builder.RealTimeMediaCalling.RealTimeCallProcessor.d__5.MoveNext()" TraceSource="WaWorkerHost.exe"

• EventName="MessageEvent" Message="No ServiceUrls added to trusted list" TraceSource="WaWorkerHost.exe"

Thanks

HueBotSample: AnswerAPpHostedMedia failed with reason: CallEnded

After deploying the HueBot on an azure instance the bot doesn't answer the call. Checking the logs it seems it receives the call and returns accept, however the call ends before that?

Response to incoming call:

228925aa-3725-4369-8655-26e6f36aacbb SendAsync,LoggingMessageHandler.cs(139)] 
|| correlationId=228925aa-3725-4369-8655-26e6f36aacbb 
|| statuscode=Accepted 
|| local.msgid=1a61ed64-5c1e-4fa4-a078-0a93fc095961 
||Response to incoming:: POST https://cloud-rtc.luware.com/api/calling/call
202 Accepted  
Connection: close  
X-Microsoft-Skype-Chain-ID: 228925aa-3725-4369-8655-26e6f36aacbb  
Content-Type: text/plain; charset=utf-8  
{ "notificationSubscriptions": [ "callStateChange" ],    
"links": {  
  "callback": "https://cloud-rtc.luware.com:10100/api/calling/callback/",  
  "notification": "https://cloud-rtc.luware.com:10100/api/calling/notification/" 
 }, 
 "actions": [
    {
      "operationId": "9172976d-12bd-4bf5-9e85-0061dbd79bd5",
      "action": "answerAppHostedMedia",
      "mediaConfiguration": {
        "mpUri": "net.tcp://cloud-rtc.luware.com:20100/MediaProcessor",
        "audioRenderContexts": [
          "29dcaadf-142f-46d3-b210-ea71ff9fbe30"
        ],
        "videoRenderContexts": [
          "b204323a-dd46-4af9-bed0-33e161c1c310"
        ],
        "audioSourceContexts": [
          "c3318e2c-116b-44f3-afd8-7641f62a8b05"
        ],
        "videoSourceContexts": [
          "dadc7d7e-8bf2-4ab1-aebc-5ce1d2527313"
        ],
        "supportedAudioFormat": "Pcm16K",
        "mpMediaSessionId": "609f1c32-4618-4643-8632-070404f42656",
        "regionAffinity": null,
        "skypeMediaBotsVersion": "1.5.0.1177",
        "mediaStackVersion": "6.0.8980.141",
        "mpVersion": "7.0.697.0"
      }
    }
  ],
  "appState": "fe9a3651-6d4e-4390-8e39-be273f31522f"}
$$END$$

Incoming Callback:

[228925aa-3725-4369-8655-26e6f36aacbb SendAsync,LoggingMessageHandler.cs(90)] 
|| correlationId=228925aa-3725-4369-8655-26e6f36aacbb 
|| local.msgid=a8c1bc78-30b8-4e02-a26b-2e6983dd6158 
||Incoming:: POST https://cloud-rtc.luware.com:10100/api/calling/callback/
X-Microsoft-Skype-Chain-ID: 228925aa-3725-4369-8655-26e6f36aacbb
X-Microsoft-Skype-Message-ID: d8372b67-5ec5-4c7c-861e-73cfe0a15970
Connection: Keep-Alive
Accept: application/json
Authorization: Bearer eyJ0eXAiOiJK...
Host: cloud-rtc.luware.com:10100
User-Agent: Microsoft-Skype/3.0,(Calling/1.0)
Content-Length: 386
Content-Type: application/json; charset=utf-8
{
  "id": "fe9a3651-6d4e-4390-8e39-be273f31522f",
  "operationOutcome": {
    "type": "answerAppHostedMediaOutcome",
    "id": "9172976d-12bd-4bf5-9e85-0061dbd79bd5",
    "outcome": "failure",
    "failureReason": "CallEnded"
  },
  "callState": "terminated",
  "appState": "fe9a3651-6d4e-4390-8e39-be273f31522f",
  "links": {
    "call": "https://b-pma-euwe-01.plat.skype.com:6706/platform/v1/calls/76e5982d-b241-4322-a645-3d6261189adc"
  }}
  $$END$$

Any Ideas where the issue lies?
Only change I did was changing the workerrole to a Standard_D2 vmsize.
Thanks

MPServiceHostLib.dll- The requested Performance Counter is not a custom counter, it has to be initialized as ReadOnly.

Just trying to build and deploy the HueBot sample, I'm failing to deploy and run in the emulator. I'm getting an exception out of the WorkerRole at OnStart related to MPServiceHostLib.dll. Exception info is below. I don't think I have changed anything that would have broken it so I'm at a loss. Any ideas what I'm missing?

Exception thrown: 'System.InvalidOperationException' in MPServiceHostLib.dll
FrontEnd Error: 0 : [- OnStart,WorkerRole.cs(68)] Exception on startup: System.InvalidOperationException: The requested Performance Counter is not a custom counter, it has to be initialized as ReadOnly.
at Microsoft.Rtc.MPService.Internal.MPServiceHostMonitor.EndInitialize(IAsyncResult result)
at Microsoft.Rtc.MPService.Internal.MPAzureAppHost.Start(String publicServiceUri, Int32 localPort, X509Certificate2 certificate, GetTrustedServersDelegate getTrustedServersDelegate, String alternateConfigFilePath)
at Microsoft.Rtc.MPService.Internal.MPAzureAppHost.Initialize(String publicRoleInstanceFQDN, Int32 publicRoleInstancePort, Int32 localPort, X509Certificate2 certificate, MPServiceEventDelegate eventDelegate, GetTrustedServersDelegate getTrustedServersDelegate, String alternateConfigFilePath)
at Microsoft.Skype.Internal.Bots.Media.InternalMediaPlatform.Initialize(MediaPlatformSettings settings, IConfigurationManager configurationManager, ILogger logger, Boolean isTest)
at Microsoft.Skype.Bots.Media.MediaPlatform.Initialize(MediaPlatformSettings settings)
at FrontEnd.Service.Initialize(IConfiguration config) in C:\Users\timrang\Source\Repos\BotBuilder-RealTimeMediaCalling\Samples\HueBot\FrontEnd\Service.cs:line 49
at WorkerRole.WorkerRole.OnStart() in C:\Users\timrang\Source\Repos\BotBuilder-RealTimeMediaCalling\Samples\HueBot\WorkerRole\WorkerRole.cs:line 57
Exception thrown: 'System.InvalidOperationException' in WorkerRole.dll

This repo is missing important files

There are important files that Microsoft projects should all have that are not present in this repository. A pull request has been opened to add the missing file(s). When the pr is merged this issue will be closed automatically.

Microsoft teams can learn more about this effort and share feedback within the open source guidance available internally.

Merge this pull request

Node.js support

Hi,

Is node.js support scheduled for this service? We're hoping to add Skype calling to our application, but our chatbots are running on node.js servers.

Support for Microsoft.CognitiveServices.Speech

It's no longer possible to add Bing Speech subscriptions to my Azure subscription as it has been retired in favour of Microsoft.CognitiveServices.Speech.

As a result I can't get the code working. When I plug the new engine in, everything seems to work, but it doesnt ever recognise any speech.

Happy to help sort this out.

Nuget Package dependency error with latest version of Microsoft.Bot.Builder

Unable to update Microsoft.Bot.Builder package to 3.8.5 as it is still referring older version of System.IdentityModel.Tokens.Jwt.5.1.4 . Please update the nuget dependency reference.

Unable to resolve dependencies. 'System.IdentityModel.Tokens.Jwt 5.1.4' is not compatible with 'Microsoft.Bot.Builder 3.8.5 constraint: System.IdentityModel.Tokens.Jwt (>= 4.0.4.403061554 && < 5.0.0)'.

Could not install package 'Microsoft.IdentityModel.Protocol.Extensions 2.0.0-beta6-207211625'. You are trying to install this package into a project that targets '.NETFramework,Version=v4.6', but the package does not contain any assembly references or content files that are compatible with that framework. For more information, contact the package author.

MS Teams API don't send statusCode when answering to a call

I have a bot which should answer to calls, here's the documentation

Actually I can Answer a call (MS Team unhook the call ) but I get no response status code or error. In documentation it mentioned that I should get a 202 accepted status code Microsoft Graph SDK ( api method right after initialisation):

client
    .api('/users/me/sendMail')
    .post({message: mail}, (err, res) => {
        console.log(res)
    })

I have tried to check res.statusCode but I get no content.
How can I get response from MS Teams API ?

Not able run this Audio Video Player example

Not able run this Audio Video Player example @ssulzer will you be able to help me to run this example using microsoft teams.

Unable to run HueBot locally using ngrok

i am facing issues running HueBot sample locally with ngrok .
when i hit the messages api from Test Web Client i get below requests on ngrok utility.
POST /api/messages 404 Not Found
OPTIONS /api/messages 200 OK
Any pointers , where i am going wrong ?

NV12 Video format

Any pointers for creating videos that are compatible for playback using the real time media library? I have uploaded a few samples and cant seem to get them to stream across. I see in the source code it requires NV12 Frames.

'You can't talk to this Bot just yet. But we are working on it' issue

Hi,
i follow the step to deploy Huebot.
the huebot deploy success and i add a certificate .
i use Bot service's test in web chat tool to test send message and it can be send success.
but when i use skype to call to this bot, the bot say 'You can't talk to this Bot just yet. But we are working on it'
what's wrong for me to get this response?

Minimum VM Scale Set size requirements for v1.17.0.39-alpha?

Hello!

I've got an application hosted Service Fabric setup which runs great locally. It used to also run from SF in Azure, but after I pulled down the non-preview packages for Microsoft.Graph.Communications.X libraries and upgraded the Microsoft.Skype.Bots.Media preview package to v1.17.0.39-alpha, it seems to lock up in Azure Service Fabric.

Here are the versions of the packages:

I can run this locally using ngrok and it is very responsive. From Azure, I can get the bot to answer the first call but it locks up during the call. Any subsequent call fails with the "You can't talk to this bot, but we are working on it message". This used to work from Azure when I was using the older versions of packages. I think the media library was 1.13.x.

One thing I had to do to get it to work at all was install the VC++ redistributable for 2015-2019. The VM Scale set we are using only had the VC++ redist up to 2017. This is the same VM which used to work before I updated the NuGet packages.

We are currently testing this with a 3 node VM Scale Set using the 'D2_v2' VM size. Could that be the problem?

Thank you!

Deployed, got cert from CA now getting System.Security.Authentication.AuthenticationException

I am not sure where I've gone wrong here, but I am having some certificate issues.

I registered a domain holovisor.net and got a SSL certificate from godaddy for bot.holovisor.net. I added a CNAME to my azure cloud service holovisor.cloudapp.net. I've uploaded the certificate, deployed the HueBot project, and reviewed the existing issues.

Let's summarize the full stack configuration to define what is inconsistent:
SubjectName of our certificate: 'bot.holovisor.net' last SAN 'www.bot.holovisor.net'
DNS name of our cloud service is 'holovisor.cloudapp.net';
We created CNAME DNS record bot.holovisor.net -> holovisor.cloudapp.net.
We created CNAME DNS record www.bot.holovisor.net -> holovisor.cloudapp.net.

Public IP address holovisor cloud service at Azure dashboard: 13.82.217.186
nslookup bot.holovisor.net

Non-authoritative answer:
Name: holovisor.cloudapp.net
Address: 13.82.217.186
Aliases: bot.holovisor.net

At the bot's dashboard we have following configuration:
Messaging endpoint: https://www.bot.holovision.net/api/messages
Webhook (For calling): https://www.bot.holovision.net/api/calling/call
Enable calling, Real Time Media, Video (Audio always enabled), Screen sharing
In the configuration of the service we have ServiceDnsName -> holovision.cloudapp.net
ServiceDnsName -> holovisor.cloudapp.net
ServiceDnsAlias -> www.bot.holovisor.net

In analytics I am seeing the following error.

POST to HoloVisorBot failed: Bad certificate
Exception type: System.Security.Authentication.AuthenticationException
Failed method: System.Net.TlsStream.EndWrite
Problem Id: System.Security.Authentication.AuthenticationException at System.Net.TlsStream.EndWrite

Any help greatly appreciated. Thanks

Calling to bot -"you can not talk to the bot just yet..."

Hi
I have the issue that when i call the bot I'm getting: "you can not talk to the bot just yet..."
I''ve properly configured the callback url and the web hook in Skype Setting. I've also seen the feedback from a same issue #32
But it still doesn't work. (Messaging works fine)

Can anyone check the log for me? Thanks in advance
appid: 9a8481d7-9330-4a46-83a6-16d1e28e6a99

How can be defined active participant in the group call?

We subscribe AudioSocket.AudioMediaReceived and AudioSocket.DominantSpeakerChanged events. When the AudioMediaReceived event is raised we have a raw audio chunk. We send this audio chunk to the transcriber as you demonstrated in the HueBot sample. But we can't define which participant this audio chunk belongs to. Therefore we use DominantSpeakerChanged event to save dominant speaker ID. But this event is raised with a big delay after the current dominant speaker changes.

What is the best way to define which speaker a currently received audio chunk belongs to?