Hostname/IP does not match certificate's altnames about botbuilder-v3

[vohuythao]

Hello guys,

I have fixed this by replace another SSL certificates.

The one from GoDaddy, has some problem.
I replace with the one from Digicert and it is ok now

[mdrichardson]

@OksanaBerdnik How is your bot hosted? If it's in Azure, can you go to Azure Portal > Your Resource Group > Your App Service > TLS/SSL Settings and ensure that TLS 1.2 is enabled and everything else looks like this:

I haven't seen this error before, but it recently became required to use TLS 1.2, so I'm guessing it's related to that. If not, please provide:

• How your bot is hosted
• Your appId
• Whether or not Test in Web Chat works
• Which channels are or are not working

[hex-344505]

the case is that one request gives valid certificate and another - not correct
same curl request return

• SSL: no alternative certificate subject name matches target host name 'facebook.botframework.com'
  and
• Server certificate:
• subject: CN=*.botframework.com
• start date: Jun 10 20:07:48 2019 GMT
• expire date: Jun 10 20:07:48 2020 GMT
• subjectAltName: host "facebook.botframework.com" matched cert's "*.botframework.com"
• issuer: C=US; ST=Washington; L=Redmond; O=Microsoft Corporation; OU=Microsoft IT; CN=Microsoft IT TLS CA 4
• SSL certificate verify ok.

in same time facebook.botframework.com have only one IP address, so it behind balancer or something else

[OksanaBerdnik]

@mdrichardson Attachments to previous comment:

In case certificate is valid:

In case the certificate is not matched (the same curl):

Host:

• How your bot is hosted
  The bots are hosted on our servers, not in Azure App Service.
• Whether or not Test in Web Chat works
  In Web Chat bot works fine
• Which channels are or are not working
  Facebook Channel

This error occurs from time to time but often

[mdrichardson]

@hex-344505 @OksanaBerdnik Thank you for the additional info! Can you try deleting (not disable, but delete) and then re-add your Facebook channel from Azure Portal > Your Resource Group > Your Web App Bot/Bot Channels Registration > Channels?

[OksanaBerdnik]

@mdrichardson I've deleted the channel and re-added it. But it doesn't work for me

[mdrichardson]

@OksanaBerdnik Alright. It was sort of a hail mary before I rope in some other folks to assist. I'm reaching out to see if one of our instances hosting facebook.botframework.com has an invalid cert. I will update this issue once I find the cause.

[mdrichardson]

@hex-344505 @OksanaBerdnik

I tried that curl command ~100 times and all were successful. We're also seeing that all of the certs are valid on the ~10 instances running facebook.botframework.com.

Can you try running nslookup facebook.botframework.com?

The output should be something like:

$> nslookup facebook.botframework.com
Server:  f5-2.redmond.corp.microsoft.com
Address:  10.50.10.50

 

Non-authoritative answer:
Name:    waws-prod-bay-051.cloudapp.net
Address:  40.78.18.232
Aliases:  facebook.botframework.com
          bc-facebook.trafficmanager.net
          bc-facebook-westus.azurewebsites.net
          waws-prod-bay-051.vip.azurewebsites.windows.net

Can you then run curl -v <HOST>, where is the equivalent in your output to bc-facebook-westus.azurewebsites.net, above?

[OksanaBerdnik]

@mdrichardson

curl -v https://facebook.botframework.com/api/v1/bots/****

*   Trying 40.89.131.148...
* Connected to facebook.botframework.com (40.89.131.148) port 443 (#0)
* found 148 certificates in /etc/ssl/certs/ca-certificates.crt
* found 592 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_256_GCM_SHA384
* 	 server certificate verification OK
* 	 server certificate status verification SKIPPED
* SSL: certificate subject name (*.azurewebsites.net) does not match target host name 'facebook.botframework.com'
* Closing connection 0
curl: (51) SSL: certificate subject name (*.azurewebsites.net) does not match target host name 'facebook.botframework.com'


nslookup facebook.botframework.com

Server:		127.0.1.1
Address:	127.0.1.1#53

Non-authoritative answer:
facebook.botframework.com	canonical name = bc-facebook.trafficmanager.net.
bc-facebook.trafficmanager.net	canonical name = bc-facebook-par.azurewebsites.net.
bc-facebook-par.azurewebsites.net	canonical name = waws-prod-par-003.sip.azurewebsites.windows.net.
waws-prod-par-003.sip.azurewebsites.windows.net	canonical name = waws-prod-par-003.cloudapp.net.
Name:	waws-prod-par-003.cloudapp.net
Address: 40.89.131.148

[mdrichardson]

This is becoming quite the head-scratcher. By any chance, can you visit dev.botframework.com and ensure that you trust the certificate there:

---

Alternatively (and I understand this isn't ideal), can you try creating a new App Service, deploying your bot to it, then pointing your Web App Bot/Bot Channels Registration to the new App Service?

---

Also, where are you running curl from? Can you try the same command from Azure Portal > Your Resource Group > Your App Service > Console and copy/paste the output here?

[hex-344505]

I tried that curl command ~100 times and all were successful. We're also seeing that all of the certs are valid on the ~10 instances running facebook.botframework.com.

Counter 20
subjectAltName: host "facebook.botframework.com" matched cert's ".botframework.com"
Thu, 31 Oct 2019 10:25:51 +0200
Counter 21
subjectAltName does not match facebook.botframework.com
SSL: no alternative certificate subject name matches target host name 'facebook.botframework.com'
curl: (60) SSL: no alternative certificate subject name matches target host name 'facebook.botframework.com'
Thu, 31 Oct 2019 10:25:51 +0200
Counter 22
subjectAltName: host "facebook.botframework.com" matched cert's ".botframework.com"

Thu, 31 Oct 2019 10:25:53 +0200
Counter 35
subjectAltName: host "facebook.botframework.com" matched cert's ".botframework.com"
Thu, 31 Oct 2019 10:25:53 +0200
Counter 36
subjectAltName does not match facebook.botframework.com
SSL: no alternative certificate subject name matches target host name 'facebook.botframework.com'
curl: (60) SSL: no alternative certificate subject name matches target host name 'facebook.botframework.com'
Thu, 31 Oct 2019 10:25:54 +0200
Counter 37
subjectAltName: host "facebook.botframework.com" matched cert's ".botframework.com"

this is output of simple script
#!/bin/bash
for i in {1..100}
do
date -R >> request.txt
echo "Counter $i" >> request.txt
curl -v https://facebook.botframework.com 2>&1 | grep match >> request.txt
done

NSLOOKUP
nslookup facebook.botframework.com
Non-authoritative answer:
Name: waws-prod-par-003.cloudapp.net
Address: 40.89.131.148

hexus@devops:~$ nslookup dev.botframework.com
Non-authoritative answer:
Name: waws-prod-par-001.cloudapp.net
Address: 52.143.137.150

same resolving i get from googles dns (8.8.8.8)

[OksanaBerdnik]

@mdrichardson The certificate is valid on dev.botframework.com.
Previously I've run curl from on AWS node or on the local machine but from Azure console, it works fine

*   Trying 40.118.29.72...
* Connected to facebook.botframework.com (40.118.29.72) port 443 (#0)
* found 148 certificates in /etc/ssl/certs/ca-certificates.crt
* found 594 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_RSA_AES_256_GCM_SHA384
*        server certificate verification OK
*        server certificate status verification SKIPPED
*        common name: *.botframework.com (matched)
*        server certificate expiration date OK
*        server certificate activation date OK
*        certificate public key: RSA
*        certificate version: #3
*        subject: CN=*.botframework.com
*        start date: Mon, 10 Jun 2019 20:07:48 GMT
*        expire date: Wed, 10 Jun 2020 20:07:48 GMT
*        issuer: C=US,ST=Washington,L=Redmond,O=Microsoft Corporation,OU=Microsoft IT,CN=Microsoft IT TLS CA 4
*        compression: NULL
* ALPN, server did not agree to a protocol
> GET /api/v1/bots/**** HTTP/1.1
> Host: facebook.botframework.com
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Cache-Control: no-cache
< Pragma: no-cache
< Expires: -1
< Server: Microsoft-IIS/10.0
< x-ms-request-id: |384d0b77cd4c0041aa976b83b5716c2e.bc782601_
< Strict-Transport-Security: max-age=31536000
< Date: Thu, 31 Oct 2019 08:19:24 GMT
< Content-Length: 0
<
* Connection #0 to host facebook.botframework.com left intact

When I try to visit Manage page to generate app password I receive 404, please see screenshot below (it`s related to all apps, not only to new one):

[hex-344505]

above script from AWS instance eu-west-3 zone - every 10th request give error
same script on EC2 in u-west-1 1000 requests - no errors
I have no idea what happens :(

[mdrichardson]

@OksanaBerdnik @hex-344505 I was finally able to repro this by VPN-ing to Paris and running that script. I'll dig into this further.

@OksanaBerdnik Regarding that 404. Where is that bot hosted? I've seen that happen occasionally if your bot is hosted off of Azure. You should be able to access the app registration by going directly to the App Registration page or https://portal.azure.com/#blade/Microsoft_AAD_RegisteredApps/ApplicationMenuBlade/Overview/appId/<yourAppId>/isMSAApp/, if that doesn't work.

[OksanaBerdnik]

@mdrichardson All our bots are hosted on our servers, not in Azure App Service. Regarding that 404: App Registration page has helped me, thank you! But regarding your advice to host our bot on Azure portal: are there other ways to solve this issue?

[mdrichardson]

@OksanaBerdnik Regarding hosting off-Azure:

I don't believe this affects the 404, as I'm able to repro through VPN on the same machine that works without VPN. We'll dig into what's going on and let you know as soon as we do.

The only issue we've talked about that you'll continue to have while hosting yourself is accessing the App Registration via the Web App Bot. I believe the Azure Bot Services team is aware of this, but it's a low-priority issue.

So, with all that said, I don't necessarily advise that you host on Azure since I don't believe it will fix this issue.

If you're looking for a temporary workaround, you might try migrating your servers to a West US instance/machine, if you can--that's where I am and where I'm unable to repro.

Again, though, we'll look into this and see if we can get it fixed.

[mdrichardson]

@OksanaBerdnik @hex-344505 Are you guys still able to repro this? I VPN'd to Paris again (which was how I repro'd yesterday) this morning and had the script run 4,000 times (not exaggerating). All of them were successful.

Edit: Never mind. Getting failures now. I think my VPN disconnected at some point while the script was running, initially.

[mdrichardson]

@OksanaBerdnik @hex-344505 Are you still able to repro this? I just tested from Paris and was successful for 200 of 200 requests.

If not, please capture a System.NET trace and paste the contents of networktrace.txt.

[mdrichardson]

@OksanaBerdnik @hex-344505 Were you able to re-test this? There's a couple of different teams looking into this and none of us can reproduce the error any more.

[OksanaBerdnik]

@mdrichardson Today we've reproduced this bug on AWS instance located in Paris

[mdrichardson]

@OksanaBerdnik Understood. I can't seem to repro this any more, but we'll continue looking into this.

[hex-344505]

@OksanaBerdnik Understood. I can't seem to repro this any more, but we'll continue looking into this.

you know, i think that i can prepare an instance for testing , if it will be suitable for you - let me know

by the way , we also repro this bug even from our local ISP.

[mdrichardson]

@hex-344505 That would likely help, if you're able to. Feel free to send details to me via email, if you'd prefer: vDASHmicricATmicrosoftDOTcom (replace all-caps with appropriate character).

[hex-344505]

@hex-344505 That would likely help, if you're able to. Feel free to send details to me via email, if you'd prefer: vDASHmicricATmicrosoftDOTcom (replace all-caps with appropriate character).

sent you an email, thanks in advance

[mdrichardson]

@hex-344505 I was able to SSH in and verify this still occurs for you. Still looking into things. I believe it's something the Azure team will need to fix.

[hex-344505]

we have mail from azure team that they fix it but issue still exist :(
perhaps ill give them access too, or , maybe, you can do it

[mdrichardson]

We’ve got a couple of separate issues still open and tracking this. I’m guessing that one got sent to you that it was fixed because they asked me to repro and I couldn't, then we didn't hear from you that it was still an issue. They're still open and being worked on, however.

I can give these other teams access, if you'd like.

[hex-344505]

I can give these other teams access, if you'd like.

it will be great.
and i updated support request.
Thanks

[mdrichardson]

@OksanaBerdnik @hex-344505 Azure reps are reporting they fixed one of their front end webservers and the issue should be fixed. Are you able to reproduce it, now?

[hex-344505]

@OksanaBerdnik @hex-344505 Azure reps are reporting they fixed one of their front end webservers and the issue should be fixed. Are you able to reproduce it, now?

looks like they fix it. i cant reproduce error :)
Thanks ! your save our bot :)

[mdrichardson]

Good to hear!

[vohuythao]

Hello guys,

Today we met exactly this problem.

I try to curl -v https://api.xxx.com

And in 100 times, we got around 20 times with this error

Trying 51.143.102.21:443...

• Connected to api.xxx.com (51.143.102.21) port 443 (#0)
• ALPN, offering h2
• ALPN, offering http/1.1
• successfully set certificate verify locations:
• CAfile: C:/Program Files/Git/mingw64/ssl/certs/ca-bundle.crt
  CApath: none
• TLSv1.3 (OUT), TLS handshake, Client hello (1):
• TLSv1.3 (IN), TLS handshake, Server hello (2):
• TLSv1.2 (IN), TLS handshake, Certificate (11):
• TLSv1.2 (IN), TLS handshake, Server key exchange (12):
• TLSv1.2 (IN), TLS handshake, Server finished (14):
• TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
• TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
• TLSv1.2 (OUT), TLS handshake, Finished (20):
• TLSv1.2 (IN), TLS handshake, Finished (20):
• SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
• ALPN, server did not agree to a protocol
• Server certificate:
• subject: CN=*.azurewebsites.net
• start date: Sep 28 19:00:01 2020 GMT
• expire date: Sep 28 19:00:01 2021 GMT
• subjectAltName does not match api.xxx.com
• SSL: no alternative certificate subject name matches target host name 'api.xxx.com'
• Closing connection 0
• TLSv1.2 (OUT), TLS alert, close notify (256):
  curl: (60) SSL: no alternative certificate subject name matches target host name 'api.xxx.com'
  More details here: https://curl.haxx.se/docs/sslcerts.html

We use Azure App Service with Traffic Manager

Any hints to help us solve this?
Thank you very much

[mdrichardson]

@vohuythao I'm assuming api.xxxx.com is the domain for your App Service? If so, it would be best to open a ticket for this through Azure support, since this is more of a service issue and less of a code one.

[vohuythao]

Hello @mdrichardson
Thank you for the reply
Yes, that is correct, it is our App Service.
I also open a ticket for Azure support.
I will update there if there is any solutions from Azure support

Thank you