Problems with certificate in mozilla / incomplete certificate chain about letsencrypt-siteextension

[sjkp]

MS is gonna try stuff on the scale unit that hosts one of my sites. Specifically they want to remove the x1 cert. Apparently that requires a scheduled roll out which is gonna happen within the next couple of days. If it works, the solution that MS can provide now is to do the same across all azure scale units. When that happens people should renew their certs. Pretty scary fix. :/ way more complicated than what I was hoping for.

[paulirwin]

@sjkp First off, I want to make clear that my frustration with this is not directed at you. You run an awesome project and I greatly appreciate your community involvement and your working with Microsoft on this issue. So my frustration is at the proposed solution, and partly at LE, and I want people reading this post later to understand the ramifications of this change.

It is highly likely that some of the websites that will have their X1 cert yanked -- at which time their customers will see scary messages in Firefox et al -- are ecommerce websites that collect money from people. This seemingly small annoyance to devs running personal websites will have a real negative economic impact to businesses conducting commerce on the web.

But apart from that, the key point is that this most likely will happen again. Even if X3 doesn't need to fail over to X4 (aside: you should never bet against a DR plan), if they continue the status quo of reusing the key when ECDSA rolls out, then we'll be here again in August. From the "Upcoming Features" page:

Let’s Encrypt only signs end-entity certificates with RSA intermediates. We will add the ability to have end-entity certs signed by an ECDSA intermediate.

That implies that it will be a new intermediate pair - X5 and X6 maybe? And this will be happening before August 1, so within the next 90 days. And I actually hope it's a new intermediate pair (with a different key) because if it is possible to reissue X3 as an ECDSA compatible cert, that seems like it would just amplify this issue and make it even worse.

The decision by LE to introduce a new intermediate pair without a key signing ceremony (per this comment, if that was the actual reason for having the same subject key) is questionable at best. But continuing to issue certificates against X3 knowing the issue is still out there and not fully resolved is even more frustrating. They should have pulled the plug on issuing new X3 certs as soon as the issue arose.

[hansarnevartdal]

I can confirm that i am now getting an A at ssllabs.com with X3 cert. Happy day!

[luisgizirian]

South Central US works fine again! Thanks @sjkp for your support!

[ptaffet]

I mitigated the problem on my site by rolling back to a certificate issued before the change. It won't expire until May, so we have a few weeks to figure something out.

[acekyd]

got it working now... Thanks @sjkp @paulirwin

[paulirwin]

Also, I recognize that it is a "public beta", so we (LE users) shouldn't probably be relying on it as much as we are. But part of a beta feedback cycle is finding real issues like this one and solving them, not sweeping them under the rug.

[sjkp]

Microsoft told me that they have not rolled it out to all scale units, they will do so as part of the May update (if we want them to. I said said YEESSS to that since most X1 certs are gonna expire at latest end of June).

They told me that they have only rolled it out to my scale unit which is in west europe.

[matrix4123]

Looks like East US is now working properly!

[sjkp]

That is strange - I can see that it fails on recently issued certificates probably because they are issued by Let's Encrypt Authority X3 whereas the older ones that works were issued by X1.

Here's how it looks on a site of mine in mozilla on windows. And that gets a green padlock.

I will ask the lets encrypt guys.

[sjkp]

https://community.letsencrypt.org/t/certificates-issued-by-lets-encrypt-authority-x3-not-valid-in-mozilla/13320

[paulirwin]

I am encountering the same issue using this extension on an Azure Web App. It does seem to be working in some browsers that have cached the intermediate but the intermediate doesn't seem to be part of the certificate chain. SSL Labs is saying the X3 intermediate cert was not included. I am getting the "untrusted" message in Firefox and Chrome for Android, but it works in my Chrome for Windows and Edge.

Message from Firefox:

The certificate is not trusted because the issuer certificate is unknown. The server might not be sending the appropriate intermediate certificates. An additional root certificate may need to be imported. (Error code: sec_error_unknown_issuer)

Is there a way to force the Azure web app to include the intermediate certificates? Is this site extension not using the full cert file with intermediates?

[paulirwin]

I confirmed what the poster in that link @sjkp posted said, that the certificate was issued by X3 but it looks like X1 might be included? Here's screenshots from Chrome's certificate details:

[paulirwin]

Okay now I'm even more confused. I found the PFX file by FTPing into the site, extracted the cert from the pfx with openssl, verified the thumbprint matches, opened it in my Ubuntu VM, and it looks like it does have the X3 intermediate:

Why would the browsers be saying the intermediate either isn't there or is the X1 intermediate, when the pfx uploaded to the site has the X3 intermediate?

[sjkp]

When I import the pfx on my windows machine it does look wrong.

But I'm investigating the code in the ACMESharp library now. It does download the correct intermediate cert, and tries to put it into the chain, but the pfx ends up with the old intermediate anyways. But must admit I'm not an expert on this, which is why I used libraries for the extension so I didnt have to deal with that. Oh well

[sjkp]

I think it is a windows issue. It selects the wrong intermediate if both x3 and x1 are present.

[paulirwin]

Unfortunately it isn't just Windows. Having the same issue on Chrome for Android. It's trying to use the X1 intermediate.

[sjkp]

Well it is a windows issue on the server. Not on the client. It is because the windows certficate store can't resolve the chain correctly, which then result in the IIS not being able to serve the correct intermediates certificates. The end result is that all clients that don't have the intermediate from somewhere else will fail.

[paulirwin]

Ah, that makes sense. I was just about to comment that I tested on an iOS device and it has the issue as well.

I imported the pfx onto my Windows machine and I'm not seeing the same results as you. On my side the intermediate for X3 is showing correctly and i can open the details of it just fine:

So it doesn't seem to be an issue with ACMESharp or the site extension, AFAICT. What makes me worry a bit and might explain part of the problem is that the Let's Encrypt Authority X3 cert was only issued 12 days ago.

[sjkp]

My guess is that you azure site hosts have the X1 intermediate maybe from an earlier installation or maybe it is just installed per default on the machines, and yes I totally agree with you, unless we are missing something obvious - this is really crappy to correct.

Yes the Let's Encrypt Authority X3 certificate is 12 days old, but certificates using it have only been issued since the 25th. https://crt.sh/?identity=%25&iCAID=16418&p=571

[paulirwin]

I just requested a new cert and it also was issued by X3, same problem. Again I opened up the pfx and it looks perfect to me, so I think this must be some Azure issue. Trying a few other things now...

[sjkp]

You could try to uninstall any old certificates you have using the azure portal. But I'm unsure if they actually remove the intermediates, if not, we are pretty much forced to open a ticket with Microsoft asking for a fix.
I created a new app hosting plan yesterday, and made a whole new site, even that had the error, so I think the certs are on the machines when we get them. Also not being able to have both certs are a major problem in the transitioning period, until all old X1 certs expire (at least from my point of view) for individual users they can just request a new one, but I can't really do anything with my extension other than asking people to update every cert (when we find a solution):

[paulirwin]

Is there any way to force the ACME process to use X1 instead of X3 until the issue is resolved?

By the way, thanks for all your help so far.

[sjkp]

ebekker/ACMESharp#119 provides some extra insight

[sjkp]

@paulirwin I don't think so.

But I was actually able to get hold of some of the people responsible for azure app services at Microsoft, so hopefully they will provide a solution.

[ptaffet]

We updated our certificate on 3/26 and are having this issue as well. ssllabs.com shows that the server is including the X1 and the X3 certificates. Firefox is the only browser that is giving an error though.

[sjkp]

@ptaffet unfortunately it is not only firefox, mobile browser might also get the warning, i know my android do get

[ComputerCowboy]

I have the exact same issue on Azure. I noticed it because of Android phones.
Is there any other viable solution? I don't have anything I can "roll back" to

[sjkp]

@ComputerCowboy atm I have a dialog with MS about a fix. But until then, the only alternative is to buy a certificate :/

[ComputerCowboy]

Thanks for letting us know
Please do post when there is any further news.

And keep up the great work, this has all been a joy until this hiccup

[ComputerCowboy]

That's a crazy quagmire.

Can you please mention to MS people that we'd love for them to be a little more in sync with the future releases of Let's Encrypt certs so that we don't have similar issues in the future?

[paulirwin]

Well unfortunately my prior (paid) certificate is expiring tomorrow so I have to pay for a renewal for another year. Hopefully come this time next year this will all be sorted out :-)

[ComputerCowboy]

Just to be clear, is there is no way to pull all the correct intermediary stuff and put it in a PFX that will serve correctly?

I don't mind doing a little work to get one or two of these X3 certs working

[sjkp]

@ComputerCowboy as of right now there nothing we can do to get it working on Azure App Services, as we don't have the proper permissions on the machines - and my confident in the MS guys to come up with a solution is not that high :(
@paulirwin I used a free 3 months cert from comodo, it only works on www.domain and domain though so if you need other subdomains you are out of luck - hopefully 3 months is enough for Microsoft.
https://www.instantssl.com/free-ssl-certificate.html?track=3311&key5sk0=2128&key5sk1=94f0794907793867e4a61a31f769047c7c95853c

[acekyd]

@sjkp Using the certreq.exe on windows following this link https://azure.microsoft.com/en-us/documentation/articles/web-sites-configure-ssl-certificate/#bkmk_certreq, i got a crt file from the ssl site instead of a .cer, how can i go around this?

[paulirwin]

@acekyd Are you referring to Let's Encrypt, or SSL certificates on Azure in general?

[acekyd]

@paulirwin I'm referring to the link @sjkp mentioned for the 90 days trial ssl -> https://www.instantssl.com/free-ssl-certificate.html?track=3311&key5sk0=2128&key5sk1=94f0794907793867e4a61a31f769047c7c95853c

I tried that and got .crt files instead... I installed and tried using certmgr.msc on windows to convert into .pfx for upload but that hasnt been possible. Seems the certificate installed doesnt have a private key. I followed this instructions for creating the .csr https://azure.microsoft.com/en-us/documentation/articles/web-sites-configure-ssl-certificate/#bkmk_certreq

[sjkp]

Oh I created a pfx with openssl. I can post the commands when I get home in a few hours.

[paulirwin]

@acekyd .crt and .cer files are the same, and you're correct, they do not have a private key on purpose. The private key was created on your machine as part of the CSR process. You need to complete the CSR with the .crt file. See step #6 on the azure link you sent.

[paulirwin]

@sjkp I too prefer OpenSSL, it is much more straightforward than doing it through the Windows certificate store. Here's the link I use as my "cheat sheet": https://www.sslshopper.com/article-most-common-openssl-commands.html

[acekyd]

Oh... Thanks @paulirwin I would check that out as well... Meanwhile I'll take a break as well while also waiting for @sjkp to post the commands as well ☺

[sjkp]

To convert a cer/crt file into pfx run this (you need openssl)

openssl pkcs12 -export -in mydomain.crt -inkey mydomain.key -out mydomain.pfx

Unless you generated the csr (the certificate signing request) with openssl aswell you probably don't have a mydomain.key file. To make that with openssl use this command

openssl genrsa -des3 -out mydomain.key 2048

[zappod]

Has there been any progress with MS on this?

[sjkp]

@zappod not really got a new date. X1 certs should be removed before friday the 22nd of April. But I dont know if they really tested if this fixes anything, or they are just removing them hoping that it will fix stuff.

[zappod]

@sjkp Thanks for the quick update. To bad MS isn't as good ;-)

[paulirwin]

@sjkp So I guess they aren't concerned at all about crippling existing websites by removing the X1 cert?

Is this going to be a problem again if LE has to fall back to the X4 intermediate?

What about when they launch ECDSA certs later this year?

[sjkp]

@paulirwin - I think that is a small price to pay in order to have it work going forward. But yes it is pretty annoying that there is not a better fix.

My hope is that letsencrypt is not gonna do this again, considering how much trouble it has caused on windows servers (not just azure).

[sjkp]

I can confirm that my site schdo.com is working now with x3 certs - so Microsoft must have done their thing on my server. Can somebody else confirm that their site is working with x3 certs or at least is broken with the old x1 cert that now should be blacklisted.
https://www.ssllabs.com/ssltest/analyze.html?d=schdo.com

[sjkp]

Interesting enough my sites on the old x1 cert seems to still be working.

[zappod]

Still getting a B here...

[flgarrid]

Me too :( Still having the X1 cert on my website

[ComputerCowboy]

I can see my sites on my phone now, that's pretty cool

[zappod]

@flgarrid Which region are your apps at? I'm on the West-Europe region..

[theoscat]

Hi there,

I get the same problem. I'm on the West-Europe region too.
Any info on that topic ?

@sjkp Great work there btw !

[zappod]

@hansarnevartdal Which region are your apps inn?

@theoscat They probably haven't done the fix in all regions yet.. I'm hoping it won't take to long.

[gazab]

I have my site in North-Europe and it didn't work before I restarted it. But now it seems to work fine!

Great job everyone involved!

[hansarnevartdal]

@zappod West-Europe for my stuff. I am running on the S0 and S1 pricing tier, not sure if it might be different update paths within the same region?

[theoscat]

@sjkp hi, do you have some news ?

[sjkp]

@theoscat nothing new. This is the last status i got from MS

We will blacklist the X1 intermediate certificate as part of our May OS patch deployment. The end result is that once the OS patch deployment completes, only the X3 intermediate cert. will be recognized.

Rough ETA for the time frame of the May OS patch deployment is sometime during May 9th - May 20th.

I will try to send and email to all domain owners that have X1 certs in production on azure over the weekend (obviously I dont have peoples email, so it will be sent to the hostmaster of the domain) to let them know.

[ComputerCowboy]

All my certs suddenly started working a week or two ago.
Not sure how that came to pass

[sjkp]

@ComputerCowboy they tested it on a scale unit that I had a site on. You were probably lucky that you were on the same one :)

[ComputerCowboy]

I thought you were in the EU?
My stuff is located on the South Central US datacenter

[sjkp]

@ComputerCowboy I am - so that is strange - maybe they patched it for some other reason with the same stuff.

[ComputerCowboy]

@sjkp I am quite pleased with your app by the way, other than the cert issue everything has been great. It could be even better though. I have looked at your code, I actually "borrowed" some of your ideas to aid with centralized credentials management for my web apps. At any rate, I have a lot of apps, and I don't really want a web job for each one. I have some designs on how to make this work. Are you interested in working something like that into your project? I am going to do it for myself regardless. The other issue I face is accidentally wiping out webjobs, not just yours but others. I have lots of slots and continuous deployment going on just about everything. So I have some solutions for that also.

I'll stop now, I just realized how OT this post is, and I have pressing issues to mend today anyhow.

Point is, you've made my life better with your code, and for that I thank you.

[sjkp]

@ComputerCowboy I will gladly accept pull request, or just even just code bits and I can incorporate them into the extension, if you don't feel like going through the whole testing hassle. Shoot me an email ([email protected]) or lets use #22 and #43 for your ideas.

[ComputerCowboy]

@sjkp I will be in touch, not immediately but it'll happen, because ultimately I need this solution to be less distributed. If anything I'll have more apps in the future, definitely not less.

[matrix4123]

Any update from MS on when the updates are rolling out to server?

[zappod]

It's update on my server (Western-Europe) now.

[flgarrid]

Still not working on North Central US

[sjkp]

I will take a little while, the service window for the update is two weeks. My guess is they take a region at a time, and monitor that everything works as expected. I don't know which order they take the region. But great to hear that they started the process @zappod

[sjkp]

And btw yesterday I sent an email to hostmaster@[domain-of-azurewebsite-using-X1-issue-letsencrypt-cert] to all azure website owners that is still running certificates issue by the X1 intermediate.

Unfortunately I don't have peoples email, so I had to find websites hosted in azure, by iterating all hostnames, from the certificate transparancy list for the Lets Encrypt certificates.

Because I sent them to hostmaster@[domain-of-azurewebsite-using-X1-issue-letsencrypt-cert] a lot of the emails bounced, so some people will see their site broken in the coming weeks.

[Jabe]

I've just reissued my X1 cert in West Europe preemptively and the new X3 is still being served with the X1 intermediate, so this is not fixed for everyone. Had to manually bind the old X1 for the site to work again... for the next 5 weeks. I wonder what will happen when the X1 intermediate is blacklisted, maybe the environment will pick up the X3 with the complete chain automatically?... Here be dragons.

[ptaffet]

I'm in South Central US and I'm still having the issue.

[merijndejonge]

My services hosted in North Europe still suffer from this problem. Is there anything I need to do to get my certificates validated correctly or is it just a matter of waiting and patience?

[sjkp]

Unfortunately it is a matter of patience as MS is taking their sweet time.

In fact I was just inform that it might not be done until end of June for all scale units.
The reason why it is working for some people is, that if you are lucky you can land on a scale unit without X1 certificate, and then it will work. It can also be that MS moves your site during maintenance to another scale unit, without X1 and then it will work too.

Unfortunately there is no way to select your scale unit (that I know of). You can see the name of the unit by a simple nslookup

nslookup schdo.azurewebsites.net

For that site the scale unit is named: waws-prod-db3-017.cloudapp.net

[chougaard]

Fixed.. Works now:-) Microsoft sis their magic

[chougaard]

*did

[phl3x0r]

I can confirm this is working for me too in North Europe as of a few days ago.

[sjkp]

Thanks for the confirmations that it is working again. I will close this bug, but thanks for the help and patience.

[phl3x0r]

Velbekomme, Det var så lidt. :-)
Jeg er rigtig glad for dit arbejde med letsencrypt. Tommel op!

Mvh.
Niklas

On Fri, Jun 24, 2016 at 8:32 AM, Simon J.K. Pedersen <
[email protected]> wrote:

Thanks for the confirmations that it is working again. I will close this
bug, but thanks for the help and patience.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#42 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/ARf6RScl4IE9-fYsXIsaY5xkrOkLc44Rks5qO3nwgaJpZM4H5pb4
.