Problem reading nested Octet Strings about bc-csharp

[peterdettman]

So the problem turned out to be that the selector's SubjectKeyIdentifier is not being set appropriately by the CMS recipients (KeyAgreeRecipientInformation and KeyTransRecipientInformation).

It might be a good idea to change the way AuthorityKeyIdentifier and SubjectKeyIdentifier are set on X509CertStoreSelector, but for the moment I've just fixed the recipients.

[peterdettman]

@jstedfast This issue affects selection of CMS recipients (KeyAgree/KeyTrans) that are specifying SubjectKeyIdentifier (instead of the more common IssuerAndSerialNumber). I am a bit surprised that we have no test coverage of that option between BouncyCastle (even bc-java) and MimeKit, so we probably should add some.

[peterdettman]

MimeKit sets the SubjectKeyIdentifier incorrectly in RsaOaepAwareRecipientInfoGenerator.Generate (also in CmsEnvelopeAddEllipticCurve method just below there). That was "cancelling out" this issue in BouncyCastle.

After updating MimeKit to use 2.4.0-beta.61, then fixing those two places in MimeKit, things go back to working (but now would be correct for third-party messages - which our test suites ought to incorporate examples of).

Edit: @ggrote presumably saw the error because he was decrypting something with MimeKit that wasn't generated by MimeKit.

[peterdettman]

@peterdettman what are your plans for releasing 2.4.0? I didn't realize that 2.3.1 fixed some security issues and so I'm getting requests to make a release with >= 2.3.1.

@jstedfast I'm planning to release 2.4.0 this coming weekend.

[cipherboy]

@ggrote Do you have an example? When I tried to reproduce this with:

using System;
using System.Collections.Generic;
using System.IO;
using System.Text;

using Org.BouncyCastle.Asn1;
using Org.BouncyCastle.Asn1.X509;
using Org.BouncyCastle.OpenSsl;
using Org.BouncyCastle.Utilities.Encoders;
using Org.BouncyCastle.X509;
using Org.BouncyCastle.X509.Extension;


var sr = File.OpenText("cert.pem");
var pr = new PemReader(sr);

var o = pr.ReadPemObject();
var cert = new X509Certificate(o.Content);

var e = cert.GetExtensionValue(X509Extensions.SubjectKeyIdentifier);
Console.WriteLine("e=" + Hex.ToHexString(e.GetOctets()));

I got:

$ dotnet run
e=0414b18deae423a77e098eb5ee31e06add9e343765ac

Where e here seems to be the correct encoding of this parsed octet value.

[ggrote]

I think you got the same problem with your example.
The subject identifier should be:
b18deae423a77e098eb5ee31e06add9e343765ac
The 0414 at the beginning marks another octet string (0x04) with the length 0x14 inside the octet string for the extension. Just have a look in the pen with openssl or any other tool. Or just create the sha1 hash of the public key which should be the subject identifier.

[peterdettman]

@ggrote I don't see any bug, you just aren't parsing the extension value.

An Extension has a value which is an OCTET STRING. In your example, the OCTET STRING is "041406a9...", encoded as "0416041406a9..." in the ASN.1. In the case of SubjectKeyIdentifier, that value is interpreted as an encoding of an ("inner") OCTET STRING. So to process this extension correctly, you need to actually parse the contents octets of the value OCTET STRING, which would give you back the OCTET STRING "06a9...".

Anyway, we have a convenience method for this type of thing, so just try:

cert.GetX509Extensions()?.GetExtensionParsedValue(X509Extensions.SubjectKeyIdentifier)

and you should get back the Asn1OctetString that you expect.

[ggrote]

Okay got it.
We faced the problem using MimeKit for decrypting S/Mime Mails which uses BouncyCastle.
The Implementation is as follows:

protected override AsymmetricKeyParameter GetPrivateKey (ISelector<X509Certificate> selector)
{
	foreach (var certificate in certificates) {
		var fingerprint = certificate.GetFingerprint ();

		if (!keys.TryGetValue (fingerprint, out var key))
			continue;

		if (selector != null && !selector.Match (certificate))
			continue;

		return key;
	}

	return null;
}

Which uses the X509CertStoreSelectors Match function in which the SubjectKeyIdentifier is compared:

if (!MatchExtension(subjectKeyIdentifier, c, X509Extensions.SubjectKeyIdentifier))
				return false;

private static bool MatchExtension(byte[] b, X509Certificate c, DerObjectIdentifier oid)
		{
			if (b == null)
				return true;
			Asn1OctetString extVal = c.GetExtensionValue(oid);
			if (extVal == null)
				return false;
			return Arrays.AreEqual(b, extVal.GetOctets());
		}

Shouldn't this be cert.GetX509Extensions()?.GetExtensionParsedValue(X509Extensions.SubjectKeyIdentifier) like you said?
Because now the parsed value from the incoming Mail is compared to the unparsed value from the certificate.

I think the Java Version gives two values. One Raw Value and one parsed Value.

I'm sorry but the code formatting doesn't like me.

---

@cipherboy edited for formatting.

[ggrote]

Any news about it?

[peterdettman]

I agree that there seems to be a problem in the selector. This code was all ported from bc-java so I think the confusion arises partly because of the behaviour of the JDK X509CertSelector class.

[ggrote]

Okay maybe there is a problem too. In my opinion the selector should compare the inner most octet string with the given value, as this could be the only option where it could match. Otherwise this whole selector can not work.

[ggrote]

Thank you!

[peterdettman]

@jstedfast I've published 2.4.0-beta.61 to allow testing of the fix.

[peterdettman]

@jstedfast With the "fix" I now get errors from MimeKit tests, and I have to take back the claim that you don't have test coverage for this case. I'm a bit confused how things used to work and why they break now though; needs more investigation.

[jstedfast]

@peterdettman I'll try to look into the test failures this weekend. Is the published fix on nuget.org?

[jstedfast]

Hmmm, odd, I installed the 2.4.0-beta.61 version and MimeKit UnitTests all passed?

[jstedfast]

Added more tests for this and now I'm getting failures for SubjectKeyIdentifier but not IssuerAndSerialNumber (using 2.4.0-beta.61). I'll check v2.3.1 as soon as the meeting I'm in is over.

[jstedfast]

all tests pass on v2.3.1

[jstedfast]

Ah, got it.

[ggrote]

Edit: @ggrote presumably saw the error because he was decrypting something with MimeKit that wasn't generated by MimeKit.
I can't 100% confirm, but I'm pretty sure the service provider we got the mail from does not use MimeKit.

[jstedfast]

The commit above depends on BouncyCastle 2.4.0-beta.61

@peterdettman What I ended up doing is writing unit tests for MimeKit that would encrypt using the System.Security backend and then verifying that the BouncyCastle backend could decrypt it. Likewise, I also added a unit test that would encrypt using the BouncyCastle backend and verifying that the System.Security backend could decrypt it.

Both tests try IssuerAndSerialNumber and SubjectKeyIdentifier so that we make sure that both work in both directions.

[jstedfast]

@peterdettman what are your plans for releasing 2.4.0? I didn't realize that 2.3.1 fixed some security issues and so I'm getting requests to make a release with >= 2.3.1.