alexdean / as2 Goto Github PK

summary

AS2 1.1 added support for message compression. We should detect compressed message bodies and properly re-inflate them.

general process for creating compressed SMIME messages is in https://www.rfc-editor.org/rfc/rfc3851#section-3.5.

Step 1. The MIME entity to be compressed is prepared according to section 3.1.

Step 2. The MIME entity and other required data is processed into a
CMS object of type CompressedData.

Step 3. The CompressedData object is wrapped in a CMS ContentInfo
object.

Step 4. The ContentInfo object is inserted into an
application/pkcs7-mime MIME entity.

The smime-type parameter for compressed-only messages is
"compressed-data". The file extension for this type of message is
".p7z".

implementation items

• in a message body, look for Content-Type with smime-type=compressed-data parameter
• compression is zlib wrapped in a CMS ContentInfo package. detailed in https://www.rfc-editor.org/rfc/rfc3274.
• ruby openssl does not have direct support for reading this data type, so we have to parse the raw ASN1 directly.
  • relevant ASN1 schemas are in:
    • https://www.rfc-editor.org/rfc/rfc3274#ref-ASN1
    • https://www.rfc-editor.org/rfc/rfc3852#section-12.1
• Need to notice Content-Transfer-Encoding at each level and add base64-decoding as necessary.
• "It is possible to apply any of the signing, encrypting, and compressing operations in any order." reference
  • So the order operations can't be assumed, and we'll need to be sure to investigate the MIME headers at each layer to apply the correct transformations.
  • This may also affect how/when we apply signature verification as well. (could be before or after decompression.)

message = As2::Message.new(...)
puts message.attachment.raw_source

# Content-Type: application/pkcs7-mime; name="smime.p7z"; smime-type=compressed-data
# Content-Transfer-Encoding: base64
# Content-Disposition: attachment; filename="smime.p7z"
# Content-Description: S/MIME Compressed Message

We need to notice smime-type=compressed-data and apply decompression. The message body will be binary ASN1.

# rough attempt to make ASN1 a little easier to access
def build_concise_asn1(item)
  if item.respond_to?(:value)
    item_value = item.value
  else
    item_value = item
  end

  if item.respond_to?(:each)
    out_value = []
    # OpenSSL::ASN1::Sequence responds to .each
    item.each { |i| out_value << build_concise_asn1(i) }
  elsif item_value.respond_to?(:each)
    out_value = []
    # OpenSSL::ASN1::ASN1Data does not respond to .each
    # but it's .value may be an array so we should recurse
    item_value.each { |i| out_value << build_concise_asn1(i) }
  else
    # when we hit a leaf node
    if item.is_a?(OpenSSL::ASN1::Integer)
      out_value = item_value.to_i
    elsif item.is_a?(OpenSSL::ASN1::ObjectId)
      out_value = {oid:item.oid, value:item_value}
    else
      out_value = item_value
    end
  end

  out_value
end

# receiving a test message from Mendelson with compression on & using `Content-Transfer-Encoding: base64`...
asn1 = OpenSSL::ASN1.decode(Base64.decode64(message.attachment.body.raw_source))

simpler = build_concise_asn1(asn1)
# => [{:oid=>"1.2.840.113549.1.9.16.1.9", :value=>"id-smime-ct-compressedData"},
# =>  [[0,
# =>    [{:oid=>"1.2.840.113549.1.9.16.3.8", :value=>"ZLIB"}],
# =>    [{:oid=>"1.2.840.113549.1.7.1", :value=>"pkcs7-data"},
# =>     [["x\x9CE\x8C\xCB\n\x82@\x14@\xF7\x82\xFF\xE0\x0FX$\x12d\xB4\xF1\x81H \b\x81\x93\xBB\xEBx\xD5\x01\xBD38\x97\xCA\xBE>[D\xDB\xC39'\xD1\xC4H\xEC\xDFV\x83\x91\a\xC6LJ\x02+M\xFB,-|q\b\\'\xF9)\v\x90\xEDq\xF13\x92\xBAS4D^\v\x16\x8F\xE1_I\x955\xDA\xAAo\xBF\xCD\x98A\x8E\xF3\xC6\xCF^\xAF&$\x98\xF1\xC2hy\xC7/v\x1D\xD7i\x83\xD3S\xAE\xB1)\xF2\xF2\xD1\x8AxmD\xF9n\xEAj\x00Q\r\xF7\xBA\e\xA1\x0E\xAF\x1F\x9C,7\xC7"]]]]]]

# this path to the actual data should always be the same, but need to check ASN1 schemas carefully to be sure.
Zlib::Inflate.inflate simpler[1][0][2][1][0].join
#=> "Content-Type: application/EDI-X12\r\nContent-Transfer-Encoding: base64\r\nContent-Disposition: attachment; filename=test.txt\r\n\r\nb29wcyBpIGNvbXByZXNzZWQgaXQgYWdhaW4K"

Have noticed the innermost OpenSSL::ASN1::Constructive which contains the actual compressed data can either be a single OpenSSL::ASN1::OctetString, or (in the case of IBM Sterling) can be an array of multiple OpenSSL::ASN1::OctetString instances. The final join in the code example seems to handle either case correctly, but we should scrutinize the ASN1 schemas closely to make sure we have all permutations covered.

references

• https://letsencrypt.org/docs/a-warm-welcome-to-asn1-and-der/
• http://luca.ntop.org/Teaching/Appunti/asn1.html
• https://www.ietf.org/proceedings/64/IDs/draft-ietf-ediint-compression-05.txt

Partner may send us a header like

# pyas2
disposition-notification-options: signed-receipt-protocol=required, pkcs7-signature; signed-receipt-micalg=optional, sha1

# mendelson
disposition-notification-options: signed-receipt-protocol=optional, pkcs7-signature; signed-receipt-micalg=optional, sha-256

# opentext
Disposition-Notification-Options: signed-receipt-protocol=optional,pkcs7-signature; signed-receipt-micalg=optional, sha1, md5

meaning (in first example): partner wants us to calculate the MIC using the sha1 algorithm.

we should attempt to detect this & honor the request when possible. currently we always use sha256, which causes message verification failures if the client requests a different algorithm.

non-critical change, since we would detect this mismatch during testing. but worth doing at some point to lessen headaches. (partner may not know what setting needs to be changed and will just report “we got a MIC failure” with no further details…)

remove the hack/fallback added in #11 once mikel/mail#1511 is released.

in #25 i added a new MDN format which does not use OpenSSL::PKCS7.write_smime.

One item i wasn't able to solve in that PR was how to specify the micalg parameter used in the Content-Type header. The workaround in that PR is to still call write_smime and use a regex to extract the micalg parameter. Would like to remove that hack when possible.

what we need to replicate

write_smime produces a header like this:

Content-Type: multipart/signed; protocol="application/x-pkcs7-signature"; micalg="sha-256"; boundary="--- FA974EDD77D0E708FEB7C0F3E6B14F13"

but i have not been able to figure out where/how it chooses the sha-256 value. it is unconnected to the certificate we use or the MIC alg we use in the MDN report we're signing.

looking for input

asked for help in https://stackoverflow.com/questions/75934159/how-does-openssl-smime-determine-micalg-parameter
should check back there periodically to look for an answer.