EVP_VerifyFinal() failed with EC cert on 0.6.11 about pam_pkcs11

[lgarbarini]

0001-hack-to-test-ECDSA-sigs.patch.txt

This seems to be an issue with the ASN1 encoding of the ECDSA signature. I hacked together some code by following the information in How can I convert a DER ECDSA signature to ASN.1 and can now successfully verify ECDSA signatures.

I'm unfamiliar with these OpenSSL functions so I was having trouble understanding the original code, but one of the primary issues seems to be "Signed big-endian encoding of minimal length", from the stackexchange post:

"Signed big-endian encoding of minimal length" means that the numerical value must be encoded as a sequence of bytes, such that the least significant byte comes last (that's what "big endian" means), the total length is the shortest possible to represent the value (that's "minimal length"), and the first bit of the first byte specifies the sign of the value (that's "signed"). For ECDSA, the r and s values are positive integers, so the first bit of the first byte must be a 0; i.e. the first byte of (vr) (respectively (vs)) must have a value between 0x00 and 0x7F.

This is my first foray into pam_pkcs11 so I'll probably have to take some time to write a clean pull request. But for now, here is the "hacked" part of verify_signature():

  if (EVP_PKEY_base_id(pubkey) == EVP_PKEY_EC) {
    rs_len = *signature_length / 2;
    `unsigned` char* old = *signature;
    // memcpy(, old+rs_len, rs_len);

    int extend_r = 0;
    int extend_s = 0;

    // first byte of (vr) would be read as negative number, remember to pad
    if ((unsigned char)*old > 127) {
      extend_r = 1;
    }
    // first byte of (vs) would be read as negative number, remember to pad
    if (((unsigned char)*(old + rs_len)) > 127) {
      extend_s = 1;
    }

    // new signature length is 0x30 b1 0x02 b2 (vr) 0x02 b3 (vs), plus padding 
    *signature_length = (rs_len * 2) + 6 + extend_r + extend_s;

    // use calloc to handle 0 padding
    *signature = calloc(*signature_length, sizeof(char*));
    void * ptr = *signature;
    DBG1("length is: %d",(*signature_length));
    memset(ptr, 0x30, 1);
    
    // single byte length of all fields after this one
    ptr += 1;
    memset(ptr, *signature_length - 2, 1);

    // marker
    ptr += 1;
    memset(ptr, 0x02, 1);

    // length of (vr), include padding (if required)
    ptr += 1;
    memset(ptr, rs_len + extend_r, 1);

    // vr (padding handled)
    ptr += 1 + extend_r;
    memcpy(ptr, old, rs_len);
    
    // marker
    ptr += rs_len;
    memset(ptr, 0x02, 1);
    
    // length of (vs), include padding (if required)
    ptr += 1;
    memset(ptr, rs_len + extend_s, 1);
    
    // vs (padding handled)
    ptr += 1 + extend_s;
    memcpy(ptr, old+rs_len, rs_len);
}

[thyagarajan-balakrishnan]

I have the same problem and I have EC key secp521r1 in my token. This patch didn't work as is but tweaking it a bit did the trick.

*signature_length = (rs_len * 2) + 6 + extend_r + extend_s + 1;
....
memset(ptr, 0x30, 1);

// set 0x81 after 0x30 - without this for EC521 key verification fails
ptr += 1;
memset(ptr, 0x81, 1);

ptr += 1;
memset(ptr, *signature_length - 3, 1);

[popovec]

IMHO this issue is already fixed by pull request #51 .. (try the 75fe90e instead of the patch listed above)

[thyagarajan-balakrishnan]

IMHO this issue is already fixed by pull request #51 .. (try the 75fe90e instead of the patch listed above)

Tested #51, and it works for EC-521. Thanks. 👍

[wolneykien]

#51 merged into master.