pem_to_der functions break handshake on error about lua-resty-core

[agentzh]

@hamishforbes Thanks for the report!

Will you contribute a patch for it? It seems that we just need to clear the cached SSL error in OpenSSL by calling ngx_ssl_clear_error() on the C land.

[hamishforbes]

Hmm, ok this might be pushing the boundaries of my C and nginx internals knowledge.

It looks like ngx_ssl_clear_error() needs the current current request.
There was just recently a change to remove that requirement from the pem_to_der functions and it doesn't make sense that you'd need the request to convert certs.

How do you feel about having a separate function on the C side to clear the ssl error from the current request, e.g. ngx_http_lua_ffi_ssl_clear_error()
Then on the Lua side in resty-core we can check if the pem_to_der function has been called in a request or at init and call ngx_http_lua_ffi_ssl_clear_error() if required?

[hamishforbes]

Oh actually, looking closer ngx_ssl_clear_error only needs the current request so it can write to the debug log.

Should we actually just directly call the OpenSSL ERR_clear_error function instead?

I've tested that out on my side and it fixes my test case.

[agentzh]

@hamishforbes We don't need the logging part. We can duplicate the body of ngx_ssl_clear_error without logging in the C implementation of the pem_to_der() function and alike.

[hamishforbes]

Hi,
I submitted a PR to the ngx_lua repo so we can probably close this issue, unless you'd like tests for the problem on resty-core as well?