SSL: CERTIFICATE_VERIFY_FAILED : PyKmip client debugging options about pykmip

[PeterHamilton]

Hi @bhuvan-rk, thanks for filing this issue.

I believe your problem may be related to this warning:

WARNING:kmip.core.config_helper:Config file ['~/.pykmip/pykmip.conf', '/etc/pykmip/pykmip.conf', '/Library/Python/2.7/site-packages/PyKMIP-0.6.0-py2.7.egg/kmip/pykmip.conf', '/Library/Python/2.7/site-packages/PyKMIP-0.6.0-py2.7.egg/kmip/kmipconfig.ini'] not found

This indicates that the client could not find a valid configuration file to pull server connection parameters from. This is fine if you are manually providing these parameters to the client in your test_client.py script. However, if you are not providing those parameters, the client won't be able to connect to the server.

To test this, I removed my own local configuration file and then ran the client and I got the same error you did:

ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed

The README has more information on the client configuration file:

https://github.com/OpenKMIP/PyKMIP#configuration

An example of the client configuration file can be found below; just copy that file to one of the expected configuration file locations (I recommend /etc/pykmip/) and fill in your client/server connection information.

https://github.com/OpenKMIP/PyKMIP/blob/master/examples/pykmip.conf

If this doesn't solve your problem please let me know so we can continue to debug this.

[bhuvan-rk]

Hi Peter,

Thanks for your prompt response.

Do we need to necessarily give these values in a configuration file only ? Or can they be given as parameters during the KMIPProxy call ?

As you can see from the trace-back I've added earlier, I have indeed provided all the certs and keys as parameters to the KMIPproxy call and I've added a debug print to find out what the client variable is taking.

client = KMIPProxy(host='10.129.130.65',port=5696,certfile='/testcert.pem',keyfile='/server.key',ca_certs='~/server-ca.cer')

attrs = vars(client)

print ', '.join("%s: %s" % item for item in attrs.items())

From this debug print :

username: None, conformance_clauses: [<ConformanceClause.DISCOVER_VERSIONS: 1>], certfile: ~/testcert.pem, timeout: 30, host_list_str: 10.129.130.65, keyfile: ~/server.key, do_handshake_on_connect: True, host_list: ['10.129.130.65'], ssl_version: 2, authentication_suites: [<AuthenticationSuite.BASIC: 1>, <AuthenticationSuite.TLS12: 2>], host: 10.129.130.65, batch_items: [], credential_factory: <kmip.core.factories.credentials.CredentialFactory object at 0x1098dfc50>, cert_reqs: 2, ca_certs: ~/server-ca.cer, logger: <logging.Logger object at 0x1098dfb90>, password: None, config: client, port: 5696, suppress_ragged_eofs: True, socket: None

Can you kindly clarify ?

Thanks

[bhuvan-rk]

I just tried the config file option instead of the dynamic-client parameters route. Now I'm getting a totally different traceback :

client.open()

File "/Library/Python/2.7/site-packages/PyKMIP-0.6.0-py2.7.egg/kmip/services/kmip_client.py", line 225, in open
self._create_socket(sock)
File "/Library/Python/2.7/site-packages/PyKMIP-0.6.0-py2.7.egg/kmip/services/kmip_client.py", line 250, in _create_socket
suppress_ragged_eofs=self.suppress_ragged_eofs)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 911, in wrap_socket
ciphers=ciphers)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 520, in init
self._context.load_verify_locations(ca_certs)
IOError: [Errno 2] No such file or directory

[bhuvan-rk]

Just got past the IOError. I had to set this one :

export REQUESTS_CA_BUNDLE=~/server-ca.cer

Now I'm back home to the CERITIFICATE_VERIFY_FAILED error. Feels comforting.

[PeterHamilton]

Sorry @bhuvan-rk, I've been away for the Labor Day weekend.

To answer your first question, you can use the configuration file or manually provide the settings with the KMIPProxy constructor. Note that if you use both, the settings provided with the constructor take priority.

Regarding the certificate verification failure, I've seen a couple issues in the past that could be related. The first is a permissions issue, where the permissions on the certificate/key/CA files themselves are blocking your Python process from reading and using the certificate data. Double check that you can open and read each of those files from Python to make sure this isn't the problem.

The second is a settings issue, where the settings on your KMIP server are not configured to match the certificate you are trying to use. Contact your server admin and try to determine if the CA file used by the server is valid for the client certificate you are using. If it is not, the server will not be able to verify your certificate and the TLS connection will fail.

In my experience, TLS certificate debugging is consistently the most troublesome aspect of setting up PyKMIP for new users, since so many subtle things can go wrong. I've thought about setting up a TLS-specific FAQ to help users through this process. Would you find that useful?

[bhuvan-rk]

Hi Peter,

Thanks once again for your response.

The key and cert files are most certainly accessible from within python. I just added an os.access() call to verify this just in case. Thanks for that suggestion.

Now that the client-side is ruled out, I'm guessing its most probably the latter, the KMIP server not being configured correctly. So I'll start digging there.

FAQ for TLS certificate debugging sounds like an excellent idea. I'm happy to pitch-in. How can I help ?

[bhuvan-rk]

Oh I forgot to add. Initially I was using tcpdump on both the client and the server side to understand which part of the handshake was failing.

And probably you guys have it documented some place ?

One of my main questions was this : " How can I enable logging ? " or how to debug the PyKmip client.

A kind soul helped me add the following lines and see stuff :

try:
import http.client as http_client
except ImportError:
import httplib as http_client

http_client.HTTPConnection.debuglevel = 1
logging.basicConfig()
logging.getLogger().setLevel(logging.DEBUG)

[PeterHamilton]

Yep, there's no configuration setting to enable debug logging; it has to be done in Python, either within the PyKMIP library itself or the calling application. Be sure to disable debug logging once you've fixed the problem. The client will dump every KMIP request/response it sends/receives to the debug log, which can include sensitive information.

For the FAQ, I'm thinking it will have sections for each of the different TLS errors I've seen over the years and what possible solutions are to each. In this case, all I need from you is whatever steps you took to resolve the error. If that ends up being mainly a server-side configuration issue, I don't need the exact steps (since different vendors have different products and configuration mechanisms), just a high-level idea of what was wrong and what needed to be changed to make the client work.

[bhuvan-rk]

Sounds good. For starters how about this :

If you run into IOError make sure you set the REQUESTS_CA_BUNDLE env variable to point to your server certificate.

Traceback and debugs can be found in a previous reply to this same thread.

export REQUESTS_CA_BUNDLE=~/server-ca.cer

File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/ssl.py", line 520, in init
self._context.load_verify_locations(ca_certs)
IOError: [Errno 2] No such file or directory

[PeterHamilton]

Hi @bhuvan-rk, it's been about a week since our last chat. Did you determine if the certificate error was caused by a server-side configuration issue?

[PeterHamilton]

Hi @bhuvan-rk, I'm going to close this issue for now. Feel free to comment further or file a new issue if there's additional debug information that should be added here. Thanks!