sethmlarson / truststore Goto Github PK

View Code? Open in Web Editor NEW

135.0 5.0 17.0 149 KB

Verify certificates using OS trust stores

Home Page: https://truststore.readthedocs.io

License: MIT License

Python 100.00%

certificates macos python tls truststore windows

truststore's Introduction

Truststore

Truststore is a library which exposes native system certificate stores (ie "trust stores") through an ssl.SSLContext-like API. This means that Python applications no longer need to rely on certifi as a root certificate store. Native system certificate stores have many helpful features compared to a static certificate bundle like certifi:

Automatically update certificates as new CAs are created and removed
Fetch missing intermediate certificates
Check certificates against certificate revocation lists (CRLs) to avoid monster-in-the-middle (MITM) attacks
Managed per-system rather than per-application by a operations/IT team
PyPI is no longer a CA distribution channel 🥳

Right now truststore is a stand-alone library that can be installed globally in your application to immediately take advantage of the benefits in Python 3.10+. Truststore has also been integrated into pip as an opt-in method for verifying HTTPS certificates with truststore instead of certifi.

Long-term the hope is to make truststore the default way to verify HTTPS certificates in pip and to add this functionality into Python itself. Wish us luck!

Installation

Truststore is installed from PyPI with pip:

$ python -m pip install truststore

Truststore requires Python 3.10 or later and supports the following platforms:

macOS 10.8+ via Security framework
Windows via CryptoAPI
Linux via OpenSSL

User Guide

Warning PLEASE READ: inject_into_ssl() must not be used by libraries or packages as it will cause issues on import time when integrated with other libraries. Libraries and packages should instead use truststore.SSLContext directly which is detailed below.

The inject_into_ssl() function is intended only for use in applications and scripts.

You can inject truststore into the standard library ssl module so the functionality is used by every library by default. To do so use the truststore.inject_into_ssl() function:

import truststore
truststore.inject_into_ssl()

# Automatically works with urllib3, requests, aiohttp, and more:
import urllib3
http = urllib3.PoolManager()
resp = http.request("GET", "https://example.com")

import aiohttp
http = aiohttp.ClientSession()
resp = await http.request("GET", "https://example.com")

import requests
resp = requests.get("https://example.com")

If you'd like finer-grained control or you're developing a library or package you can create your own truststore.SSLContext instance and use it anywhere you'd use an ssl.SSLContext:

import ssl
import truststore

ctx = truststore.SSLContext(ssl.PROTOCOL_TLS_CLIENT)

import urllib3
http = urllib3.PoolManager(ssl_context=ctx)
resp = http.request("GET", "https://example.com")

You can read more in the user guide in the documentation.

License

MIT

truststore's People

Contributors

Stargazers

Watchers

Forkers

guerda leogregianin pactortester adriangb yeraydiazdiaz battyone systemtruststores bellyfat kislyuk mgorny robocorp versionbayjc stefanor layday

truststore's Issues

In pip, "--use-feature truststore" in requirements.txt does not seem to work

Hi,
In my org, we have our own PyPI server with self-signed certs.
I have managed to use truststore successfully through pip on both Windows and Ubuntu through the command line option --use-feature truststore, like this :

pip install --extra-index-url https://[address-of-our-pypiserver] --use-feature truststore [some_package]

However, when I try the exact same thing using a requirements file like this:

requirements.txt

--extra-index-url https://[address-of-our-pypiserver]
--use-feature truststore

[some_package]

and: pip install -r requirements.txt

It does not work, and I get the unable to get local issuer certificate error.

pip seems to recognizes the feature because otherwise it raises an error when I type an unknown feature by pip.

I am not sure whether all this is a pip issue or a trustore issue, so I am posting this here.

Thanks a lot for your help !

Extra info:
python version: 3.10
pip version: 23.2.1
truststore version: 0.8.0
Windows 10, Ubuntu 20.04

Add type hints

Add type hints to all functions
Add mypy --strict SOURCE_FILES to a linting job

Add test case for async HTTPX client

Failing consistently on only macOS with the following error:

______________________ test_async_httpx_works_with_inject ______________________

cls = <class '_pytest.runner.CallInfo'>
func = <function call_runtest_hook.<locals>.<lambda> at 0x11222e320>
when = 'call'
reraise = (<class '_pytest.outcomes.Exit'>, <class 'KeyboardInterrupt'>)

    @classmethod
    def from_call(
        cls,
        func: "Callable[[], TResult]",
        when: "Literal['collect', 'setup', 'call', 'teardown']",
        reraise: Optional[
            Union[Type[BaseException], Tuple[Type[BaseException], ...]]
        ] = None,
    ) -> "CallInfo[TResult]":
        """Call func, wrapping the result in a CallInfo.
    
        :param func:
            The function to call. Called without arguments.
        :param when:
            The phase in which the function is called.
        :param reraise:
            Exception or exceptions that shall propagate if raised by the
            function, instead of being wrapped in the CallInfo.
        """
        excinfo = None
        start = timing.time()
        precise_start = timing.perf_counter()
        try:
>           result: Optional[TResult] = func()

.nox/test/lib/python3.10/site-packages/_pytest/runner.py:339: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
.nox/test/lib/python3.10/site-packages/_pytest/runner.py:260: in <lambda>
    lambda: ihook(item=item, **kwds), when=when, reraise=reraise
.nox/test/lib/python3.10/site-packages/pluggy/_hooks.py:265: in __call__
    return self._hookexec(self.name, self.get_hookimpls(), kwargs, firstresult)
.nox/test/lib/python3.10/site-packages/pluggy/_manager.py:80: in _hookexec
    return self._inner_hookexec(hook_name, methods, kwargs, firstresult)
.nox/test/lib/python3.10/site-packages/_pytest/unraisableexception.py:88: in pytest_runtest_call
    yield from unraisable_exception_runtest_hook()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

    def unraisable_exception_runtest_hook() -> Generator[None, None, None]:
        with catch_unraisable_exception() as cm:
            yield
            if cm.unraisable:
                if cm.unraisable.err_msg is not None:
                    err_msg = cm.unraisable.err_msg
                else:
                    err_msg = "Exception ignored in"
                msg = f"{err_msg}: {cm.unraisable.object!r}\n\n"
                msg += "".join(
                    traceback.format_exception(
                        cm.unraisable.exc_type,
                        cm.unraisable.exc_value,
                        cm.unraisable.exc_traceback,
                    )
                )
>               warnings.warn(pytest.PytestUnraisableExceptionWarning(msg))
E               pytest.PytestUnraisableExceptionWarning: Exception ignored in: <socket.socket fd=-1, family=AddressFamily.AF_INET6, type=SocketKind.SOCK_STREAM, proto=0>
E               
E               Traceback (most recent call last):
E                 File "/Users/runner/work/truststore/truststore/.nox/test/lib/python3.10/site-packages/anyio/_core/_eventloop.py", line 153, in get_asynclib
E                   return sys.modules[modulename]
E               KeyError: 'anyio._backends._asyncio'
E               
E               During handling of the above exception, another exception occurred:
E               
E               Traceback (most recent call last):
E                 File "/Users/runner/hostedtoolcache/Python/3.10.10/x64/lib/python3.10/ast.py", line 50, in parse
E                   return compile(source, filename, mode, flags,
E               ResourceWarning: unclosed <socket.socket fd=15, family=AddressFamily.AF_INET6, type=SocketKind.SOCK_STREAM, proto=0, laddr=('::1', 9999, 0, 0)>

.nox/test/lib/python3.10/site-packages/_pytest/unraisableexception.py:78: PytestUnraisableExceptionWarning

Did truststore work correctly for your operating system?

If you're here from the article and running pip with --use-feature=truststore worked on your machine then leave a reaction below based on your operating system. To avoid spam, please do not comment unless your operating system isn't represented below.

👍 (Windows)
😄 (macOS 10.x or earlier)
🚀 (macOS 11 or later)
👀 (Linux flavors like Ubuntu, Debian, Mint, CentOS, RedHat, Fedora, or similar derivatives)
❤️ (*BSD)
🎉 (Any Linux that uses LibreSSL)

If your operating system isn't represented in this list then you can comment below.

If truststore didn't work then please open an issue describing your environment and operating system.

Truststore not fetching missing certs in AWS ECS linux OS

Hi Guys,

I tested my service with truststore library in MacOS to fetch missing certs and it is working fine. However, when the service is deployed to AWS ECS with linux OS, it doesn't seem to work and still throws SSL CERT VERIFY FAILED error.
Do you know if there's other setting I need to enable?

truststore and certifi-win32 cause error stating file is already open

To reproduce:

Install the following on a Windows machine:

requests - latest
certifi
python-certifi-win32
truststore
python 3.10

Then run:

import requests

The error pops up stating that the file is already open by another program.

Add memray to CI

Since we're using ctypes we should run memray to ensure we're not leaking/regressing on memory.

Please package testsuite to the distribution tarball

I would like to fix packaging for OpenSUSE, but your release tarball does not contain the tests directory. Could you please distribute them as well?

Fix package description

Current PyPI page showing: "Verify certificates using OS trust stores. This is useful when your system contains". Apparently we only get one line for the module-level short description?

See: https://pypi.org/project/truststore/0.6.0/

Add assert for no warnings for integration test cases

Shouldn't be receiving any warnings when using urllib3 and aiohttp.

Make sure we pass through setting attributes to the underlying SSLContext

Add integration test using pip's --use-feature=truststore

Depends on pypa/pip#11082

We should ensure we don't break any of pip's users trying to use our experimental feature flag --use-feature=truststore, at least for a basic installation from PyPI. It's tougher for us to simulate installing from corporate proxies/package indices but maybe we can do something with mkcert and warehouse?

Client certs

@jacobian pointed out that it would be nice if truststore could also use client certs from the system trust store, particularly on devices that provide device identity by creating certs using a private key stored in a Trusted Platform Module. I suspect this may require actually using system APIs for encrypting the data stream and not just for cert validation though.

Does not work for openai package

python -m pip install openai --use-feature=truststore

Collecting openai
Using cached openai-0.26.5.tar.gz (55 kB)
Installing build dependencies: started
Installing build dependencies: finished with status 'error'
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> [8 lines of output]
WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)'))': /simple/setuptools/
WARNING: Retrying (Retry(total=3, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)'))': /simple/setuptools/
WARNING: Retrying (Retry(total=2, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)'))': /simple/setuptools/
WARNING: Retrying (Retry(total=1, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)'))': /simple/setuptools/
WARNING: Retrying (Retry(total=0, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)'))': /simple/setuptools/
Could not fetch URL https://pypi.org/simple/setuptools/: There was a problem confirming the ssl certificate: HTTPSConnectionPool(host='pypi.org', port=443): Max retries exceeded with url: /simple/setuptools/ (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:997)'))) - skipping
ERROR: Could not find a version that satisfies the requirement setuptools (from versions: none)
ERROR: No matching distribution found for setuptools
[end of output]

note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× pip subprocess to install build dependencies did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

Seems to not verify self-signed certs

Hello! 🙂

We normally use pip_system_certs, which makes Python automatically use the Windows cert store. My understanding is that truststore is supposed to provide the same functionality. If not, then maybe this bug report is bogus 😄

Anyway, I installed truststore and some packages, like so:

python -m venv ".venv"
.\.venv\Scripts\activate\

python -m pip install --upgrade pip
python -m pip install truststore --upgrade

python -m pip install robotframework --upgrade --use-feature=truststore
python -m pip install robotframework-requests --upgrade --use-feature=truststore
python -m pip install urllib3 --upgrade --use-feature=truststore
python -m pip install requests --upgrade --use-feature=truststore
python -m pip install RESTinstance --upgrade --use-feature=truststore
...

When we then make a call using robot-framework-requests, I get this error:

SSLError: HTTPSConnectionPool(host='localhost', port=44327): Max retries exceeded with url: /example (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:997)')))

OS: Windows
Python: 3.10.5

Let me know if there's any more info that I can provide! 🙂

[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate

Hello, could the source of this problem be that the python system does not use the certification repository? I am very new to this topic, I may be asking a silly question. Actually what I want to do is to login users to the 2nd website using GitLab as an OAuth between two websites but I get the following error. I created a certificate in Gitlab. I have Python 3.8 and pip 19.3 in my environment. How can I solve this problem?

thankyou for support.

An error occurred while attempting to login via your social network account.

HTTPSConnectionPool(host='gitlab.tutel', port=443): Max retries exceeded with url: /oauth/token
(Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed:
self-signed certificate (_ssl.c:997)')))

Better error handling and messages for ssl.SSLCertificateError

Have different messages for revoked certificates, weak certificates, etc.

Joining the dots with other languages

Hi Seth,

Thank you for your work on this - all of us who are using local HTTPS proxies, or who are working in corporate environments with TLS inspection, or have otherwise had to confront the 'certificate verify failed' error, are immensely grateful for it.

I'm working with @bnoordhuis on a Node version of this (https://github.com/bnoordhuis/node-native-certs), which takes the approach of using the Rust rustls-native-certs library to do the hard work. This also means we don't have to touch C for the native parts of the module - our glue code is all Rust - which makes cross platform packaging a lot easier. If you're interested, I have put together a little prototype of what using rustls-native-certs from Python could look like.

Secondly, with your work in Python, our work in Node, and the Rust team's foundational work on this in Rust, it's now clear that a pattern is emerging. People are waking up to the idea that even if a language is cross-platform, it should still be a good citizen of each platform, and get its certificates from the native trust store. To this end I've started making a list of languages and CLI tools which have native trust store integration here: https://native-certs.github.io. (Naturally truststore is featured in that list.) If you know of any others that I haven't thought of yet, please let me know!

Document how OS APIs map to Python's SSLContext APIs

Linux/BSD system trust

I suggest that you reconsider the load order for system trust store on Linux and BSD. OpenSSL prefers SSL_CERT_FILE / SSL_CERT_DIR env vars and then falls back to compile-time locations for cafile and capath. If you deviate from the default, then it may not only break applications that rely on the order. Linux vendors will patch and modify your code to re-establish default settings, too. You can get the defaults from Python's ssl module:

>>> ssl.get_default_verify_paths()
DefaultVerifyPaths(cafile='/etc/pki/tls/cert.pem', capath='/etc/pki/tls/certs', openssl_cafile_env='SSL_CERT_FILE', openssl_cafile='/etc/pki/tls/cert.pem', openssl_capath_env='SSL_CERT_DIR', openssl_capath='/etc/pki/tls/certs')

The cafile and capath attributes resolve the env vars.

For cafile, do

check for SSL_CERT_FILE, use it if the env var is set and don't search for additional locations. An empty or invalid SSL_CERT_FILE is
check the hard-coded value from OpenSSL, stop if the file exists (even when it is empty)
search common default paths

For capath, do

check for SSL_CERT_DIR
check the hard-coded value from OpenSSL is a valid directory with at least one file matching r"[0-9a-fA-F]{8}\.[0-9]"
search common default directories

You may even want to be more strict. If either SSL_CERT_FILE is set, SSL_CERT_DIR is set, default cafile exists, or default capath contains certs, then call load_default_certs() or set_default_verify_paths(). Only fall back to alternative search paths if the defaults don't work.

I also noticed that you are using an old, outdated list of cert file locations. I have seen the list before in other projects. It contains wrong paths (FreeBSD) or discontinued distros (OpenELEC). You can find an up to date list of paths for Linux at https://github.com/tiran/certifi-system-store/blob/main/src/certifi/core.py and some research at https://github.com/tiran/distro-truststore/.

Using PyOpenSSL SSLContexts for verifying certificate chains without an SSL connection

Hi,

I use functionality that is unique to PyOpenSSL contexts (OpenSSL.SSL.Context) to verify certificate chains without having an SSL connection open. PyOpenSSL's trust store support is very limited (see notes in https://www.pyopenssl.org/en/stable/api/ssl.html#OpenSSL.SSL.Context.set_default_verify_paths). Because of this, I've been falling back to certifi, but would prefer to make use of the functionality supported by truststore instead. Do you have any thoughts on the feasibility of this?

0.8.0: test_trustme_cert_loaded_via_capath fails

Hi! 👋

I'm currently trying to package this project for Arch Linux and ran into one failing test (test_trustme_cert_loaded_via_capath).

Maybe you have any ideas what might be going wrong here. I think I have all required (development) dependencies installed.

test_trustme_cert_loaded_via_capath failed (1 runs remaining out of 2).
        <class 'urllib3.exceptions.MaxRetryError'>
        HTTPSConnectionPool(host='localhost', port=32851): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed:
 unable to get local issuer certificate (_ssl.c:1006)')))
        [<TracebackEntry /usr/lib/python3.11/site-packages/_pytest/runner.py:341>, <TracebackEntry /usr/lib/python3.11/site-packages/_pytest/runner.py:262>, <TracebackEntry /usr/lib/python3.11/site-pa
ckages/pluggy/_hooks.py:493>, <TracebackEntry /usr/lib/python3.11/site-packages/pluggy/_manager.py:115>, <TracebackEntry /usr/lib/python3.11/site-packages/pluggy/_callers.py:152>, <TracebackEntry /usr
/lib/python3.11/site-packages/pluggy/_result.py:114>, <TracebackEntry /usr/lib/python3.11/site-packages/pluggy/_callers.py:77>, <TracebackEntry /usr/lib/python3.11/site-packages/_pytest/runner.py:177>
, <TracebackEntry /usr/lib/python3.11/site-packages/_pytest/runner.py:169>, <TracebackEntry /usr/lib/python3.11/site-packages/_pytest/python.py:1792>, <TracebackEntry /usr/lib/python3.11/site-packages
/pluggy/_hooks.py:493>, <TracebackEntry /usr/lib/python3.11/site-packages/pluggy/_manager.py:115>, <TracebackEntry /usr/lib/python3.11/site-packages/pluggy/_callers.py:152>, <TracebackEntry /usr/lib/p
ython3.11/site-packages/pluggy/_result.py:114>, <TracebackEntry /usr/lib/python3.11/site-packages/pluggy/_callers.py:77>, <TracebackEntry /usr/lib/python3.11/site-packages/_pytest/python.py:194>, <Tra
cebackEntry /build/python-truststore/src/truststore-0.8.0/tests/test_api.py:316>, <TracebackEntry /usr/lib/python3.11/site-packages/urllib3/request.py:77>, <TracebackEntry /usr/lib/python3.11/site-pac
kages/urllib3/request.py:99>, <TracebackEntry /usr/lib/python3.11/site-packages/urllib3/poolmanager.py:376>, <TracebackEntry /usr/lib/python3.11/site-packages/urllib3/connectionpool.py:827>, <Tracebac
kEntry /usr/lib/python3.11/site-packages/urllib3/connectionpool.py:827>, <TracebackEntry /usr/lib/python3.11/site-packages/urllib3/connectionpool.py:827>, <TracebackEntry /usr/lib/python3.11/site-pack
ages/urllib3/connectionpool.py:799>, <TracebackEntry /usr/lib/python3.11/site-packages/urllib3/util/retry.py:592>]
test_trustme_cert_loaded_via_capath failed; it passed 0 out of the required 1 times.
        <class 'urllib3.exceptions.MaxRetryError'>
        HTTPSConnectionPool(host='localhost', port=32851): Max retries exceeded with url: / (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed:
 unable to get local issuer certificate (_ssl.c:1006)')))
        [<TracebackEntry /usr/lib/python3.11/site-packages/_pytest/runner.py:341>, <TracebackEntry /usr/lib/python3.11/site-packages/_pytest/runner.py:262>, <TracebackEntry /usr/lib/python3.11/site-pa
ckages/pluggy/_hooks.py:493>, <TracebackEntry /usr/lib/python3.11/site-packages/pluggy/_manager.py:115>, <TracebackEntry /usr/lib/python3.11/site-packages/pluggy/_callers.py:152>, <TracebackEntry /usr
/lib/python3.11/site-packages/pluggy/_result.py:114>, <TracebackEntry /usr/lib/python3.11/site-packages/pluggy/_callers.py:77>, <TracebackEntry /usr/lib/python3.11/site-packages/_pytest/runner.py:177>
, <TracebackEntry /usr/lib/python3.11/site-packages/_pytest/runner.py:169>, <TracebackEntry /usr/lib/python3.11/site-packages/_pytest/python.py:1792>, <TracebackEntry /usr/lib/python3.11/site-packages
/pluggy/_hooks.py:493>, <TracebackEntry /usr/lib/python3.11/site-packages/pluggy/_manager.py:115>, <TracebackEntry /usr/lib/python3.11/site-packages/pluggy/_callers.py:152>, <TracebackEntry /usr/lib/p
ython3.11/site-packages/pluggy/_result.py:114>, <TracebackEntry /usr/lib/python3.11/site-packages/pluggy/_callers.py:77>, <TracebackEntry /usr/lib/python3.11/site-packages/_pytest/python.py:194>, <Tra
cebackEntry /build/python-truststore/src/truststore-0.8.0/tests/test_api.py:316>, <TracebackEntry /usr/lib/python3.11/site-packages/urllib3/request.py:77>, <TracebackEntry /usr/lib/python3.11/site-pac
kages/urllib3/request.py:99>, <TracebackEntry /usr/lib/python3.11/site-packages/urllib3/poolmanager.py:376>, <TracebackEntry /usr/lib/python3.11/site-packages/urllib3/connectionpool.py:827>, <Tracebac
kEntry /usr/lib/python3.11/site-packages/urllib3/connectionpool.py:827>, <TracebackEntry /usr/lib/python3.11/site-packages/urllib3/connectionpool.py:827>, <TracebackEntry /usr/lib/python3.11/site-pack
ages/urllib3/connectionpool.py:799>, <TracebackEntry /usr/lib/python3.11/site-packages/urllib3/util/retry.py:592>]
test_trustme_cert_still_uses_system_certs passed 1 out of the required 1 times. Success!
test_macos_10_7_import_error passed 1 out of the required 1 times. Success!

===End Flaky Test Report===
=========================== short test summary info ============================
FAILED tests/test_api.py::test_trustme_cert_loaded_via_capath - urllib3.excep...
================== 1 failed, 41 passed, 10 skipped in 17.85s ===================

Full build and test logs:

python-truststore-0.8.0-1-x86_64-check.log
python-truststore-0.8.0-1-x86_64-build.log

RecursionError: maximum recursion depth exceeded on Python 3.10.4

Behaviour

I'm hitting a RecursionError: maximum recursion depth exceeded while running https://github.com/pradyunsg/did with a patch to add truststore.inject_ssl() for use with httpx (i.e. not the requests/urllib3 stack). Patch below...

diff --git a/src/did.py b/src/did.py
index 31ebad3..a6eaf2b 100644
--- a/src/did.py
+++ b/src/did.py
@@ -6,6 +6,10 @@
 OSS update blog posts.
 """
 
+import truststore
+
+truststore.inject_into_ssl()
+
 __version__ = "0.1.0"
 
 import asyncio

I'm not quite sure why the recursion error is happening, but filing an issue eagerly to flag this to y'all. :)

Environment

❯ pip freeze
aiorwlock==1.3.0
anyio==3.6.2
attrs==22.2.0
certifi==2022.12.7
cffi==1.15.1
click==8.1.3
cryptography==40.0.1
fasteners==0.17.3
gidgethub==5.2.1
h11==0.14.0
httpcore==0.16.3
httpx==0.23.3
httpx-cache==0.7.0
idna==3.4
markdown-it-py==2.2.0
mdurl==0.1.2
msgpack==1.0.5
-e git+https://github.com/pradyunsg/did.git@60816da46caf348e8262698e21043bcd7e6d0118#egg=pradyunsg_did
pycparser==2.21
Pygments==2.14.0
PyJWT==2.6.0
rfc3986==1.5.0
rich==13.3.2
sniffio==1.3.0
truststore==0.6.0
uritemplate==4.1.1

Output

This is a traceback printed with rich.traceback with_locals=True, trimmed. You should see this if you run did this quarter or did this month after installing that repository.

[manual note: trimmed for brevity]
│ /Users/pgedam/Developer/github/did/src/did.py:481 in this                                        │
│                                                                                                  │
│   478 │   """stats for current [period header time]"""                                           │
│   479 │   period_ref, since, until = get_this_period(today=date.today(), period=period)          │
│   480 │   print(f"# Status update for {period_ref} ({since} to {until}*)")                       │
│ ❱ 481 │   main(since=since, until=until)                                                         │
│   482                                                                                            │
│   483                                                                                            │
│   484 @did.command("yesterday")                                                                  │
│                                                                                                  │
│ ╭──────────────── locals ─────────────────╮                                                      │
│ │     period = 'quarter'                  │                                                      │
│ │ period_ref = 'Q1 2023'                  │                                                      │
│ │      since = datetime.date(2023, 1, 1)  │                                                      │
│ │      until = datetime.date(2023, 3, 30) │                                                      │
│ ╰─────────────────────────────────────────╯                                                      │
│                                                                                                  │
│ /Users/pgedam/Developer/github/did/src/did.py:435 in main                                        │
│                                                                                                  │
│   432 def main(*, since: date, until: date) -> None:                                             │
│   433 │   print()                                                                                │
│   434 │   try:                                                                                   │
│ ❱ 435 │   │   asyncio.run(github(since, until))                                                  │
│   436 │   except gidgethub.BadRequest:                                                           │
│   437 │   │   rich.print(rich.traceback.Traceback(suppress=[asyncio]), file=sys.stderr)          │
│   438 │   │   gh_token_issue("Maybe the token expired? https://github.com/settings/tokens/")     │
│                                                                                                  │
│ ╭────────────── locals ──────────────╮                                                           │
│ │ since = datetime.date(2023, 1, 1)  │                                                           │
│ │ until = datetime.date(2023, 3, 30) │                                                           │
│ ╰────────────────────────────────────╯                                                           │
│                                                                                                  │
│ /Users/pgedam/.asdf/installs/python/3.10.4/lib/python3.10/asyncio/runners.py:44 in run           │
│                                                                                                  │
│ /Users/pgedam/.asdf/installs/python/3.10.4/lib/python3.10/asyncio/base_events.py:646 in          │
│ run_until_complete                                                                               │
│                                                                                                  │
│ /Users/pgedam/Developer/github/did/src/did.py:305 in github                                      │
│                                                                                                  │
│   302 │   #     "SponsorshipEvent",                                                              │
│   303 │   # }                                                                                    │
│   304 │                                                                                          │
│ ❱ 305 │   async with httpx_cache.AsyncClient(cache=CACHE) as client:                             │
│   306 │   │   gh = gidgethub.httpx.GitHubAPI(                                                    │
│   307 │   │   │   client,                                                                        │
│   308 │   │   │   "pradyunsg",                                                                   │
│                                                                                                  │
│ ╭─────────────────────────────────────────── locals ───────────────────────────────────────────╮ │
│ │  searches = {                                                                                │ │
│ │             │   'Issues created': 'author:pradyunsg type:issue                               │ │
│ │             created:2023-01-01..2023-03-30',                                                 │ │
│ │             │   'Assigned issues closed': 'assignee:pradyunsg type:issue                     │ │
│ │             closed:2023-01-01..2023-03-30',                                                  │ │
│ │             │   'PRs created': 'author:pradyunsg type:pr created:2023-01-01..2023-03-30',    │ │
│ │             │   'PRs reviewed': 'reviewed-by:pradyunsg type:pr                               │ │
│ │             reviewed:2023-01-01..2023-03-30',                                                │ │
│ │             │   'Assigned PRs closed': 'assignee:pradyunsg type:pr                           │ │
│ │             closed:2023-01-01..2023-03-30'                                                   │ │
│ │             }                                                                                │ │
│ │     since = datetime.date(2023, 1, 1)                                                        │ │
│ │ time_term = '2023-01-01..2023-03-30'                                                         │ │
│ │     until = datetime.date(2023, 3, 30)                                                       │ │
│ ╰──────────────────────────────────────────────────────────────────────────────────────────────╯ │
│                                                                                                  │
│ /Users/pgedam/Developer/github/did/.venv/lib/python3.10/site-packages/httpx_cache/client.py:167  │
│ in __init__                                                                                      │
│                                                                                                  │
│ /Users/pgedam/Developer/github/did/.venv/lib/python3.10/site-packages/httpx/_client.py:1400 in   │
│ __init__                                                                                         │
│                                                                                                  │
│ /Users/pgedam/Developer/github/did/.venv/lib/python3.10/site-packages/httpx_cache/client.py:200  │
│ in _init_transport                                                                               │
│                                                                                                  │
│ /Users/pgedam/Developer/github/did/.venv/lib/python3.10/site-packages/httpx/_client.py:1448 in   │
│ _init_transport                                                                                  │
│                                                                                                  │
│ /Users/pgedam/Developer/github/did/.venv/lib/python3.10/site-packages/httpx/_transports/default. │
│ py:261 in __init__                                                                               │
│                                                                                                  │
│ /Users/pgedam/Developer/github/did/.venv/lib/python3.10/site-packages/httpx/_config.py:51 in     │
│ create_ssl_context                                                                               │
│                                                                                                  │
│ /Users/pgedam/Developer/github/did/.venv/lib/python3.10/site-packages/httpx/_config.py:75 in     │
│ __init__                                                                                         │
│                                                                                                  │
│ /Users/pgedam/Developer/github/did/.venv/lib/python3.10/site-packages/httpx/_config.py:87 in     │
│ load_ssl_context                                                                                 │
│                                                                                                  │
│ /Users/pgedam/Developer/github/did/.venv/lib/python3.10/site-packages/httpx/_config.py:124 in    │
│ load_ssl_context_verify                                                                          │
│                                                                                                  │
│ /Users/pgedam/Developer/github/did/.venv/lib/python3.10/site-packages/httpx/_config.py:160 in    │
│ _create_default_ssl_context                                                                      │
│                                                                                                  │
│ /Users/pgedam/Developer/github/did/.venv/lib/python3.10/site-packages/httpx/_compat.py:30 in     │
│ set_minimum_tls_version_1_2                                                                      │
│                                                                                                  │
│ /Users/pgedam/Developer/github/did/.venv/lib/python3.10/site-packages/truststore/_api.py:226 in  │
│ minimum_version                                                                                  │
│                                                                                                  │
│   223 │                                                                                          │
│   224 │   @minimum_version.setter                                                                │
│   225 │   def minimum_version(self, value: ssl.TLSVersion) -> None:                              │
│ ❱ 226 │   │   self._ctx.minimum_version = value                                                  │
│   227 │                                                                                          │
│   228 │   @property                                                                              │
│   229 │   def options(self) -> ssl.Options:                                                      │
│                                                                                                  │
│ ╭────────────────────────── locals ──────────────────────────╮                                   │
│ │  self = <truststore._api.SSLContext object at 0x105e2f8c0> │                                   │
│ │ value = <TLSVersion.TLSv1_2: 771>                          │                                   │
│ ╰────────────────────────────────────────────────────────────╯                                   │
│                                                                                                  │
│ /Users/pgedam/.asdf/installs/python/3.10.4/lib/python3.10/ssl.py:603 in minimum_version          │
│                                                                                                  │
│    600 │   │   def minimum_version(self, value):                                                 │
│    601 │   │   │   if value == TLSVersion.SSLv3:                                                 │
│    602 │   │   │   │   self.options &= ~Options.OP_NO_SSLv3                                      │
│ ❱  603 │   │   │   super(SSLContext, SSLContext).minimum_version.__set__(self, value)            │
│    604 │   │                                                                                     │
│    605 │   │   @property                                                                         │
│    606 │   │   def maximum_version(self):                                                        │
│                                                                                                  │
[manual note: this stack entry is repeated a _lot_ of times, trimmed for brevity]
│ ╭──────────────────── locals ────────────────────╮                                               │
│ │  self = <ssl.SSLContext object at 0x105edc7c0> │                                               │
│ │ value = <TLSVersion.TLSv1_2: 771>              │                                               │
│ ╰────────────────────────────────────────────────╯                                               │
│                                                                                                  │
│ /Users/pgedam/.asdf/installs/python/3.10.4/lib/python3.10/ssl.py:603 in minimum_version          │
│                                                                                                  │
│    600 │   │   def minimum_version(self, value):                                                 │
│    601 │   │   │   if value == TLSVersion.SSLv3:                                                 │
│    602 │   │   │   │   self.options &= ~Options.OP_NO_SSLv3                                      │
│ ❱  603 │   │   │   super(SSLContext, SSLContext).minimum_version.__set__(self, value)            │
│    604 │   │                                                                                     │
│    605 │   │   @property                                                                         │
│    606 │   │   def maximum_version(self):                                                        │
│                                                                                                  │
│ ╭──────────────────── locals ────────────────────╮                                               │
│ │  self = <ssl.SSLContext object at 0x105edc7c0> │                                               │
│ │ value = <TLSVersion.TLSv1_2: 771>              │                                               │
│ ╰────────────────────────────────────────────────╯                                               │
│                                                                                                  │
│ /Users/pgedam/.asdf/installs/python/3.10.4/lib/python3.10/ssl.py:603 in minimum_version          │
│                                                                                                  │
│    600 │   │   def minimum_version(self, value):                                                 │
│    601 │   │   │   if value == TLSVersion.SSLv3:                                                 │
│    602 │   │   │   │   self.options &= ~Options.OP_NO_SSLv3                                      │
│ ❱  603 │   │   │   super(SSLContext, SSLContext).minimum_version.__set__(self, value)            │
│    604 │   │                                                                                     │
│    605 │   │   @property                                                                         │
│    606 │   │   def maximum_version(self):                                                        │
│                                                                                                  │
│ ╭──────────────────── locals ────────────────────╮                                               │
│ │  self = <ssl.SSLContext object at 0x105edc7c0> │                                               │
│ │ value = <TLSVersion.TLSv1_2: 771>              │                                               │
│ ╰────────────────────────────────────────────────╯                                               │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
RecursionError: maximum recursion depth exceeded

pypy3.10: AttributeError: 'SSLContext' object has no attribute '_ctx'. Did you mean: 'ctx'?

When running the test suite on pypy3.10 7.3.12, every test fails with the following error:

_____________________________________________________ test_verify_mode_cert_none ______________________________________________________

    def test_verify_mode_cert_none():
>       ctx = truststore.SSLContext(ssl.PROTOCOL_TLS_CLIENT)

tests/test_sslcontext.py:39: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/usr/lib/pypy3.10/ssl.py:496: in __new__
    self = _SSLContext.__new__(cls, protocol)
/usr/lib/pypy3.10/_cffi_ssl/_stdssl/__init__.py:1221: in __new__
    lib.SSL_CTX_set_post_handshake_auth(self.ctx, self.post_handshake_auth)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <truststore._api.SSLContext object at 0x00005599f6700138>

    @property
    def post_handshake_auth(self) -> bool:
>       return self._ctx.post_handshake_auth
E       AttributeError: 'SSLContext' object has no attribute '_ctx'. Did you mean: 'ctx'?

.nox/test/lib/pypy3.10/site-packages/truststore/_api.py:232: AttributeError

Fallback on deprecated `SecTrustEvaluate` function on macOS 10.13 and earlier

I am trying to use truststore on Mac OS X 10.11 (El Capitan). Both Python 3.10 and truststore would seem to work with this combination, but it is failing to load a symbol, which seems to have been introduced in 10.14 when verifying a certificate in a HTTPS request.

Are older macOS versions really tested with the current truststore?
Please let me know what additional debug info I can provide, but here is output from a Terminal with some useful steps to check if truststore is working properly:

Last login: Fri Oct 13 14:03:12 on ttys004
Mac-mini-ElCapitan-VM:~ jorik$ uname -a
Darwin Mac-mini-ElCapitan-VM.local 15.6.0 Darwin Kernel Version 15.6.0: Thu Jun 21 20:07:40 PDT 2018; root:xnu-3248.73.11~1/RELEASE_X86_64 x86_64
Mac-mini-ElCapitan-VM:~ jorik$ source ~/src/trust_test/venv/bin/activate
(venv) Mac-mini-ElCapitan-VM:~ jorik$ pip list
Package            Version
------------------ ---------
certifi            2023.7.22
charset-normalizer 3.3.0
idna               3.4
pip                23.2.1
requests           2.31.0
setuptools         68.1.2
truststore         0.8.0
urllib3            2.0.6
(venv) Mac-mini-ElCapitan-VM:~ jorik$ python
Python 3.10.13 (main, Oct 12 2023, 14:16:20) [Clang 8.0.0 (clang-800.0.42.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import requests
>>> R = requests.get('https://www.python.org/')
>>> 
(venv) Mac-mini-ElCapitan-VM:~ jorik$ python
Python 3.10.13 (main, Oct 12 2023, 14:16:20) [Clang 8.0.0 (clang-800.0.42.1)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import truststore
>>> truststore.inject_into_ssl()
>>> import requests
>>> R = requests.get('https://www.python.org/')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/jorik/src/trust_test/venv/lib/python3.10/site-packages/requests/api.py", line 73, in get
    return request("get", url, params=params, **kwargs)
  File "/Users/jorik/src/trust_test/venv/lib/python3.10/site-packages/requests/api.py", line 59, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Users/jorik/src/trust_test/venv/lib/python3.10/site-packages/requests/sessions.py", line 589, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/jorik/src/trust_test/venv/lib/python3.10/site-packages/requests/sessions.py", line 703, in send
    r = adapter.send(request, **kwargs)
  File "/Users/jorik/src/trust_test/venv/lib/python3.10/site-packages/requests/adapters.py", line 486, in send
    resp = conn.urlopen(
  File "/Users/jorik/src/trust_test/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 790, in urlopen
    response = self._make_request(
  File "/Users/jorik/src/trust_test/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 467, in _make_request
    self._validate_conn(conn)
  File "/Users/jorik/src/trust_test/venv/lib/python3.10/site-packages/urllib3/connectionpool.py", line 1092, in _validate_conn
    conn.connect()
  File "/Users/jorik/src/trust_test/venv/lib/python3.10/site-packages/urllib3/connection.py", line 642, in connect
    sock_and_verified = _ssl_wrap_socket_and_match_hostname(
  File "/Users/jorik/src/trust_test/venv/lib/python3.10/site-packages/urllib3/connection.py", line 783, in _ssl_wrap_socket_and_match_hostname
    ssl_sock = ssl_wrap_socket(
  File "/Users/jorik/src/trust_test/venv/lib/python3.10/site-packages/urllib3/util/ssl_.py", line 469, in ssl_wrap_socket
    ssl_sock = _ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
  File "/Users/jorik/src/trust_test/venv/lib/python3.10/site-packages/urllib3/util/ssl_.py", line 513, in _ssl_wrap_socket_impl
    return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
  File "/Users/jorik/src/trust_test/venv/lib/python3.10/site-packages/truststore/_api.py", line 104, in wrap_socket
    _verify_peercerts(ssl_sock, server_hostname=server_hostname)
  File "/Users/jorik/src/trust_test/venv/lib/python3.10/site-packages/truststore/_api.py", line 300, in _verify_peercerts
    _verify_peercerts_impl(
  File "/Users/jorik/src/trust_test/venv/lib/python3.10/site-packages/truststore/_macos.py", line 437, in _verify_peercerts_impl
    sec_trust_eval_result = Security.SecTrustEvaluateWithError(
  File "/usr/local/Cellar/[email protected]/3.10.13/Frameworks/Python.framework/Versions/3.10/lib/python3.10/ctypes/__init__.py", line 387, in __getattr__
    func = self.__getitem__(name)
  File "/usr/local/Cellar/[email protected]/3.10.13/Frameworks/Python.framework/Versions/3.10/lib/python3.10/ctypes/__init__.py", line 392, in __getitem__
    func = self._FuncPtr((name_or_ordinal, self))
AttributeError: dlsym(0x7fc3d9462db0, SecTrustEvaluateWithError): symbol not found
>>>

From this page, it seems SecTrustEvaluateWithError is available since 10.14? Or maybe I'm misreading the documentation?

Add or update references to PDM, Conda, and pip in docs

PDM has adopted Truststore by default
Conda is adding an option in v23.9.0 to use Truststore when installing, similar to pip
Add note that pip will now bundle Truststore starting in v23.3, reducing the need to "bootstrap"

Implement Windows trust stores with CryptoAPI

https://docs.microsoft.com/en-us/windows/win32/seccrypto/cryptography-functions#certificate-trust-list-functions

Add automated publishing to PyPI

Maybe a manually triggered github workflow with required review? https://docs.github.com/en/actions/managing-workflow-runs/reviewing-deployments

❓ Use truststore with requests library

Requests is quite a popular library for http requests. It uses urllib3 as the powerhouse and it would be great if I could use truststore also with requests.
I am not sure if this something for requests to implement or needs instruction here.

I have not found a direct injection point where can I access the SSL context in urllib3 from a requests Session .

Would it be possible to use truststore with requests somehow?

Double CI jobs

I'd suggest picking either the push or pull_request events to trigger CI (and be required for merge), instead of both, depending on whether you want it to run for branches without a PR or not. As it is we end up with:

Can I use this in production yet?

First: Awesome work. Certificate issue in enterprise is the main cause of at least 10% of my grey hair.
Second: I watched your talk from 2022 and read the documentation and release log and still not fully confident. Is this ready for production now? Can I already import this magic and have it fix my problem?

Fix example in README to use PROTOCOL_TLS_CLIENT

Avoids a DeprecationWarning.

The dreaded CERTIFICATE_VERIFY_FAILED after using trusstore on python requests (pip-system-certs was not helping either)

Truststore version : 0.8.0
Win 10 (64bit)
Python version: 3.11.5

Hi (completely newby here)

Coming here after reading :
https://pip.pypa.io/en/stable/topics/https-certificates/#using-system-certificate-stores

I have used:
python.exe -m pip install requests --use-feature=truststore

I used this approach because validationissues while placing request to corporate network server. When I developed the validation issues code had been workaurond using pip-system-certs, But for a last few weeks I am getting errors (I do not know what could have changed).

The SSL errors are of the form
urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1129)

While googling to find some fix I read the entry in:

https://pip.pypa.io/en/stable/topics/https-certificates/#using-system-certificate-stores

where they wrote:

"If you encounter a TLS/SSL error when using the truststore feature you should open an issue on the truststore GitHub issue tracker instead of pip’s issue tracker. The maintainers of truststore will help diagnose and fix the issue."

See below the dup I am getting after going the trusstore route on requests library.

File "C:\Users\AB5720\PycharmProjects\ReportCertificates\venv\Lib\site-packages\urllib3\util\ssl_.py", line 469, in ssl_wrap_socket
ssl_sock = ssl_wrap_socket_impl(sock, context, tls_in_tls, server_hostname)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\AB5720\PycharmProjects\ReportCertificates\venv\Lib\site-packages\urllib3\util\ssl.py", line 513, in _ssl_wrap_socket_impl
return ssl_context.wrap_socket(sock, server_hostname=server_hostname)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\AB5720\AppData\Local\Programs\Python\Python311\Lib\ssl.py", line 517, in wrap_socket
return self.sslsocket_class._create(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\AB5720\AppData\Local\Programs\Python\Python311\Lib\ssl.py", line 1108, in _create
self.do_handshake()
File "C:\Users\AB5720\AppData\Local\Programs\Python\Python311\Lib\ssl.py", line 1379, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1006)

This is the log while i installed truststore and applied it to requests

(venv) PS C:\Users\AB5720\PycharmProjects\ReportCertificates\venv\Scripts> pip install truststore
Requirement already satisfied: truststore in c:\users\ab5720\pycharmprojects\reportcertificates\venv\lib\site-packages (0.8.0)
(venv) PS C:\Users\AB5720\PycharmProjects\ReportCertificates\venv\Scripts> python.exe -m pip install requests --use-feature=truststore
Collecting requests
Obtaining dependency information for requests from https://files.pythonhosted.org/packages/70/8e/0e2d847013cb52cd35b38c009bb167a1a26b2ce6cd6965bf26b47bc0bf44/requests-2.31.0-py3-none-any.whl.metadata
Downloading requests-2.31.0-py3-none-any.whl.metadata (4.6 kB)
Collecting charset-normalizer<4,>=2 (from requests)
Obtaining dependency information for charset-normalizer<4,>=2 from https://files.pythonhosted.org/packages/91/6e/db0e545302bf93b6dbbdc496dd192c7f8e8c3bb1584acba069256d8b51d4/charset_normalizer-3.2.0-cp311-cp311-win_amd64.whl.metadata
Downloading charset_normalizer-3.2.0-cp311-cp311-win_amd64.whl.metadata (31 kB)
Collecting idna<4,>=2.5 (from requests)
Using cached idna-3.4-py3-none-any.whl (61 kB)
Collecting urllib3<3,>=1.21.1 (from requests)
Obtaining dependency information for urllib3<3,>=1.21.1 from https://files.pythonhosted.org/packages/37/dc/399e63f5d1d96bb643404ee830657f4dfcf8503f5ba8fa3c6d465d0c57fe/urllib3-2.0.5-py3-none-any.whl.metadata
Downloading urllib3-2.0.5-py3-none-any.whl.metadata (6.6 kB)
Collecting certifi>=2017.4.17 (from requests)
Obtaining dependency information for certifi>=2017.4.17 from https://files.pythonhosted.org/packages/4c/dd/2234eab22353ffc7d94e8d13177aaa050113286e93e7b40eae01fbf7c3d9/certifi-2023.7.22-py3-none-any.whl.metadata
Using cached certifi-2023.7.22-py3-none-any.whl.metadata (2.2 kB)
Using cached requests-2.31.0-py3-none-any.whl (62 kB)
Using cached certifi-2023.7.22-py3-none-any.whl (158 kB)
Downloading charset_normalizer-3.2.0-cp311-cp311-win_amd64.whl (96 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 96.6/96.6 kB 2.8 MB/s eta 0:00:00
Using cached urllib3-2.0.5-py3-none-any.whl (123 kB)
Installing collected packages: urllib3, idna, charset-normalizer, certifi, requests
Successfully installed certifi-2023.7.22 charset-normalizer-3.2.0 idna-3.4 requests-2.31.0 urllib3-2.0.5

Add a test case for missing intermediates without AIA information

We're not aware of a website that sends a certificate chain that's missing an intermediate and doesn't use Authority Information Access (AIA). After #9 is implemented we can create a certificate chain manually ourselves with trustme to verify this functionality.

Client authentication on Windows?

I am currently using Python for performing client authentication on Windows as part of a HTTP request with code like this:

import http.client, ssl

# load clientAuth cert (with private key) from file
context = ssl.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
context.load_default_certs()
context.load_cert_chain(certfile="MyClientCert.pem")

# submit HTTP request
conn = http.client.HTTPSConnection(hostname, port=443, context=context)
conn.request('GET', '/')

# print response
r = conn.getresponse()

This works fine but has the downside of requiring the client certificate (including private key) to be provided through a PEM file. I would like to avoid this and instead use a certificate directly from the Windows certificate store, so that the private key can be stored non-exportable in the TPM chip for improved security.

Is it possible to use this library for client authentication on Windows?

Desired pseudocode

This is roughly the type of code that I want to write to enable TPM-based client authentication:

import socket, ssl, truststore

# Load clientAuth cert named "MyClientCert" from "CurrentUser/My" store
ctx = truststore.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
ctx.load_cert_chain("MyClientCert", "My", StoreLocation.CurrentUser)

# Connect and initiate TLS handshake with client auth.
sock = socket.create_connection((hostname, 443))
sock = ctx.wrap_socket(sock, server_hostname=hostname)
...

This will then utilize the following certificate from the Windows certificate store:

Add test cases for PROTOCOL_TLS_SERVER

Currently we're testing many cases using ssl.PROTOCOL_TLS_CLIENT and not many with the server. We ran into an issue with specifically macOS that was tough to debug w/ CI alone. Need an actual macOS machine to see what's going on easier: https://github.com/sethmlarson/truststore/actions/runs/4258995744/jobs/7410790041

Empty or no certificate error

Testing out requests and truststore with some corporate CA yields errors, depending on how the certificate is set up.

urllib3\connection.py:458: SubjectAltNameWarning: Certificate for xxx.intranet.cnb has no `subjectAltName`, falling back to check for a `commonName` for now. This feature is being removed by major browsers and deprecated by RFC 2818. (See https://github.com/urllib3/urllib3/issues/497 for details.)
  warnings.warn(
ERROR empty or no certificate, match_hostname needs a SSL socket or SSL context with either CERT_OPTIONAL or CERT_REQUIRED

Trying the same servers with urllib3 and truststore:

>>> import urllib3
>>> import truststore
>>> ctx = truststore.SSLContext()
<stdin>:1: DeprecationWarning: ssl.SSLContext() without protocol argument is deprecated.
<stdin>:1: DeprecationWarning: ssl.PROTOCOL_TLS is deprecated
>>> http = urllib3.PoolManager(ssl_context=ctx)
>>> resp = http.request('GET', 'https://xxx.intranet.cnb')
C:\Users\wdwni\.virtualenvs\sdr-api-example-WgBfLP6C\lib\site-packages\urllib3\connectionpool.py:1045: InsecureRequestWarning: Unverified HTTPS request is being made to host 'xxx.intranet.cnb'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
>>> resp = http.request('GET', 'https://yyy.intranet.cnb')
C:\Users\wdwni\.virtualenvs\sdr-api-example-WgBfLP6C\lib\site-packages\urllib3\connectionpool.py:1045: InsecureRequestWarning: Unverified HTTPS request is being made to host 'yyy.intranet.cnb'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/1.26.x/advanced-usage.html#ssl-warnings
  warnings.warn(
>>> truststore.__version__
'0.4.0'
>>> urllib3.__version__
'1.26.11'

Originally discovered in #71

Supporting older python

Hi, I'm generally quite in favour of keeping up with latest stable python however there are many common OS / distros which still ship with Putin 3.8 / 3.9.

Are there particular features of 3.10 that this library depends on that aren't available in older versions?

As per #75 I'm interested in ways to use this everywhere by default, however the 3.10 minimum feels a bit restrictive still.

Document how to use truststore with `requests`

I expect this will be a FAQ

Is it necessary for truststore to do its own hostname matching?

Currently on all 3 platforms, truststore is setting TrustStoreSSLContext.check_hostname to False to suppress the built-in hostname check, and then doing its own check at the bottom of _verify_peercerts

On Mac, this isn't necessary because the hostname is passed to Security.SecPolicyCreateSSL -- the tests still pass if I comment out the call to _match_hostname. On Windows the situation is similar with the hostname passed to CertVerifyCertificateChainPolicy.

On Linux I think we should be able to set check_hostname = True and rely on the built-in check, since verify_mode is not set to None. But, in fact this fails the test with urllib3 when the hostname is an IP address because urllib3 is not passing the hostname along to wrap_socket in that case.

So the question is a) am I missing something, and b) can we get rid of _match_hostname?

(And one more minor question: is there any problem that _verify_peercerts is using sslobj.get_verified_chain() to get the hostname, given that the ssl module doesn't know how to verify the chain on its own?)

Tests not actually running on Python 3.11

I just noticed that nox is skipping the tests on Python 3.11 because of

truststore/noxfile.py

Line 30 in f7d492d

@nox.session(python="3.10")

Override SSLContext._load_windows_store_certs with a no-op

SSLContext has a _load_windows_store_certs method which is called by SSLContext.load_default_certs to load CA certs on Windows. We could replace it with a method that does nothing to avoid unnecessary work.

Start using native modules instead of ctypes

CRYPT_E_NO_REVOCATION_CHECK when checking a self-signed certificate created by mkcert on Windows

After generating a certificate with mkcert (using the example in README) and starting a server on localhost, https://localhost correctly runs in the browser, but using socket with truststore fails with the following:

Python 3.10.0 (tags/v3.10.0:b494f59, Oct  4 2021, 19:00:18) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>> import socket
>>> import ssl
>>> import truststore
>>> sock = socket.create_connection(("127.0.0.1", 443))
>>> ctx = truststore.SSLContext(ssl.PROTOCOL_TLS_CLIENT)
>>> sock = ctx.wrap_socket(sock, server_hostname="127.0.0.1")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "site-packages\truststore\_api.py", line 54, in wrap_socket
    _verify_peercerts(ssl_sock, server_hostname=server_hostname)
  File "site-packages\truststore\_api.py", line 106, in _verify_peercerts
    _verify_peercerts_impl(
  File "site-packages\truststore\_windows.py", line 318, in _verify_peercerts_impl
    _get_and_verify_cert_chain(
  File "site-packages\truststore\_windows.py", line 410, in _get_and_verify_cert_chain
    raise err from None
ssl.SSLCertVerificationError: ('Certificate chain policy error 0x80092012 [0]',)

Create an issue template with common debugging information

Things like:

Truststore version
OS variant and version
Python version
OpenSSL version
verify paths
Public URL if applicable

Maybe more?

`certificate verify failed: self-signed certificate in certificate chain` when installing a zip file from github

When installing a package from a zip file, like this:

pip install https://github.com/goldmann/docker-squash/archive/4a7fc2c3a2175d868ff60eefdbab53240a7641d5.zip --use-
feature=truststore

I received the following error:

Collecting https://github.com/goldmann/docker-squash/archive/4a7fc2c3a2175d868ff60eefdbab53240a7641d5.zip
  WARNING: Retrying (Retry(total=4, connect=None, read=None, redirect=None, status=None)) after connection broken by 'SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1000)'))': /goldmann/docker-squash/archive/4a7fc2c3a2175d868ff60eefdbab53240a7641d5.zip
[...]
ERROR: Could not install packages due to an OSError: HTTPSConnectionPool(host='github.com', port=443): Max retries exceeded with url: /goldmann/docker-squash/archive/4a7fc2c3a2175d868ff60eefdbab53240a7641d5.zip (Caused by SSLError(SSLCertVerificationError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self-signed certificate in certificate chain (_ssl.c:1000)')))

I'm not sure if installing via a zip file specifically is the issue, because AFAIK our pypi certificates are not being substituted by corporate.
I can confirm that curl https://github.com works without any cert problems.
Also, using the pip-system-certs package instead of the truststore option worked in this case.

From my limited understanding, it seems that in our corporate environment github returns a certificate signed by our corporate root CA with no intermediates. This root CA was manually installed using ca-certificates package on ubuntu 22.04.

Implement SSLContext.get_ca_certs()

I suspect this will be possible in both macOS and Windows trust store APIs.

RecursionError when SSLContext imported from urllib3

We recently encountered an interesting interaction with boto3 and arcgis due to arcgis' adoption of truststore (boto/boto3#3912). Boto3 has existing usage of the SSLContext from urllib3.utils.ssl_ to work around historical issues with pyopenssl. Most of that no longer exists, but the imports/behaviors are left in place for backwards compatibility.

When truststore.inject_into_ssl() is called, it's currently patching the SSLContext referenced in urllib3 resulting in a RecusionError depending on the order these operations are performed. I've created a minimal repro with only Truststore and urllib3.

Minimal Reproduction

from urllib3.util.ssl_ import SSLContext, PROTOCOL_TLS_CLIENT

import truststore
truststore.inject_into_ssl()

s = SSLContext(PROTOCOL_TLS_CLIENT)
s.options |= 0  # Any arbitrary options int

This issue is avoided if the truststore injection is done first. This is fine for code you control, but becomes more complicated to root cause when a module using truststore is imported later in user code.

Environment details

Original report is for Windows and Python 3.11.5, I've also repro'd on macOS 13.5.2 with Python 3.11.5.

urllib3==2.0.7
truststore==0.8.0

Verify with additional certificates in addition to system certificates

Allow passing additional certificates via SSLContext.load_verify_locations(). Only load system certificates when SSLContext.load_default_certs() is used.

Possibility to register truststore more globally

This is more a question/discussion than an issue.
I'm using a lot of Python libraries, which build on urllib3, aiohttp or requests. In seldom cases, I am able to get the pool manager or SSL context to inject truststore's context into it.

Is there an option to inject truststore more globally, so that dependent libraries use it?
Here are some examples:

overpass uses urllib3 and requests for tests
elasticsearch uses urllib3