twisted / txacme

Twisted client for the ACME (Automatic Certificate Management Environment) protocol

txacme: A Twisted implementation of the ACME protocol

ACME is Automatic Certificate Management Environment, a protocol that allows clients and certificate authorities to automate verification and certificate issuance. The ACME protocol is used by the free Let's Encrypt Certificate Authority.

txacme is an implementation of the protocol for Twisted, the event-driven networking engine for Python.

txacme is still under heavy development, and currently only an implementation of the client side of the protocol is planned; if you are interested in implementing or have need of the server side, please get in touch!

txacme’s documentation lives at Read the Docs, the code on GitHub. It’s lightly tested on Python 3.6+, and PyPy3.

txacme's Issues

Issuing service API for removing a certificate

This is necessary, rather than going directly through the ICertificateStore interface, because the issuing service may be in the process of reissuing the certificate, resulting in the new cert being inserted into the store after the removal, "resurrecting" it.

See #76 for the counterpart of this.

Implement libcloud-based DNS responder

This is going to involve deferToThread and probably not be very "production-ready", but it has the advantage of not requiring writing a Twisted client library for some provider first (since apparently none exist currently), and providing reasonable support for a whole ton of providers at once. It also doubles as an example for implementing your own responder for, say, your internal PostgreSQL-backed PowerDNS or something.

See also #45.

Add the server_name and challenge to IResponder methods

These are necessary for implementing dns-01 challenge responders, and probably useful in general.

`acme` API change

There were some changes in error handling, apparently:

[FAIL]
Traceback (most recent call last):
Failure: testtools.testresult.real._StringException: Traceback (most recent call last):
  File "/home/travis/build/mithrandi/txacme/.tox/py27-twlatest/lib/python2.7/site-packages/txacme/test/test_client.py", line 1344, in test_check_invalid_error
    failed_with(IsInstance(errors.ClientError)))
  File "/home/travis/build/mithrandi/txacme/.tox/py27-twlatest/lib/python2.7/site-packages/testtools/testcase.py", line 498, in assertThat
    raise mismatch_error
testtools.matchers._impl.MismatchError: '(Error(detail=None, title=None, typ='about:blank'), TestResponse(code=403, content_type='application/problem+json', nonce=None, json=<function <lambda> at 0x7f06fefd4398>, links=None))' is not an instance of ClientError: after <operator.attrgetter object at 0x7f06fe655210> on <twisted.python.failure.Failure txacme.client.ServerError: (Error(detail=None, title=None, typ='about:blank'), TestResponse(code=403, content_type='application/problem+json', nonce=None, json=<function <lambda> at 0x7f06fefd4398>, links=None))>
txacme.test.test_client.JWSClientTests.test_check_invalid_error

This is a bit backwards incompatible for any error handling code, but it's unlikely this actually affects anyone.

Implement http-01 filesystem-based responder

There's a lot of ways to get an http-01 response into place, but writing it to a directory on the local filesystem is pretty commonplace.

answer_challenge is a bit of a footgun

The logic of making sure you clean up the challenge after you poll is left up to the caller, which sucks. There should probably just be a helper that combines answering and polling.

logging is insufficient

The only logs I'm getting are twisted.web.client factory messages. Txacme should at least log cert renewals and challenge responses.

Implement dns-01 Rackspace DNS responder

This is one of the common DNS providers we should support. See also #45.

Integration tests for http-01 challenges

We should have these to go along with the other integration tests.

Directory fetching issues

Currently, a client object is passed in to AcmeIssuingService; this means that the directory is only fetched once. However, the ACME service might have changes to the directory at any time; while such changes are unlikely to be frequent, the service should tolerate this happening during operation. We should probably start every client "session" by fetching the directory, thus a "client creator" should be passed in instead of a client object.

high availability certificate provisioning

This may or may not be something that txacme has to do anything about directly; it may be a documentation issue.

doc bug: suggestion of invalid Client.from_url invocation

https://txacme.readthedocs.io/en/latest/using.html says client_creator could be, "for example", partial(Client.from_url, reactor=reactor, directory=LETSENCRYPT_STAGING_DIRECTORY, key=acme_key, alg=RS256). Except Client.from_url doesn't take a directory argument, so that's an error.

Document the "API" of the certs directory more strictly

Because we may want to add things like #27 in future, I'm going to reserve all non-documented paths under the certs dir that the endpoint uses for future use.

Implement dns-01 Route 53 responder

This is going to be a tough one since acme doesn't implement dns-01 yet (see certbot/certbot#2061) and there's no Route 53 implementation for Twisted anywhere.

Open questions:

1. Should we try to get Route 53 support into txaws or just write our own library?
2. Are there other common DNS APIs we can/should support?

ReactorAlreadyInstalledError

$ mktmpenv
...
$ pip install txacme
...
Successfully installed Twisted-16.4.1 acme-0.9.3 attrs-16.2.0 cffi-1.8.3 cryptography-1.5.2 eliot-0.11.0 enum34-1.1.6 funcsigs-1.0.2 idna-2.1 ipaddress-1.0.17 mock-2.0.0 ndg-httpsclient-0.4.2 pbr-1.10.0 pem-16.1.0 pyOpenSSL-16.2.0 pyasn1-0.1.9 pyasn1-modules-0.0.8 pycparser-2.16 pyrfc3339-1.0 pyrsistent-0.11.13 pytz-2016.7 requests-2.11.1 service-identity-16.0.0 six-1.10.0 treq-15.1.0 txacme-0.9.0 txsni-0.1.6 zope.interface-4.3.2
$ twist --help
Traceback (most recent call last):
  File "/Users/glyph/.virtualenvs/tmp-4d999315aefd700e/bin/twist", line 11, in <module>
    sys.exit(Twist.main())
  File "/Users/glyph/.virtualenvs/tmp-4d999315aefd700e/lib/python2.7/site-packages/twisted/application/twist/_twist.py", line 132, in main
    options = cls.options(argv)
  File "/Users/glyph/.virtualenvs/tmp-4d999315aefd700e/lib/python2.7/site-packages/twisted/application/twist/_twist.py", line 38, in options
    options.parseOptions(argv[1:])
  File "/Users/glyph/.virtualenvs/tmp-4d999315aefd700e/lib/python2.7/site-packages/twisted/application/twist/_options.py", line 157, in parseOptions
    self.installReactor()
  File "/Users/glyph/.virtualenvs/tmp-4d999315aefd700e/lib/python2.7/site-packages/twisted/application/twist/_options.py", line 77, in installReactor
    self["reactor"] = installReactor(name)
  File "/Users/glyph/.virtualenvs/tmp-4d999315aefd700e/lib/python2.7/site-packages/twisted/application/reactors.py", line 82, in installReactor
    installer.install()
  File "/Users/glyph/.virtualenvs/tmp-4d999315aefd700e/lib/python2.7/site-packages/twisted/application/reactors.py", line 60, in install
    namedAny(self.moduleName).install()
  File "/Users/glyph/.virtualenvs/tmp-4d999315aefd700e/lib/python2.7/site-packages/twisted/internet/selectreactor.py", line 198, in install
    installReactor(reactor)
  File "/Users/glyph/.virtualenvs/tmp-4d999315aefd700e/lib/python2.7/site-packages/twisted/internet/main.py", line 32, in installReactor
    raise error.ReactorAlreadyInstalledError("reactor already installed")
twisted.internet.error.ReactorAlreadyInstalledError: reactor already installed

This doesn't happen with twisted[tls], so I'm assuming it's txacme's fault, but I haven't investigated much further than this.

Document compatibility between DirectoryStore and txsni

These are deliberately compatible in the sense that you can point txacme and txsni at the same directory and have everything work (for example, web server with txacme listening on port 443 to issue certs, and mail server with txsni listening on port 25 sharing those certts).

The compatibility should be documented so that we ensure these remain compatible, and to clue in users about this synergy.

subjectAlternativeName support (eg www.example.org also gets a sAN example.org)

It'd be incredibly convenient if txacme would, when renewing a certificate whose name starts with www, request that the cert contain a subjectAlternativeName without the www.

Retry-After support

Boulder doesn't support this yet, so it's not urgent, but we should do this.

Put key creating/loading from AutoTLSEndpoint somewhere public

This logic is quite useful when assembling an issuing service, so it should be split out as something standalone and public.

Does it work with http2?

Was getting errors when pip install Twisted[http2], does this work for you?

Twisted HTTP/2 support does not work with txacme

This is because of glyph/txsni#9, ideally this would just be fixed upstream but we may need to work around it ourselves.

DirectoryStore should call asTextMode on the path

Currently the code implicitly requires unicode ("text mode") paths, but there's nothing stopping you passing in a bytes mode FilePath which will result in confusing errors.

Retry `badNonce` errors directly

The overall error handling logic should mean that these errors are not fatal, but we ought to be able to retry immediately with the fresh nonce, which will make things a little more reliable.

Errors prevent shutdown of twistd

Sometimes when I start up a twistd using program that is using txacme (specifically the lets, don't know if le exhibits the same behavior) an error will get printed like:

2016-06-18T17:57:19+0000 [HTTP11ClientProtocol (TLSMemoryBIOProtocol),client] Unhandled Error
    Traceback (most recent call last):
      File "/usr/local/lib/pypy2.7/dist-packages/treq/client.py", line 62, in connectionLost
        self.original.connectionLost(reason)
      File "/usr/local/lib/pypy2.7/dist-packages/treq/content.py", line 38, in connectionLost
        self.finished.callback(None)
      File "/usr/local/lib/pypy2.7/dist-packages/twisted/internet/defer.py", line 393, in callback
        self._startRunCallbacks(result)
      File "/usr/local/lib/pypy2.7/dist-packages/twisted/internet/defer.py", line 501, in _startRunCallbacks
        self._runCallbacks()
    --- <exception caught here> ---
      File "/usr/local/lib/pypy2.7/dist-packages/twisted/internet/defer.py", line 588, in _runCallbacks
        current.result = callback(current.result, *args, **kw)
      File "/usr/local/lib/pypy2.7/dist-packages/eliot/twisted.py", line 82, in callbackWithContext
        return self._action.run(callback, *args, **kwargs)
      File "/usr/local/lib/pypy2.7/dist-packages/eliot/_action.py", line 381, in run
        return f(*args, **kwargs)
      File "/usr/local/lib/pypy2.7/dist-packages/txacme/client.py", line 710, in _got_json
        messages.Error.from_json(jobj), response)
    txacme.client.ServerError: (Error(typ=u'urn:acme:error:serverInternal', title=None, detail=u'Unable to update registration'), <treq.response._Response object at 0x0000000006f17ef8>)

This gets printed, and everything continues to run and it even appears to get certificates, however whenever you attempt to shut down the twistd program using ctrl-c instead of shutting down a message like this grets printed:

^C2016-06-18T18:00:58+0000 [-] Received SIGINT, shutting down.
2016-06-18T18:00:58+0000 [-] Unhandled Error
    Traceback (most recent call last):
      File "/usr/local/lib/pypy2.7/dist-packages/twisted/application/app.py", line 390, in startReactor
        self.config, oldstdout, oldstderr, self.profiler, reactor)
      File "/usr/local/lib/pypy2.7/dist-packages/twisted/application/app.py", line 311, in runReactorWithLogging
        reactor.run()
      File "/usr/local/lib/pypy2.7/dist-packages/twisted/internet/base.py", line 1194, in run
        self.mainLoop()
      File "/usr/local/lib/pypy2.7/dist-packages/twisted/internet/base.py", line 1203, in mainLoop
        self.runUntilCurrent()
    --- <exception caught here> ---
      File "/usr/local/lib/pypy2.7/dist-packages/twisted/internet/base.py", line 798, in runUntilCurrent
        f(*a, **kw)
      File "/usr/local/lib/pypy2.7/dist-packages/twisted/internet/base.py", line 581, in stop
        "Can't stop reactor that isn't running.")
    twisted.internet.error.ReactorNotRunning: Can't stop reactor that isn't running.

This happens once for each time you hit Ctrl-C, and the only way I've been able to shut down twistd once this occurs, is to kill -9 the process.

If I start up twistd without txacme I don't have a problem nor do I have a problem if I start it up and txacme manages to do everything without an error being generated.

Importing any challenge type requires libcloud

Importing any challenge type requires the libcloud package to be installed. This should only be the case for the LibcloudDNSResponder class, not for HTTP01Responder and TLSSNI01Responder.

>>> from txacme.challenges import HTTP01Responder
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/jamie/.virtualenvs/tmp-bcf6d230fbc44c61/lib/python2.7/site-packages/txacme/challenges/__init__.py", line 2, in <module>
    from ._libcloud import LibcloudDNSResponder
  File "/Users/jamie/.virtualenvs/tmp-bcf6d230fbc44c61/lib/python2.7/site-packages/txacme/challenges/_libcloud.py", line 7, in <module>
    from libcloud.dns.providers import get_driver
ImportError: No module named libcloud.dns.providers

Restore 100% coverage

Some error-handling changes in acme mean that we are no longer at 100% coverage.

Support dns-01 challenges

This ticket is for tracking dns-01 support in general; #32 covers one specific backend for dns-01 challenges.

This is blocked on certbot/certbot#2061 which adds support for the dns-01 challenge to the acme library, which is used by txacme for all the protocol stuff. I'm not sure if any support code is needed in txacme beyond this and the IResponder interface, but at the very least, there should be documentation about how to implement your own responder (eg. inserting records into the SQL database backing a PowerDNS deployment).

Libcloud responder doesn't handle providers that include a trailing `.` in zone names.

In particular, route53 suffers from this.

Document certificate store implementations

There should be documentation about implementing the ICertificateStore interface. In particular, there is a nuance of this interface whereby you can handle keys in a "write-only" fashion; reading a certificate back only needs the certificate (to check the expiration date), not the private key, which is probably better security practice for networked implementations of this.

Finalize generic responder API

I think we don't actually need to implement #31 or #32 pre-1.0.0, but we should at least have the API shape for this finished. Currently things may be a bit too tied to the tls-sni-01 challenge type, it's hard to tell without attempting an implementation of a responder for another challenge type.

Figure out how to do caching

Something like txsni.maputils.Cache is too simplistic, because it assumes the underlying store will never change, which is obviously false in the case of txacme.

Prepare for the 0.9 release

Set up Towncrier, final documentation fixes, anything else?

Avoid re-authorizing if an existing authorization is still valid

Authorizations may last for much longer than certificates, we should check to see if a new authorization is in fact needed before authorizing.

Support HSM-style certificate stores

This would require enhancing the ICertificateStore interface (IOpaqueCertificateStore, maybe?); instead of txacme generating the private key and then signing the CSR with it, txacme would need to hand the CSR over to the certificate store for signing.

This is necessary for supporting HSMs, or HSM-like certificate stores (many software stores behave like an HSM in that access to the private key is restricted by policy, even though it obviously is still possible to extract the key).

Add an introductory tutorial

Hi!

I think it might be useful to have a tutorial which covers "here are the steps you should take as someone who wants to set up a website with HTTPS on a new (sub)domain using txacme".

E.g., I am passingly familiar with letsencrypt, and with the letsencrypt binary, having used them a total of one time before discovering txacme. I'm also familiar with twisted.web, and specifically with endpoints. What I was missing though was a way to connect the dots. Sample outline, just based on the steps you just told me to perform:

• Install txacme
• Create a directory for certs, and touch an empty file to signal txacme to renew the corresponding cert
• Edit DNS settings to add a record for that domain to allow auth
• Run txacme

(This is awesome by the way, looks like I'm quite close, once I remember my router password. Sigh.)

`twistd acme` command for standalone issuing

There should be a twistd command that runs an issuing service standalone. This would be most useful with http-01 and dns-01 challenge types, putting certificates into a store (eg. a directory) for other software to use. (It would be possible to make it work for tls-sni-01 as well, for that matter, but that's probably a niche use case)

Include Twisted API docs inventory

Twisted actually does have a sphinx inventory for the API docs, located at http://twistedmatrix.com/documents/current/api/objects.inv

Allow providing a registration email

Currently all ACME registrations are anonymous. Let's Encrypt doesn't have a problem with this, but some other future ACME CA might not allow this, and providing an email address with your registration enables you to receive certificate expiration warning emails from Let's Encrypt, which are useful if something goes wrong and you're not watching your logs closely to see the issuing failures from txacme.

Support TLS protocol negotiation (ALPN/NPN) in the endpoint

stopService() leaves pooled connections open

With the default client, a persistent HTTPConnectionPool is created for connections to the ACME directory. This pool isn't tracked, however, and when stopService() is called on the txacme service a pooled connection remains waiting on the reactor. This breaks my integration tests.

Issuing service API for issuing a new certificate on demand

It's currently only possible to add/remove certificates from txacme by touching/deleting certificate files in the certificate directory (or adding/removing from some other store implementation).

It's technically possible to directly store empty certificates in an ICertificateStore and then call the AcmeIssuingService._check_certs() but that is messy and may be prone to race conditions with the timed invocation of the same method.

It should be possible to safely add and remove certificates from txacme.

A further iteration on this is perhaps the ability to modify certificates, e.g. to change (or as it is described in the certbot client, "expand") the SANs (#37).

User hint about missing certificates

Currently, if you intended a service to be available for some name, but you forgot to put a cert for it into the store, the client will just get an inscrutable TLS failure, and nothing much useful will happen on the server end.

Is there something better we can do to clue the server admin in that they might be missing a cert?

Don't call default_backend() everywhere

We should really only call this in one or two places, and then pass the backend around. This would also pave the way towards allowing the user to pass in a backend of their own choosing.

Allow selecting responders on more than challenge type

For example, for the dns-01 challenge, it is likely that different hostnames may live in different zones or even different providers, requiring different responders to handle. There should probably be a responder method that determines whether the responder can respond to a particular challenge, rather than the current hardcoded logic that only checks the challenge type.

Rename directory to acme_directory in AutoTLSEndpoint

In Client there's not much ambiguity, but in the endpoint it's easy to see directory and think it's what you want when you actually want cert_store=DirectoryStore(...).

Debian / Ubuntu packaging

To faster and support a secure adoption of the library it would be important to have txacme packaged for Debian/Ubuntu.

A good target could be probably to reach the Ubuntu xenial/Debian stretch that are just release so maybe there is still possibility for this.

Investigate `privilegedStartService` code path

This hasn't been tested at all, there's probably some glitches here.

Selection of "default" certificate for non-SNI clients

Currently, we use txsni, which allows for a DEFAULT.pem, but txacme will try to issue a certificate for DEFAULT when it tries to renew this, which will fail. We should have a better way for specifying which certificate to use for this.

Possibly related to #37 as both of these will require slightly more information than we are fitting into hostnames currently.

Responders passed ChallengeBody objects rather than Challenge objects as interface suggests

The IResponder interface documentation says that for the start_responding() and stop_responding() methods, the challenge parameter should be "The acme.challenges challenge object".

The object passed to those methods in txacme.client.answer_challenges is currently actually the acme.messages.ChallengeBody challenge body.

This hasn't been a problem thus far because (@mithrandi on IRC)...

ChallengeBody has this:
def __getattr__(self, name):
return getattr(self.chall, name)
so mostly you can use the body interchangeably with the actual challenge, but not when calling .encode()

This does, however, cause a problem with the HTTP01Responder as it calls .encode('token') on the challenge it is passed.

Link to *_DIRECTORY constants more prominently

No need for people to type these URLs out in the common (ie. Let's Encrypt) case.