cloud-gov / cf-domain-broker Goto Github PK

Currently, we take great pains to ensure that we can restart the certificate registration process without impacting the user (ie: cert registration is idempotent):

1. We re-implement the Obtain() entry point from the lego library.
2. We checkpoint the instance state (read, check, and write the instance info to DB) aggressively throughout.
3. We implement the virtual actor pattern across all managers to enable complete async behavior.

This adds a lot of complexity, which translates to difficulty in maintenance and increased time to debug issues. Since issues in this broker's operation can result in real downtime for our end users, I'd like to explore ways we can reduce this complexity, and hopefully the time it would take to fix any such issues.

To that end: What would be the expected impact on the end user if we didn't implement idempotent certificate registration? Here's what I'm imagining, but I'm hoping this starts a discussion around the idea in the comments.

Scenario 1:

Obtain() is restarted before the user sees the DNS instructions.

1. 👩 User creates instance
2. Provisioning begins
3. ACME validation token (A) is generated
4. 💥 Broker is restarted
5. Provisioning restarts
6. New ACME validation token (B) is generated
7. 👩 User gets service status and sees instructions to update DNS with TXT record including B
8. 👩 User updates DNS
9. Broker verifies DNS
10. ACME provisioning proceeds as normal

I believe this is pretty straight forward -- the user never notices. I also wouldn't bet on this being a more likely path. The bigger issue is...

Scenario 2:

Obtain() is restarted after the user sees the DNS instructions.

1. 👩 User creates instance
2. Provisioning begins
3. ACME validation token (A) is generated
4. 👩 User gets service status and sees instructions to update DNS with TXT record including B
5. 💥 Broker is restarted
6. Provisioning restarts
7. New ACME validation token (B) is generated
8. 👩 User updates DNS
9. Broker fails to verify DNS 🙅
10. ACME provisioning stalls

Either one of two paths would then happen:

1. User gets status on instance, and we can report that there is a TXT record, but that it doesn't match the expected token (B). We can explain that this might be due to a restart on our end.
2. User doesn't bother checking the instance status, 24hrs pass, and we kill the instance entirely.

This isn't great by any means, but it's also not catastrophic.

Thoughts?

Are there other scenarios I'm not considering?

Are there other impacts in the above scenarios?

What's even the likelihood we'll hit either of these? (maybe we'll be deploying somewhat often, but we won't be provisioning new certs that frequently)

How much could we mitigate the impact on the user via documentation, clear error messages and our own support time?

How savvy are the users who'll be provisioning instances of this service? Is it primarily Federalist? If so, can we reach out to educate them of this issue beforehand?

Thanks for reading all of the above, and thanks in advance for any thoughts and ideas you can contribute below. I'll update the above with any new information from the comments.