Replay attack prevention about stripe_event

[invisiblefunnel]

Sorry, accidentally closed.

[invisiblefunnel]

Is there a similar hook that runs after all subscribers have seen the event, so I can ensure that I don't record events where the handler raised an error?

That seems like a good idea. @peterkeen @rmm5t thoughts on this?

[peterkeen]

One way to handle this is to push all of the interesting behavior into background jobs. The subscribers just queue jobs and immediately return success, and retry behavior is then the responsibility of the worker runtime. The event retriever would record the id as having been seen.

That said, I can see an after-all-subscribers hook being useful. Maybe something like StripeEvent.event_recorder that would called immediately after the instrument call.

[invisiblefunnel]

Here's a quick sketch of an after-event hook. Lots to think about before committing to an actual implementation. I'll let this simmer and wait for feedback.

[invisiblefunnel]

Actually, my previous idea was half-baked. @peterkeen's approach is correct. I've updated the branch to fire a finalize event.

[peterkeen]

I'm not sure that's really what we're looking for, since that will propagate to anyone listening for all events, and they're not going to expect two invocations.

Here's more along the lines of what I was thinking of:

module StripeEvent
  class << self
    attr_accessor :event_finalizer

    def instrument(params)
      # ...
      if event
        backend.instrument namespace.call(event[:type]), event
        event_finalizer.call(event) if event_finalizer
      end
    end

  end
end

[invisiblefunnel]

@peterkeen ah you're right, great point. Can you make a pull request?

[peterkeen]

Sure.

[rmm5t]

I commented on the pull-request, but wanted to chime in here too.
#54 (comment)

I'd prefer to see an around hook instead of just an after hook. Afterall, the logical conclusion to adding an after hook is to later also add support for a before hook, so let's just nip that now and skip straight to an around hook that kills two birds with one stone.

Before and after hooks are syntactic sugar for an around hook, but I don't think they're necessary for this gem. This is a fairly more advanced use-case anyway.

[peterkeen]

@rmm5t I disagree. The event retriever + event finalizer give you before and after. You can already do whatever you want in the event retriever, including returning nil to halt the instrumentation.

[rmm5t]

@peterkeen True, the event retriever can be used as a before hook, but I'd consider this a bit of a bastardized means of accomplishing that. The event_retriever has a much different method signature and different intended purpose than both the subscribers and this event_finalizer. The around implementation can maintain the same method signature and accomplish everything an event_finalizer can accomplish, plus much more.

More importantly, because this is about replay-attack prevention, I think there's one or two potential very important use-cases that you lose with just a before and after hook. I'm sure there are other scenarios that are made easier by an around filter too that I can't think of. In general, before and after hooks are not nearly as powerful as one around hook. Consider these scenarios.

1. We're discussing this because we'd like to log when we've successfully handled a particular stripe event so that we don't respond to it again due to a replay attack. What happens when one of the subscribers throws an exception? With the before/after or the around implementation, it might be nice to log the stripe event first and then mark it as complete in the after portion. That way, you can recognize when you're handling a second notification from Stripe (b/c of a Stripe retry), and maybe you want to log how many times a particular event was delivered from Stripe. This is possible in either scenario, but only in the around implementation can you actually rescue the exception and do something with it to determine whether you want Stripe to retry. If we're logging the event first, we might not want Stripe to send us a retry if say 3 out of 4 of our subscribers processed successfully before the exception. Perhaps, it's not possible to make our subscribers idempotent; you can't as easily handle avoiding a retry otherwise.
2. Let's say you wanted to simply print to your logfile the amount of time it takes to process all subscribers for an event. You could do this with a before/after, but you're either stuck using Thread local variables or storing the start time somewhere for the after hook to calculate the total time. With an around hook, this calculation is much easier with just a local variable in the closure.

[peterkeen]

@rmm5t that does sound quite a bit more elegant than the situation #54 creates. A few questions:

1. Would you expect the event_retriever to still do the replay-detection and have the around filter do the bookkeeping? If we expect the event_retriever to always ask Stripe for an event it seems like a DoS vector, but maybe I worry to much.
2. What about a stack of around filters? My experience with ActiveRecord hooks makes me hope that this is a resounding "won't implement".

Regardless of the answers (if there are any to be had), I'd be happy to review a pull request for an around filter.

[rmm5t]

@peterkeen

1. Would you expect the event_retriever to still do the replay-detection and have the around filter do the bookkeeping? If we expect the event_retriever to always ask Stripe for an event it seems like a DoS vector, but maybe I worry to much.

Great question. Yes, I suppose replay-detection would be best done in the retriever. Afterall, it is the "event_retriever".

1. What about a stack of around filters? My experience with ActiveRecord hooks makes me hope that this is a resounding "won't implement".

Just one around filter. A scenario where nested arounds might be beneficial is too rare. The only case where it makes sense is if stripe_event plugins or extensions became a thing.

💥 BUT WAIT A SECOND! 💥

Why are we worrying about this? I think it might be safe to drop this issue entirely, because the replay-attack warning by Stripe might be obsolete...

1. Stripe now requires SSL/TLS for your payment pages, so your server should already handle it.
2. SSL/TLS 1.1 already has built-in protection against replay attacks.
   Source 1: "The SSL/TLS channel itself is protected against replay attacks using the MAC, computed using the MAC secret and the sequence number."
   Source 2: "TLS also provides two additional benefits that are commonly overlooked; integrity guarantees and replay prevention."
   Also See TLS 1.1 specification Appendix F.2

Furthermore, if replay attacks are a serious concern, I think stripe_event should handle this internally. I think there are two potential ways to handle this without requiring a historic log of every event already processed.

1. If Stripe's event object's pending_webhooks value is zero (0), can't we safely assume that we should now ignore this event, because it must be a replay? stripe_event might not be able to check for this in all scenarios, because event_retriever overrides can currently return anything that _quacks_like event[:type]. i.e. This might be the responsibility of customized event_retriever's, but the default event_retriever could also handle it.
2. If an SSL webhook (live mode) arrives, couldn't we just drop any webhook events that don't use SSL/TLS? Maybe this should be a config option to guarantee backwards compatibility, but I'd argue that this should also be an opinionated stance that this gem takes to thwart any possibility of replay attack.

Proposed actions moving forward

Solution 2 (guaranteeing for SSL/TLS) is probably the easiest and cleanest to maintain; however, I'm not sure off the top of my head whether a Rails controller can recognize which SSL transport protocol was used. Does anyone else know?

Also, any security experts willing to chime in to make sure I'm not bonkers here?

---

TL;DR: An around hook might still be a useful thing, but I'm not sure any additional hooks are necessary to prevent replay attacks. SSL/TLS should automatically handle this for us.

[rmm5t]

UPDATE: I referenced and brought this particular issue about replay attacks to the Stripe API Discussion mailing list.
https://groups.google.com/a/lists.stripe.com/forum/#!topic/api-discuss/-zqruS0lTig

Hoping someone from Stripe will chime in to verify (or refute) my analysis above.

[rmm5t]

Reply from Richo@Stripe:

The replay attack you describe TLS protecting you from is at the protocol layer, in which a malicious actor in a privileged network position replays frames and causes the same TLS record to be processed twice. You're correct in that you're already protected.

The attack we're referring to, is one in which an attacker has a priori knowledge of your webhook endpoint, and is able to capture a webhook payload. A (contrived) real world example of this would be:

• Your system handles the process of issuing license keys by sending an email to a user on charge.successful
• A malicious user is able to capture the body of the webhook for their payment
• If they're able to send the hook repeatedly to your servers, you will issue them an unlimited number of license keys

You can protect yourself from this eventuality in two complimentary ways:

• By protecting your webhook endpoint by making it's route obscure, and including (and verifying) a token as a GET parameter.
• By asserting that a given webhook is only ever processed once.

The second part of this would require access to some form of persistence, so it's not clear to me that stripe_event can handle this in an opaque fashion, but I'm very happy to discuss further if any of this isn't clear.

[rmm5t]

Since it's going to be difficult for stripe_event gem itself to fully protect against the extremely rare and difficult chance that a replay attack happens (especially with TLS), I like the simple solution of just recommending that applications obscure their webhook endpoint.

[beccadax]

Thanks for responding to this so enthusiastically, everyone!

@rmm5t:

I like the simple solution of just recommending that applications obscure their webhook endpoint.

There's a saying about obscurity and security.

An attacker with your source code in hand should not be able to mount an attack. An attacker with complete knowledge of your app except for passwords, API keys, and other designated secrets should not be able to mount an attack. And secrets should not be logged, ever.

With TLS in place, a replay attack requires that you know the following:

• The route to the Stripe event endpoint.
• Any extra GET parameters required on the Stripe event endpoint.
• A Stripe event ID that triggers the desired behavior. (You don't need a whole Stripe event object; stripe_event seems to read the ID and ignore the rest.)

So, which component of that is going to be secret?

• Routes are in the routes.rb file. That means it's in the non-secret source code unless you do something like this, which I don't think I've ever seen done:

  mount StripeEvent::Engine, at: "/stripe/#{Rails.application.secrets.stripe_event_key}"
  

  Even if you do that, your alleged "secret" is going to be smeared all over the log files.

• An extra GET parameter could easily be made secret. However, if that's going to be the recommended approach, we should explicitly recommend it and build in a feature to do it. (i.e. "set this attribute to a value and stripe_event will reject any request that doesn't have ?key=your-value. Oh, and don't forget to update filter_parameter_logging.rb.")

• There are a lot of valid Stripe event IDs. Ideally, you don't want to protect that many secrets. And even if you decide to do that, are you really never going to log an event ID?

That's why I think we should have some way to prevent replay attacks. Replay attack prevention doesn't rely on any additional secrets.

If the pending_webhooks solution works, great, let's integrate that and forget about the whole thing. If not, let's get the right hooks in place to do it the hard way.

(If we do integrate pending_webhooks, though, we'll need a way to turn it off in development. I don't know about you guys, but the only way I make any progress when I'm debugging my webhooks is by replaying events over and over until the damn thing works.)

[rmm5t]

An extra GET parameter could easily be made secret.

Exactly. That's how I would suggest implementing this, but I should have been more specific. Check for a special param value in your webhook endpoint and configure rails to never log this parameter. Don't store this secret in your source code (treat it like an internal API key/secret).

However, if that's going to be the recommended approach, we should explicitly recommend it and build in a feature to do it. (i.e. "set this attribute to a value and stripe_event will reject any request that doesn't have ?key=your-value.)

I could get behind that. I might suggest a GET param name of secret, and a config attribute of StripeEvent.secret that defaults to ENV["STRIPE_WEBHOOK_SECRET"]. I'd also suggest adding :secret to Rails.application.config.filter_parameters automatically.

Drop all webhooks that don't match the secret and log an Error. It might also be nice to log a Warning when a webhook arrives but StripeEvent.secret is unset -- encouraging existing applications using stripe_event to up their game.

That's why I think we should have some way to prevent replay attacks. Replay attack prevention doesn't rely on any additional secrets.

Keep in mind, replay attacks shouldn't be as much of a concern if you ensure that your webhook subscribers are idempotent. I realize that's easier said than done sometimes, but in truth, you're application will always have vulnerable and/or buggy vectors otherwise.

Lastly, wouldn't all of this be easier if Stripe just made webhook requests with a client SSL certificate?

If the pending_webhooks solution works

Yeah, even though I suggested it, I haven't yet been able to convince myself that this is a 100% solution yet. A stripe account can have many webhook endpoints, and many Stripe Connect applications. No one webhook endpoint has enough control to zero-out that pending_webhooks value by itself which potentially makes it somewhat unreliable.

Overall, I'm really just trying to find solutions that avoid storing every processed Stripe Event ID, because I'd prefer to have a solution that stripe_event fully handles instead of requiring every application to re-invent the wheel just to keep things secure.

If we do integrate pending_webhooks, though, we'll need a way to turn it off in development. I don't know about you guys, but the only way I make any progress when I'm debugging my webhooks is by replaying events over and over until the damn thing works.

Ahem. That's what a test suite is for. And I prefer to never allow my test suites to touch the network. stripe-ruby-mock is a good tool to help with that.

[invisiblefunnel]

StripeEvent could also support HTTP basic auth. Then webhook urls could be registered as https://<username>:<password>@www.example.com/obscured/path. Rails doesn't log HTTP basic auth creds.

[rmm5t]

StripeEvent could also support HTTP basic auth. Then webhook urls could be registered as https://:@www.example.com/obscured/path. Rails doesn't log HTTP basic auth creds.

That's a good idea. Does Stripe support sending username/password for basic auth creds in webhook URLs?

[beccadax]

Just tried it out, and if you add "username:password@" to your URL as you normally would, Stripe does indeed send an Authorization: Basic [base64 garbage] header.

[rmm5t]

@brentdax Perfect. Thanks for checking on that. I also have an open
question to Stripe to ensure they'll maintain this feature, because I dont
think they've documented support for this.

[rmm5t]

Good news. Stripe just made a commitment to guaranteeing basic-auth credentials in webhook endpoints:

I've checked with the team on this, and we can confirm that providing
HTTP basic auth credentials in the webhook URL (a url like
https://username:[email protected]/stripe/webhooks, in your
example) is a feature that we will support. I've confirmed that we
have existing live webhook endpoints that rely on this, and I've added
a regression test to ensure that we don't break our support for it.
-- Jim@Stripe

[invisiblefunnel]

Thanks for looking into this @rmm5t! Does basic-auth support address the original concerns, @brentdax? Would anyone like to take the lead on a pull request?

[beccadax]

@invisiblefunnel Combined with HTTPS (to protect the basic auth password in transit and prevent byte-for-byte replay attacks), it should ensure that requests that aren't from Stripe are rejected. In fact, if you use HTTPS + basic auth, I'm not sure that retrieving the event from Stripe is necessary at all—every request is guaranteed to have come from a source you trust not to maliciously replay events.

It might be worth keeping it in just to limit the damage if your basic auth password is compromised, though. And in the same spirit, it might make sense to include the pending_webhooks check suggested by @rmm5t.

In any case, a suitable combination of these features would satisfy me that recording and rejecting already-processed events isn't necessary. Which is great, because I wasn't looking forward to keeping a table full of used event IDs!

[rmm5t]

if you use HTTPS + basic auth, I'm not sure that retrieving the event from Stripe is necessary at all

Stripe still suggests and recommends that we continue to re-fetch the event object, because of API versioning. By re-requesting the event object, you can better specify and/or guarantee what version of the API you want to support for the event object and the underlying associated data object.

It might make sense to include the pending_webhooks check suggested by @rmm5t.

The more I thought about this, if we do check the pending_webhooks value (with the default event_retriever), maybe we should only log a warning message. I'm fearful of some manual webhook retry use-cases that might cause backwards compatibility confusion with people who use this gem. Nonetheless, I would consider this a separate issue for now.

I wasn't looking forward to keeping a table full of used event IDs!

Neither was I. 😝

[rmm5t]

Also, I liked the the direction of @invisiblefunnel's authenticate_with_http_basic support in #53 (comment).

• Should we also log a Warning message if you don't implement it?
• Alternatively, should we just instead provide a default implementation that looks for an env variable (like ENV["STRIPE_WEBHOOK_PASSWORD"] ... we can just ignore the username)? Under this scenario, we might only log a Warning message if that ENV variable (or an override password setting) isn't defined?

[beccadax]

I've added some, um, basic support for basic authentication in pull request #55.

[rmm5t]

Thanks everyone. We just merged #55 with new basic auth secret support. I think it's safe to close this out too.