balanced / status.balancedpayments.com Goto Github PK

Currently if whatever is feeding updates to the app goes down this will not be reflected. Consider integrating an external health check to augment and fix this.

There is a cron job that fires internally from Google App Engine every two minutes that fetches tweets from @balancedstatus and displays them on the status page. The the url that is called is:

/twitter?_method=post

Which in the following class in the code:

TwitterHandler

For some odd reason we are getting Rate Limit exceeded. This shouldn't happen though, since we are not firing a lot of requests at twitter. Something is debunk either in the code, or in the 3rd party library we are using tweepy.

    <pre>Traceback (most recent call last):
  File &quot;/Users/justin/Desktop/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.1/webapp2.py&quot;, line 1536, in __call__
    rv = self.handle_exception(request, response, e)
  File &quot;/Users/justin/Desktop/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.1/webapp2.py&quot;, line 1530, in __call__
    rv = self.router.dispatch(request, response)
  File &quot;/Users/justin/Desktop/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.1/webapp2.py&quot;, line 1278, in default_dispatcher
    return route.handler_adapter(request, response)
  File &quot;/Users/justin/Desktop/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.1/webapp2.py&quot;, line 1102, in __call__
    return handler.dispatch()
  File &quot;/Users/justin/Desktop/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.1/webapp2.py&quot;, line 572, in dispatch
    return self.handle_exception(e, self.app.debug)
  File &quot;/Users/justin/Desktop/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/lib/webapp2-2.5.1/webapp2.py&quot;, line 570, in dispatch
    return method(*args, **kwargs)
  File &quot;/Users/justin/Sites/status.balancedpayments.com/situation/main.py&quot;, line 105, in get
    self.post()
  File &quot;/Users/justin/Sites/status.balancedpayments.com/situation/main.py&quot;, line 120, in post
    self.tweet_manager.run()
  File &quot;/Users/justin/Sites/status.balancedpayments.com/situation/tweeter.py&quot;, line 88, in run
    tweets = self._get_tweets(last_check)
  File &quot;/Users/justin/Sites/status.balancedpayments.com/situation/tweeter.py&quot;, line 82, in _get_tweets
    return self.twitter.user_timeline(**filters)
  File &quot;/Users/justin/Sites/status.balancedpayments.com/situation/tweepy/binder.py&quot;, line 185, in _call
    return method.execute()
  File &quot;/Users/justin/Sites/status.balancedpayments.com/situation/tweepy/binder.py&quot;, line 168, in execute
    raise TweepError(error_msg, resp)
TweepError: [{u'message': u'Rate limit exceeded', u'code': 88}]

I just see that it appears the twitter API keys are exposed here

https://github.com/balanced/status.balancedpayments.com/blob/master/situation/settings.py

# Generate these for your app at https://dev.twitter.com/apps/new
TWITTER = {
    'AUTH': {
        'consumer_key': 'xxx',
        'consumer_secret': 'xxx',
        'token': 'xxx',
        'token_secret': 'xxx',
    },
}

I am not a heavy twitter user, I have no twitter application development experience, not sure what kind of key they are. however, I see the twitter documents say

Tokens are passwords

Keep in mind that the consumer key & secret, bearer token credentials, and the bearer token itself grant access to make requests on behalf of an application. These values should be considered as sensitive as passwords and must not be shared or distributed to untrusted parties.

https://dev.twitter.com/docs/auth/application-only-auth

I think maybe anyone with that key can post message with your twitter account or do something worser? If it is, we should better regenerate new tokens, keep the real key in production environment only (load from another setting file for production).

Establish an acceptable level of latency and post updates if this value is exceeded by x percentage.

Between 9:23am PST and 10:28am Balanced ran into issues with a locked database table which affected all card processing transactions. This was due to an oversight in new code that was issuing inefficient database queries, causing a request backlog.

Fixing this issue took longer than anticipated since there were some runaway processes that were causing high load on our database machines which did not respond to our usual application tooling.

Balanced has reached out to all affected customers directly via email to ensure that they are aware of any transactions or customers that may require attention.

In order to mitigate these issues in the future Balanced is implementing the following:

• Segregating our application environments to reduce the chances of issues occuring to processing customers.
• Changing the way our application handles requests to transparently handle transient errors.

To help our customers be aware of these issues Balanced will:

• Implement an SMS and email based notification service which will allow you to proactively be notified of any potential issues in the future. balanced/balanced-api#39

Any transactions that were affected will be discussed with the affected parties before being processed.

This was not related to our outage on Thursday however; it has highlighted a flaw in our our deployment infrastructure and we've prioritized its overhaul.

Our twitter (https://twitter.com/balancedstatus), status page (https://status.balancedpayments.com), and IRC room (#balanced on freenode) are always up to date and available in these situations.

Sometimes the API/dashboard isn't offline, but it's under load and responding slowly. It would be useful to see the avg response times on the status page over some period of time.

There have been two "false alarm" notifications two days in a row.

@nodesocket What's the cause and how can this be corrected?

https://twitter.com/balancedstatus/status/375042764376797184

https://twitter.com/balancedstatus/status/375447307526754304

When tweeting from @balancedstatus, the tweet should be sent via SMS/email to everyone that subscribed to updates for that service.

For example, the following tweets should have been pushed out as notifications, but they weren't:

@nodesocket: you talked about working on this. What's your progress, and what's blocking you from finishing this? This is important functionality in communicating service availability with marketplaces.

Right now, we include dependencies for the GAE in a virtualenv, when we should be zipping up those imports and importing them on app.

We're also symlinking in a bunch of places, we should clean that up.