nervous-systems / eulalie Goto Github PK

We'd have to stop assuming js/Buffer is available, mostly, and expose the XML parsing stuff in a way which would work in-browser.

Mostly by cleaning up the response XML

Description

AWS did introduce X-Ray few weeks ago (maybe months now). It would be nice to have X-Ray integrated into libraries like Hildebrand, ... These libraries are based on eulalie, so, it probably makes sense to add it here.

AWS-SDK

var ddb = AWSXRay.captureAWSClient(new AWS.DynamoDB());

You can use ddb as DynamoDB client and it will track all requests. More details.

HTTPS

var AWSXRay = require('aws-xray-sdk');
var http = AWSXRay.captureHTTPs(require('http'));

You can use http as Node's http and it will track all requests. More details.

Custom

Or you can use custom subsegments to provide whatever you want.

Discussion

I would like to have X-Ray support in your libraries. One option is to use aws-sdk directly, wrap it with X-Ray SDK and case closed. But this will lead to lot of changes everywhere.

Another option is to add X-Ray support to the eulalie directly (issuer, request, response). Sample subsegment for DynamoDB:

{
  "id": "24756640c0d0978a",
  "start_time": 1.480305974194E9,
  "end_time": 1.4803059742E9,
  "name": "DynamoDB",
  "namespace": "aws",
  "http": {
    "response": {
      "content_length": 60,
      "status": 200
    }
  },
  "aws": {
    "table_name": "scorekeep-user",
    "operation": "UpdateItem",
    "request_id": "UBQNSO5AEM8T4FDA4RQDEB94OVTDRVV4K4HIRGVJF66Q9ASUAAJG",
  }
}

Any thoughts? Any plans about aws-sdk integration? About X-Ray?

e.g. https://gist.github.com/mattdeboard/b1e82bd022157ab6c365

Using the latest version of core.async, I get the following error: Can only recur from tail position

This is browserified: https://github.com/SheetJS/js-crc32

Not 100% clear to me how the module situation is going to work - I guess if we included the above in the build as an entry in :foreign-libs, we could just take its presence for granted.

It (sometimes at least) seems to not be XML

They fail, it's confusing.

https://github.com/feross/buffer

platform.cljs is where most of the action is.

I think this might be a good candidate for a small external library - figure out if we're running on Node, or in-browser and either require('http') or use XHR. I like the idea of using this feature to nail down the approach to runtime platform checks and setting up parallel builds.

First, I'm going to apologise because this issue report is going to be kinda woolly 😸

We're using fink-nottle with SQS. We're finding that every now and then (say, once every couple of weeks), one of our instances seems to get stuck at 100% CPU usage and needs a restart. I'm using the hprof agent to try and take some CPU samples and see what's stuck looping here, and early indications are that this may be a problem in eulalie.

I captured some CPU samples over a short period when the problem was ongoing, here are some snapshots:

https://www.refheap.com/112686

https://www.refheap.com/112687

https://www.refheap.com/112685

I see a lot of:

• eulalie issue-request
• eulalie handle-result
• XML parsing (related to handle-result?)
• camel-snake-kebab (related to handle-result?)

I can't easily see what SQS communication is going on here because we're currently connecting via https. I'll dump some information when I can capture this - I'll need to switch to the http endpoint then wait for this to happen again 😄

So I totally introduced this one, but I'm not really sure how to deal with it 😕

Right now if you pass a create-function request, using the :zip-file parameter to push a base64 encoded string up, and it errors (e.g. a bad arn), it'll print out that string a couple of times. For my quick hacked together test of a lambda function that uploads itself, that resulted in printing just under six millions characters. (It's in both the request and body bits of the error message)

So I would like to exclude it from the error message, if possible (I'm assuming in general these will be logged somewhere, and a base64 encoded zip isn't amazingly useful as a debugging aid, especially since you could just dump it directly to disk?).

Since it's in an already encoded string the only thing I can think off the top of my head to deal with it is a regex which seems like a bad idea. I'm looking for other ideas of how to remove it, if anyone has any.

Thanks in advance!

src/eulalie/platform/crypto.cljs

Hi @moea I see that things have changed quite a bit in eulalie regarding expiring credentials. Originally we referred to nervous-systems/fink-nottle#4 and used the function suggested there, but we were finding that that our service was becoming broken after a few hours when credentials expired. It now looks like eulalie.creds/iam is the correct way to build expiring credentials, however on switching to this method we've found that we can no longer make successful requests.

Using fink-nottle 0.4.2 we're declaring our creds like this:

(def creds (-> (eulalie.creds/iam) (assoc :endpoint endpoint)))

However any attempt to receive a message from SQS results in the following exception:

Exception in thread "main" clojure.lang.ExceptionInfo: invalid-client-token-id: The security token included in the request is invalid. {:type :invalid-client-token-id, :message "The security token included in the request is invalid."}
    at clojure.core$ex_info.invoke(core.clj:4593)
    at eulalie.support$error__GT_throwable.invoke(support.cljc:17)
    at eulalie.support$issue_request_BANG_$fn__15759$state_machine__9420__auto____15760$fn__15762.invoke(support.cljc:29)
    at eulalie.support$issue_request_BANG_$fn__15759$state_machine__9420__auto____15760.invoke(support.cljc:22)

Using a repl running on this instance I've tried to replicate what eulalie will be doing by running the following:

(<!! (eulalie.creds/creds->credentials (iam)))

when I do this I see what looks like a valid set of credentials (I've truncated the sensitive values):

{:eulalie/type :expiring,
 :current
 #<Atom@4ffe91e4:
   {:code "Success",
    :last-updated "2015-08-14T08:53:32Z",
    :type "AWS-HMAC",
    :token
    "AQoDYXdzEOr//////////wEa0AM58//iKa1kwziRDN...",
    :expiration 1439564263000,
    :access-key "AS...",
    :secret-key "lE..."}>,
 :refresh
 #object[eulalie.instance_data$default_iam_credentials_BANG_ 0x70d2bad7 "eulalie.instance_data$default_iam_credentials_BANG_@70d2bad7"],
 :threshold 300000}

So I'm wondering if there is something wrong with the way that eulalie is attempting to sign requests.