send call sometimes hangs server about node-gcm

[2fours]

Hung again last night...going to try this change in production will let you know.

[goelvivek]

Checking this.

[goelvivek]

Timeout is getting called. you can try this by reducing timeout to 1 ms.

[goelvivek]

But I am facing the same problem related to random hanging. :(

[goelvivek]

Are you getting error message
: possible EventEmitter memory leak detected. 11 listeners added. Use emitter.setMaxListeners() to increase limit. ?

[2fours]

No. Does adding callback("timeout", null) on line 96 fix the issue? Eg.

    post_req.on('socket', function (socket) {
        socket.setTimeout(timeout);
        socket.on('timeout', function() {
            post_req.abort();
            callback("timeout", null);
        });
    });

[goelvivek]

I think it should not make any difference. Did you tried decreasing timeout to 1 ms. As timeout is getting called I think something else is issue.

[2fours]

Yes you might be right, the server doesn't hang when i set timeout to 1 ms.

[goelvivek]

What is your node version ?

[2fours]

Sorry I meant to say the server doesn't hang when timeout is 1 ms. 0.10.22

[goelvivek]

Did you find anything ?

[2fours]

No did you?

[goelvivek]

No.

regards
Vivek Goel

On Fri, Dec 6, 2013 at 7:40 PM, 2fours [email protected] wrote:

No did you?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/36#issuecomment-29996654
.

[2fours]

Wonder if it could be related to keep-alive being true by default: http://nodejs.org/api/http.html#http_class_http_agent

[2fours]

I added more logging to my server and it looks like the hang occurs before it even tries to send. My server code looks like: (link here: https://github.com/surespot/web-server/blob/master/server/cluster.coffee#L1050)

        logger.debug "sending gcms for message"
        gcmmessage = new gcm.Message()
        sender = new gcm.Sender("#{googleApiKey}")
        gcmmessage.addData("type", "message")
        gcmmessage.addData("to", message.to)
        gcmmessage.addData("sentfrom", message.from)
        gcmmessage.addData("mimeType", message.mimeType)
        #pop entire message into gcm message if it's small enough
        if messagejson.length <= 3800
          gcmmessage.addData("message", messagejson)

        gcmmessage.delayWhileIdle = false
        gcmmessage.timeToLive = GCM_TTL

        logger.debug "sending push messages to: #{ids[0]}"
        sender.send gcmmessage, gcmIds, 4, (err, result) ->
          return logger.error "Error sending gcm: #{err}" if err? or not result?
          logger.debug "sendGcm result: #{JSON.stringify(result)}"

          if result.failure > 0
            removeGcmIds message.to, gcmIds, result.results

Right now one of my server processes is hung and the last log entry is "sending gcms for message", not "sending push messages to ..." as one would expect.

[2fours]

Looking at this code for the first time in a while I notice that I'm creating a new Sender for every gcm message I want to send. Wonder if this has anything to do with it. Are you creating a new Sender or re-using one sender?

[goelvivek]

On 08-Dec-2013 10:54 PM, "2fours" [email protected] wrote:

Looking at this code for the first time in a while I notice that I'm
creating a new Sender for every gcm message I want to send. Wonder if this
has anything to do with it. Are you creating a new Sender or re-using one
sender?

No. I am using the same Sender object.

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/36#issuecomment-30086122
.

[goelvivek]

Do you have test case for hanging ? or is it randomly hanging for you ?

[goelvivek]

And can you provide more details. Like:
What are the memory usages and cpu usages at that time ?

[2fours]

No test case, it's random. CPU usage stays the same but memory usage grows until I restart the process.

[goelvivek]

If removing those 3 lines of dealing with setTimeout solves your problem ?

[RomanIakovlev]

Sounds like something is trying to JSON.stringify() some object with cyclic reference. I observed exactly same behavior (no CPU usage, constant memory growth) before, and the reason was JSON.stringify().

Edit: Luckily, there's only one reference to JSON.stringify() in the code, and that is requestBody = JSON.stringify(body); in sender.js. Is there some way to get rid of that call and re-test?

[goelvivek]

can you use util.inspect to log circular JSON object ?

regards
Vivek Goel

On Tue, Dec 10, 2013 at 9:48 PM, RomanIakovlev [email protected]:

Sounds like something is trying to JSON.stringify() some object with
cyclic reference. I observed exactly same behavior (no CPU usage, constant
memory growth) before, and the reason was JSON.stringify().

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/36#issuecomment-30241630
.

[RomanIakovlev]

@2fours:

I added more logging to my server

I had an experience when log output stopped earlier than hitting the line where hang happens, still referring to the case where I had cyclic reference in argument of JSON.Stringify().

[2fours]

I'm not calling JSON.Stringify() before sending the message.

[goelvivek]

@2fours Is your issue solved after latest update ?

[2fours]

I haven't updated yet, I'm not 100% sure it is node-gcm yet. I've added more logging to the server to make sure. Once I'm 100% sure I will upgrade and see if it fixes the issues.

[2fours]

@goelvivek are you using winston by any chance for logging?

[goelvivek]

No. I am not using it.
On 22-Dec-2013 3:39 AM, "2fours" [email protected] wrote:

@goelvivek https://github.com/goelvivek are you using winston by any
chance for logging?

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/36#issuecomment-31073469
.

[2fours]

I've made 2 changes, replaced winston with bunyan, and upgraded node to 0.10.23, and my server has been stable for 2 days. Holding off on calling it "fixed" but it looks promising.

[lordnynex]

I'm experiencing this same issue. I'm noticing the hang on messages that have really weird UTF-8 characters. These are my initial findings (60 minutes in) so take it with a grain of salt. Do you think this is a possibility?

[goelvivek]

@lordnynex
Are you using latest version ?
Was hang happened approx 2014-01-01 19:40:23 GMT ?

[hypesystem]

Hello guys! It's now been quite a while since you last checked this bug. Can you see if it is still present? If so, what are the steps to reproduce?

Sorry to poke you affter so long (I mean, the last comment on here was from 3 Jan 2014), but I'd like to know if this issue persists.

[hypesystem]

No new reports of this problem, no idea if it is still one. Gonna close this.