Frequent timeouts, ECONNRESET, and ECONNREFUSED on Node 0.12.7 about parse-sdk-js

[TylerBrock]

I'm having the same issue. I filed a bug report here if you want to add additional input or subscribe.

I'm running Node 4.x and version 1.5 of the JS SDK so it is not unique to your version of Node.

I would agree that is is exceedingly difficult to get a hold of anyone to support these issues (especially as a paying customer of Parse). There needs to be a better way of reporting production issues and receiving support than getting funneled into the bin of Facebook "bugs" that happen to be tagged as Parse issues.

[bblurock]

We are also facing this ECONNRESET issue especially while performing saving , we use latest Parse JS SDK 1.6.7 with NodeJS 4.0.0 and we're hosting on Heroku as well.

We've test the saveAll API with a very simple script which generates 3000 records with timestamp and splice into 100 records per batch saving.

_parse.Cloud.useMasterKey();

var i, pages, promise;
var testPrototype = _parse.Object.extend("test");
var perBatch = 100;
var data = [];

for (i = 0; i < 3000 ; i++ )
{
    var test = new testPrototype();
    test.set("ts", (new Date).getTime());
    data.push(test);
}

// Calculation of total pages
pages = Math.floor(data.length / perBatch);
pages = (data.length % perBatch) > 0 ? pages + 1 : pages;
promise = _parse.Promise.as(0);

for (i = 0; i < pages ; i++)
{
    promise = promise.then(function(k)
    {
        var spliceAmount = data.length > perBatch ? perBatch : data.length;
        var dataToSave = data.splice(0, spliceAmount);

        return _parse.Object.saveAll(dataToSave).then(
            function(objs)
            {
                console.log("Saved Page. " + k);

                return _parse.Promise.as(k+1);
            },
            function(e)
            {
                console.log(JSON.stringify(e));

                return promise.reject("Save error");
            }
        );
    });
}

On Heroku, every time when it saved around 17xx records, the script will be abruptly shut down without catching any error response in the error promise.

Here is the error message I got from node-inspector:

unhandled res:{"seq":1458,"type":"event","success":true,"running":true,"event":"Network._requestWillBeSent","body":{"requestId":"103"}}

unhandled res:{"seq":1459,"type":"event","success":true,"running":true,"event":"Network.loadingFailed","body":{"requestId":"103","timestamp":1445575.700141,"type":"XHR","errorText":"ECONNRESET","canceled":false}}

unhandled res:{"seq":1460,"type":"event","success":true,"running":true,"event":"Network._loadingFailed","body":{"requestId":"103"}} 

[TylerBrock]

Thanks @bblurock I would encourage you to subscribe to the Facebook bug report so that this gets greater visibility. It doesn't look like we will get any response here.

[TylerBrock]

@bblurock, please post your repro in the but report ticket so that they look at it.

[bblurock]

@TylerBrock Sure. I'll post to the report ticket later.

As for my particular example script. I further dived in to its behaviour on parse.

I've found out every promise of saveAll resolved almost immediately(1~10 millisecond) causing my request rate limit exhausted in few seconds, which never happened in my local environment.

Hence, I made a workaround with setTimeout. If the saveAll promise resolved latency is smaller than 1 second(I'm using the Parse free plan, 30 requests/second), I let it sleep the remaining time until my script fire another saveAll request.

The saving can finish without any ECONNRESET for now.

I'm not sure if your case fit this. Dose your rate limit exceed when you received the ECONNRESET error?

_parse.Cloud.useMasterKey();

    var i, pages, promise;
    var testPrototype = _parse.Object.extend("test");
    var perBatch = 20;
    var data = [];

    for (i = 0; i < 3000 ; i++ )
    {
        var test = new testPrototype();
        test.set("ts", (new Date).getTime());
        data.push(test);
    }

    // Calculation of total pages
    pages = Math.floor(data.length / perBatch);
    pages = (data.length % perBatch) > 0 ? pages + 1 : pages;
    promise = _parse.Promise.as({"index": 0, "ts": (new Date).getTime()});

    for (i = 0; i < pages ; i++)
    {
        promise = promise.then(function(k)
        {
            var spliceAmount = data.length > perBatch ? perBatch : data.length;
            var dataToSave = data.splice(0, spliceAmount);
            var pagePromise = new Parse.Promise();

            Parse.Object.saveAll(dataToSave).then(
                function(objs)
                {
                    var ts = (new Date).getTime();
                    var diff = ts - k.ts;
                    // I'm using the free plan of parse, 30 request/per second
                    var threshold = 1000;

                    if (diff < threshold)
                    {
                        var toSleep = threshold - diff;

                        try
                        {
                            setTimeout(function(){
                                pagePromise.resolve({"index": k.index+1, "ts": (new Date).getTime()});
                            }, toSleep);
                        }
                        catch (e) {
                            console.log(JSON.stringify(e));
                        }
                    }
                    else {
                        pagePromise.resolve({"index": k.index+1, "ts": (new Date).getTime()});
                    }
                },
                function(e)
                {
                    console.log(JSON.stringify(e));
                    return promise.reject("Save error");
                }
            );

            return pagePromise;
        });
    }

[TylerBrock]

I did the same thing :-), however I don't think this is a symptom of rate
limiting as we get these errors on single requests with zero other load.
On Sun, Oct 25, 2015 at 6:27 AM bblurock [email protected] wrote:

@TylerBrock https://github.com/TylerBrock Sure. I'll post to the report
ticket later.

As for my particular example script. I further dived in to its behaviour
on parse.

I've found out every promise of saveAll resolved almost immediately(1~10
millisecond) causing my request rate limit exhausted in few seconds, which
never happened in my local environment.

Hence, I made a workaround with setTimeout. If the saveAll promise
resolved latency is smaller than 1 second(I'm using the Parse free plan, 30
requests/second), I let it sleep the remaining time until my script fire
another saveAll request.

The saving can finish without any ECONNRESET for now.

_parse.Cloud.useMasterKey();

var i, pages, promise;
var testPrototype = _parse.Object.extend("test"

);
var perBatch = 20;

var data = [];

for (i = 0; i < 3000 ; i++ )
{
    var test = new testPrototype();
    test.set("ts", (new Date).getTime());
    data.push(test);
}

// Calculation of total pages
pages = Math.floor(data.length / perBatch);
pages = (data.length % perBatch) > 0 ? pages + 1 :

pages;
promise = _parse.Promise.as({"index": 0, "ts": (new Date).getTime()});

for (i = 0; i < pages ; i++)
{
promise = promise.then(function(k)
{
var spliceAmount = data.length > perBatch ? perBatch : data.length;
var dataToSave = data.splice(0

, spliceAmount);
var pagePromise = new Parse.Promise();

        Parse.Object.saveAll(dataToSave).then(
            function(objs)
            {
                var ts = (new Date).getTime();
                var diff = ts - k.ts;
                // I'm using the free plan of parse, 30 request/per second
                var threshold = 1000;

                if (diff < threshold)
                {
                    var toSleep = threshold - diff;

                    try
                    {
                        setTimeout(function(){
                            pagePromise.resolve({"index": k.index+1, "ts": (new Date).getTime()});
                        }, toSleep);
                    }
                    catch (e) {
                        console.log(JSON.stringify(e));
                    }
                }
                else {
                    pagePromise.resolve({"index": k.index+1, "ts": (new Date).getTime()});
                }
            },

function(e)
{
console.log(JSON.stringify(e));
return promise.reject("Save error"

);
}
);

        return pagePromise;
    });
}

—
Reply to this email directly or view it on GitHub
#64 (comment)
.

[kevinwritescode]

Also, while this may be a platform issue, I think there is a bug here in that Parse JSON cannot properly handle the ETIMEOUT, ECONNRESET, or ECONNREFUSED responses and instead reutnrs Parse error 107 of invalid JSON. Shouldn't it ideally return err code 100? Maybe this was already addressed in 1.6.7.

http://parse.com/docs/dotnet/api/html/T_Parse_ParseException_ErrorCode.htm

[andrewimm]

I'll look into this issue to see if there's a problem with the SDK or the XMLHttpRequest polyfill we use in Node.js. If we can't trace it to the SDK, though, you guys are correct that we will close this issue. We want to keep GitHub issues tied to the client SDKs. For backend / platform bugs, the Facebook Developers bug reporting flow is much better -- we have a dedicated team that helps reproduce and triage bugs, and we can collect information from individuals confidentially (for instance, we need real app ids to investigate this particular bug). Thanks for providing the information you already have there, we're actively investigating.

As far as I can tell right now, the only action that we can take on the SDK is to make sure errors from the xmlhttprequest third party module we use are translated to the appropriate client error codes, rather than getting grouped into "107: invalid JSON" errors. However, I think it's odd that this error only occurs on Node environments, so I'll do my best to dig into the code of xmlhttprequest and see if it might be doing some sort of inefficient socket handling that would affect a node client's ability to connect to Parse.

[TylerBrock]

Thanks @andrewimm. It would be nice if the errors mapped correctly so that we can hypothetically handle them when they occur.

The underlying issue however, as you mention, is probably not SDK related and we have provided all related Application IDs to Facebook/Parse in the bug report ticket. I would be happy to help reproduce it with Parse/Facebook if you want. It is killing our reliability right now.

[andrewimm]

Given the reproducible case above, I'm curious to see whether it's related to memory usage of node-xmlhttprequest. Did these issues start recently (without code changes), or are they tied to recent increases in outgoing requests?

If they started recently, I'd be looking at an Amazon-related explanation (Parse, Heroku, and Modulus are all AWS-hosted). Otherwise, my guess is that it's a memory issue in node-xmlhttprequest

[TylerBrock]

Hi Andrew, from what I can tell our memory usage doesn't fluctuate at all. In our case it fails to connect, resets, and times out when it is the only singlular request going through our entire system.

We've had dozens of requests fail on our single dyno today and I've attached the memory usage graph for it (to summarize 76-78mb being used out of 1gb -- less than 1/13th the capacity of the dyno is being used.

I'm not seeing this as a memory constraint issue at all.

[kevinwritescode]

It's difficult to say for my scenario.

We upgraded a new release with Parse 1.6.7 the day Parse launched the Parse + Heroku announcement. We started getting memory alert issues, but that may have been related to how we were still working with currentUser() in 1.6.7 and Client.User.enableUnsafeCurrentUser();

After seeing memory issues, I rolled back to our stable build from 14 days prior (Parse 1.5.0 and without various currentUser() changes) and we started to notice the ECONNRESET and ECONNREFUSED. Unfortunately, our traffic had also increased by 50% from a marketing push on the same day so it became very difficult to isolate if the cause was our load or an issue in Parse.

I then noticed Tyler and others announcing the same issue that day. Our memory stays at max swap of 25MB and Max total of 500MB out of a 1GB limit. Our dyno load is > 1%.

[TylerBrock]

We have a different scenario: Parse SDK 1.5, Node 4.3.1, and virtually zero load. We've seen it under high spikey load as well but also under zero load like this morning/afternoon.

[TylerBrock]

@andrewimm would you mind poking the other folks involved to see if they can look at the ticket again. We are dropping a large percentage of our requests in production here and our ticket isn't getting any attention. We'd appreciate any help you can provide. I will do whatever you all need in order to help and provide information.

[andrewimm]

@TylerBrock I can assure you that we're actively investigating this issue, and that no poking is necessary.

[TylerBrock]

ok thanks!

[andrewimm]

Closing out as this has been resolved.

[TylerBrock]

@andrewimm do you want me to make a new ticket for properly handling the connection failures and translating them to a promise rejection rather than an exception?