How to run with nightmarejs ? about node-headless

[mateuszmazurek]

same problem here

[mateuszmazurek]

@GautierT use xvfb package instead, look at this comment: segment-boneyard/nightmare#602 (comment)

[GautierT]

@mateuszmazurek : Thanks ! Just got it working this morning with xvfb !
But as soon as i call the script multiple time some of them fail with timeout error.
Did u try running multiple script where each one have his own display on the same machine ?

[mateuszmazurek]

@GautierT currently I have one Xvfb process started by xvfb-run and one script where i use xvfb module = 2 displays in total (:99 permanently and :100 when running nightmare inside script). They work ok. That's all i can say at this moment.

[GautierT]

Okay thanks !
3 days that i try to understand how to launch multiple nightmare script on the same machine without timeout/crash... :(

[mateuszmazurek]

How many instances you need to run to make it crash? 2 or more? Did you try to use reuse option?

[GautierT]

When i launch the script 4 or 5 times some of then will crash (Last line of nightmare log : nightmare:log crashed [{},false])
And after the timeout delay nightmare send me a timeout error : Error: .wait() timed out after 180000msec
And i think that when one of the script crash all electron process are killed so the other running script stop too...

[mateuszmazurek]

Hmm... Maybe you have no enough RAM? Why do you need to run this script multiple times in parallel?

[GautierT]

Because it's a script launch by user when the visit a url of my website. And multiple user can access this url at the same time.
Probably a ram problem yeah..
I don't see any other alternative to handle this case. I need this script launched when an user access the page to send him back some screenshot and files...

[mateuszmazurek]

Can't you just launch this script once and there create http server (maybe with express) which will create new nightmare instance for every user request?

[GautierT]

That's what i do ! I have an expressjs server running. And when someone go to /api/scripts/script_name a function creating a new Xvfb() and a new Nightmare() is launched. But if i go to this url 5 times at the same time some of the script will never finish (timeout or crash) in no particular order. 4 times : sometimes all script finish well sometimes not !

Thanks for trying man ! I'm feeling less alone !

[mateuszmazurek]

Try to run you whole process inside Xvfb using xvfb-run tool without using node-xvfb module inside your script. If it won't help (95% sure it won't because it's simply problem with lack of RAM, not with node-xvfb) you should queue your user's requests. To do so you can use promise-queue package.

Example:

// at the top of your script
const Queue = require('promise-queue');
const queue = new Queue(1, Infinity); // change 1 with the number of concurrent requests you want to handle

...

// inside express request callback

queue.add(() => {
    const nightmare = new Nightmare({
        //show: true,
        webPreferences: {
            partition: 'nopersist'
        }
    });
    console.log('nightmare started')
    return
        nightmare.goto('http://www.google.fr')
        // ..... other instruction....
        .then(() => {
            // do sth
        });
});

[GautierT]

I will try that ! Thanks !
Why do you think it's better to run the whole nodejs/expressjs with xvfb-run instead of using node-xvfb in script ?

[mateuszmazurek]

It's not better, I'm proposing to test it just to be 100% sure that it's not problem with node-xvfb.

[GautierT]

Thanks a lot @mateuszmazurek. The queue work very well... I run it with 1 concurrency and i used node-xvfb and the cpu goes between 40 and 60%. And the script goes pretty fast !
(I used async.queue)
I will try with xvfb-run to test the cpu.
Thanks again.