Roadmap for v1.0 about puppeteer-cluster

[ermolaev1337]

Is there a way to connect the puppeteer-cluster to a remote instance of chromium? (“connect” instead of “launch”)

[generic11]

Hello - just wanted to get a feel for how active this project is. I see puppeteer cluster as being useful for several projects I'd like to work on. However, I'm hesitant to use it if development will be abandoned. Is development still happening? Thanks!

[j-manu]

1. Add a mixed concurrency model. i.e for PAGE or CONTEXT concurrency model, have the option to distribute the jobs to more than one browser instance. So a crash won't affect all jobs and this offers a good balance between reliability and resource usage.

2. Add API to return the length of queue, time when the oldest item in queue was added and Number of jobs processed in the last minute. For a continuously operating cluster i.e jobs being added continuously, this information is valuable.

[cyxou]

Cool, glad to hear that. Feel free to ping me if you need any help)

[barpaw]

I have a question. How many browsers I can spawn in parallel for processor core? Lets Say my server has processor with 4 cores. How many browsers I can spawn in one time for my tests to pass?

[thomasdondorf]

Next time, please open a separate issue if it has nothing to do with this issue.

Regarding your question: It depends on your use case. For simple DOM handling I was able to run ~10 worker on my machine (i5 quad core). Just give it a try with the option (monitor: true) and see how your machine is handling the tasks.

[cyxou]

Unfortunately, the current implementation of custom concurrency doesn't address the case when you need to provide custom puppeteer parameters to jobInstances. IMHO this would effectively solve the #36 with puppeteer args: [ '--incognito', '--proxy-server=${proxyServer}' ] and await page.authenticate(credentials).

@thomasdondorf , what do you think about this?

[thomasdondorf]

I'm currently thinking about completely reworking the concurrency implementations. Then there would be no more "WorkerInstance" and "JobInstance". Just one function that is called when a page is needed. Then the concurrency implementation would have 100% flexibility when a puppeteer instance is started and when one is reused.

Expect some code changes in the next two weeks ;)

[strarsis]

+1 for Docker container support.
https://github.com/skalfyfan/dockerized-puppeteer

[rennokki]

(Long-term runs of puppteer-cluster #25) Make sure it's reliable and crawl more than 10 million pages with it (so far the maximum I crawled was ~800k pages)

I use k6 benchmarks in my CI tests for soketi, making sure all releases are passing benchmarks in most of the cases.

Would it be a great idea to set it up for you for page rendering testing?