[Question] How to setUserAgent for puppeteer. about html-get

[Kikobeats]

Hello,

const result = await getHTML('https://www.bloomberg.com/', {
  headers: {
    'user-agent': 'googlebot'
  }
})

particularly Bloomberg is validating you are a human you need to figure a way to bypass the validation 🙂

[sidster-io]

Hey @Kikobeats, That did not get the desired result I was looking for. The way I get around the human validation on Bloomberg is by adding .setUserAgent("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/601.3.9 (KHTML, like Gecko) Version/9.0.2 Safari/601.3.9"). Here is how my Puppeteer scraper looks;

    console.log("🧹 Scrape Webpage " + url)
    try {
        const browser = await puppeteer.launch({ args: ['--no-sandbox'] });
        const page = await browser.newPage();
        await page.setUserAgent("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/601.3.9 (KHTML, like Gecko) Version/9.0.2 Safari/601.3.9")
        await page.goto(url);
        const html = await page.content()
        browser.close()
        return html
    } catch (error) {
        return error
    }
}

I honestly don't know if this is the right way but for some reason it works and if I remove the setUserAgent it will fail and ask me for human validation. When you get a chance try it out on your script, also I randomize the user agent so I don't get flagged 😄