microlinkhq / html-get Goto Github PK

The dependency @metascraper/helpers was updated from 4.10.1 to 4.10.2.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

@metascraper/helpers is a direct dependency of this project, and it is very likely causing it to break. If other packages depend on yours, this update is probably also breaking those in turn.

Your Greenkeeper Bot 🌴

• https://github.com/sayem314/hooman
• https://github.com/TypeCtrl/cloudflare (slow)
• https://github.com/codemanki/cloudscraper (unmaintained)
• https://github.com/k4m4/cloudflare-detect (too basic)

related: microlinkhq/open#35

Bloomberg website returns "To continue, please click the box below to let us know you're not a robot.", I used to have my own fetch and scraper and I used puppeteer with page.setUserAgent("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/601.3.9 (KHTML, like Gecko) Version/9.0.2 Safari/601.3.9") to avoid the robot error. Wondering how to set that using html-get. Appreciate your help.

The function defined getBrowserless takes zero arguments but when called, it is passed an argument browser

Can you explain/correct it?

const getContent = async url => {
  // create a browser context inside Chromium process
  const browserContext = browserlessFactory.createContext()
  const getBrowserless = () => browserContext
  const result = await getHTML(url, { getBrowserless })
  // close the browser context after it's used
  await getBrowserless((browser) => browser.destroyContext())
  return result
}

I was just testing against the demo. Sometimes it works, sometimes it throws this error:

/Users/songkeys/GitHub/Crossbell-Box/faas/node_modules/p-cancelable/index.js:48
                                        throw new Error('The `onCancel` handler was attached after the promise settled.');
           ^
Error: The `onCancel` handler was attached after the promise settled.
    at onCancel (/Users/songkeys/GitHub/Crossbell-Box/faas/node_modules/p-cancelable/index.js:48:12)
    at makeRequest (/Users/songkeys/GitHub/Crossbell-Box/faas/node_modules/got/dist/source/as-promise/index.js:38:13)
    at Request.<anonymous> (/Users/songkeys/GitHub/Crossbell-Box/faas/node_modules/got/dist/source/as-promise/index.js:143:17)
    at Object.onceWrapper (node:events:628:26)
    at Request.emit (node:events:513:28)
    at Timeout.retry (/Users/songkeys/GitHub/Crossbell-Box/faas/node_modules/got/dist/source/core/index.js:1278:30)

Environment: macOS m1, nodejs v18.

[email protected] /Users/songkeys/GitHub/Crossbell-Box/faas
└─┬ [email protected]
  ├─┬ [email protected]
  │ └── [email protected] deduped
  ├── [email protected]
  └─┬ [email protected]
    └─┬ [email protected]
      ├── [email protected] deduped
      └─┬ [email protected]
        └── [email protected] deduped

I believe this is an issue related to the got package. See sindresorhus/got#1489.

I would also highly recommend deprecating the use of got as it's somehow buggy and slow. Instead, use the undici package, which would be much more performant. In the future (less than a month later), when nodejs v18 comes out, there is going to be built-in standard fetch which is also provided by undici.

edit: After taking another deep look, I think I found the fix...

+  if (req._isPending) {
      onCancel(() => {
        debug('fetch:cancel', { url, reflect })
        req.cancel()
      })
+   }

let me know what you think.

Hello,

I am currently using that package for one of my work and facing an issue that relative URLs in HTML are getting resolved to absolute URLs.

URL: https://spiritualtactics.com/

After some trial and error found out that if the following function in index.js (Line : 140)
rewriteHtmlUrls({ $, url, headers })

is comment out then the downloaded HTML contains relative URLs.

Request your assistance in solving this issue. As I want to have the downloaded HTML unmodified i.e. if the URLs in the original HTML is absolute it should be absolute else if it is relative, then the downloaded should have relative.

Thanks

Version 3.6.2 of browserless was just published.

This version is covered by your current version range and after updating it in your project the build failed.

browserless is a direct dependency of this project, and it is very likely causing it to break. If other packages depend on yours, this update is probably also breaking those in turn.

Your Greenkeeper Bot 🌴

Can't get it to work because of this error. How to get around this error? The example in the README file is pretty vague about this.

The dependency require-one-of was updated from 1.0.0 to 1.0.1.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

require-one-of is a direct dependency of this project, and it is very likely causing it to break. If other packages depend on yours, this update is probably also breaking those in turn.

Your Greenkeeper Bot 🌴

As I suggest in the code comment https://github.com/Kikobeats/html-get/blob/master/src/index.js#L13

This will change in the next puppeteer major release

puppeteer/puppeteer#2079

The 'Chromium' process stay alive even after getHTML is done(shows up on task manager). It's ok if it does nothing and just stay alive , but it takes so much CPU resource.

Is there any way to automatically close chromium after getHTML function has done?

I need to investigate if could be possible move from got to curl

curl has better accuracy for getting HTML from securized websites.

That's the main library used by Insomnia Client:

https://github.com/getinsomnia/node-libcurl

Example

var Curl = require('insomnia-node-libcurl').Curl

var curl = new Curl()

curl.setOpt('URL', 'www.google.com')
curl.setOpt('FOLLOWLOCATION', true)

curl.on('end', function (statusCode, body, headers) {
  console.info(statusCode)
  console.info('---')
  console.info(body.length)
  console.info('---')
  console.info(this.getInfo('TOTAL_TIME'))
  console.log(body)
  this.close()
})

curl.on('error', curl.close.bind(curl))
curl.perform()

My concern is that the API is a bit hard, need to create a good wrapper similar to got interface.

Testing URLs

• https://startse.com/noticia/netflix-do-esporte-planeja-chegada-ao-brasil-ma-noticia-para-globo

Version 8.3.2 of got was just published.

This version is covered by your current version range and after updating it in your project the build failed.

got is a direct dependency of this project, and it is very likely causing it to break. If other packages depend on yours, this update is probably also breaking those in turn.

Your Greenkeeper Bot 🌴

Hi,

are you currently using mem dependency in master?
Can't find it in repository.

With usage here there is a problem, because in 1-day cache maxAge an attacker could put many url-lookups in memory.

Version 4.1.1 of browserless was just published.

This version is covered by your current version range and after updating it in your project the build failed.

browserless is a direct dependency of this project, and it is very likely causing it to break. If other packages depend on yours, this update is probably also breaking those in turn.

Your Greenkeeper Bot 🌴

If I try to get html structure by:

(async () => {
  const { url, html, stats } = await getHTML('https://www.thewashingtondailynews.com/2019/09/04/early-voting-ends-early-in-advance-of-dorian/');
  console.log(html)
})()

Returned html structure is not complete, for example it doesn't contain og:description tag.

Result

The dependency tldts was updated from 4.0.5 to 4.0.6.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

tldts is a direct dependency of this project, and it is very likely causing it to break. If other packages depend on yours, this update is probably also breaking those in turn.

Your Greenkeeper Bot 🌴

i am trying fetch html content for url : चंद्रयान-3

the url contains character in language other than english

which looks like this when copied : https://hi.wikipedia.org/wiki/%E0%A4%9A%E0%A4%82%E0%A4%A6%E0%A5%8D%E0%A4%B0%E0%A4%AF%E0%A4%BE%E0%A4%A8-3

here is my code :

import createBrowserless from 'browserless'
import getHTML from 'html-get'

export var browserlessFactory

// Kill the process when Node.js exit
process.on('exit', () => {
    console.log('Closing browser!')
    browserlessFactory && browserlessFactory.close()
})

function initializeBrowserless() {
    console.log('Creating browserless...')
    browserlessFactory = createBrowserless()
}

const getContent = async url => {
    // Spawn Chromium process once
    if (!browserlessFactory) {
        initializeBrowserless()
    }

    // create a browser context inside Chromium process
    const browserContext = browserlessFactory.createContext()
    const getBrowserless = () => browserContext
    const result = await getHTML(url, { getBrowserless, rewriteUrls: true })
    // close the browser context after it's used
    await getBrowserless((browser) => browser.destroyContext())

    if (!result) {
        throw new Error('Failed to get HTML content')
    }
    if (result.statusCode !== 200) {
        throw new Error(`Failed to get HTML content. Status code: ${result.status}`)
    }

    // browserlessFactory.close()
    return result.html
}

export { getContent }

this is throwing error for the url :

<path>/node_modules/browserless/src/index.js:62
    getBrowser().then(browser => browser.createIncognitoBrowserContext(contextOpts))
                                         ^

TypeError: browser.createIncognitoBrowserContext is not a function
    at <path>/node_modules/browserless/src/index.js:62:42
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

can somebody help me with this, if i am doing smething wrong?

Version 4.1.2 of browserless was just published.

This version is covered by your current version range and after updating it in your project the build failed.

browserless is a direct dependency of this project, and it is very likely causing it to break. If other packages depend on yours, this update is probably also breaking those in turn.

Your Greenkeeper Bot 🌴

I'm trying to figure out why I keep getting this error:

(node:22272) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 exit listeners added. Use emitter.setMaxListeners() to increase limit
(node:22272) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 SIGINT listeners added. Use emitter.setMaxListeners() to increase limit
(node:22272) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 SIGTERM listeners added. Use emitter.setMaxListeners() to increase limit
(node:22272) MaxListenersExceededWarning: Possible EventEmitter memory leak detected. 11 SIGHUP listeners added. Use emitter.setMaxListeners() to increase limit

To be honest I'm quite new in NodeJS coding but I found online that this error is caused by too many listeners being attached to a EventEmitter. I don't know how to solve this, but I'm quite sure it's caused by this module because removing it everything works fine.
This is my code:

getHTML(url)
        .then(({url, html, stats, headers, statusCode})=>{
            if(statusCode==200){
                console.log(statusCode);
            }
        })
        .catch((err)=>{
            onCheckError(err);
        });

It is executed every 5 seconds.

The dependency @metascraper/helpers was updated from 5.2.0 to 5.2.4.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

@metascraper/helpers is a direct dependency of this project, and it is very likely causing it to break. If other packages depend on yours, this update is probably also breaking those in turn.

Your Greenkeeper Bot 🌴

The dependency require-one-of was updated from 1.0.8 to 1.0.9.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

require-one-of is a direct dependency of this project, and it is very likely causing it to break. If other packages depend on yours, this update is probably also breaking those in turn.

Your Greenkeeper Bot 🌴

The dependency got was updated from 9.2.1 to 9.2.2.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

got is a direct dependency of this project, and it is very likely causing it to break. If other packages depend on yours, this update is probably also breaking those in turn.

Your Greenkeeper Bot 🌴

One my machine HTML-GET is prerendering JavaScript to HTML. Which is what I want.
But on a colleagues machine it is not prerendering the JavaScript.

On the command line how do I force the prerendering? Or do I need to make a script adjustment some where? If so then where specifically. I am only a NPM command line user not a JavaScripter, so need very specific directions.

The dependency parse-domain was updated from 2.1.4 to 2.1.5.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

parse-domain is a direct dependency of this project, and it is very likely causing it to break. If other packages depend on yours, this update is probably also breaking those in turn.

Your Greenkeeper Bot 🌴

The dependency require-one-of was updated from 1.0.7 to 1.0.8.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

require-one-of is a direct dependency of this project, and it is very likely causing it to break. If other packages depend on yours, this update is probably also breaking those in turn.

Your Greenkeeper Bot 🌴

Version 9.2.1 of got was just published.

This version is covered by your current version range and after updating it in your project the build failed.

got is a direct dependency of this project, and it is very likely causing it to break. If other packages depend on yours, this update is probably also breaking those in turn.

Your Greenkeeper Bot 🌴