sc0vu / jspachong

Js crawler library.

Javascript Pachong

Pachong which means a generic term for vertebrates in chinese, you can find here chinese dictionary, and it's something like crawler.

This is a crawler library written in javascript, so you can use this in server side or browser.

Usage

npm install jspachong

Benchmark

It will crawl 10 pages parallelly and sequentially.

npm run benchmark

Server Side

var Pachong = require('jspachong')
var crawler = new Pachong(requestObject, options)

crawler.queue(requestObject)
       .queue(requestObject)
       .queue(requestObject)
       .run()
       .then((res) => {})
       .catch((err) => {})

• requestObject

  request library options

  simple: {
    method: 'GET',
    uri: 'https://www.google.com'
    callback: function (err, res) {
      if (err) return
      // Do something here...
    }
  }
  

  see request document for more information

• options

  parallel bool
  Run crawlers parallel.
  
  max integer
  Max crawlers run each time.
  

Browser

To do.

License

MIT

jspachong's Issues

Mock the test

Right now we will request the url when run the test.

I think we can mock the request when test.

See http://sinonjs.org/

An in-range update of tape is breaking the build 🚨

The devDependency tape was updated from 4.9.1 to 4.9.2.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

tape is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Status Details

• ❌ continuous-integration/travis-ci/push: The Travis CI build failed (Details).

---

Commits

The new version differs by 11 commits.

• a1e8f7e v4.9.2
• 4b9c951 [Dev Deps] update eslint, eclint
• 9ced991 [Fix] notEqual and notDeepEqual show "expected" value on failure
• 75c467e [Docs] Updated readme to make test, test.only and test.skip consistent.
• 1225d01 Merge pull request #450 from axelpale/patch-1
• f53e3f1 Clarify doesNotThrow parameters
• 96de340 Adding tap-junit
• b1df632 [readme] Change broken image to use web archive
• 5f1c5a2 [Docs] cleanup from #439
• 6c633d0 Merge pull request #439 from abelmokadem/master
• 4337f58 Convert list of tap reporters to links

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.

---

Your Greenkeeper Bot 🌴

An in-range update of uuid is breaking the build 🚨

The devDependency uuid was updated from 3.3.2 to 3.3.3.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

uuid is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Status Details

• ❌ continuous-integration/travis-ci/push: The Travis CI build failed (Details).

---

Commits

The new version differs by 3 commits.

• ab33c8a V3.3.3 prep (#320)
• 19c22dc June '19 update (#306)
• 36a5f18 Add dev files to .npmignore (#291)

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.

---

Your Greenkeeper Bot 🌴

Allow using with stream

Right now we cannot use with stream.

So we load reponse body to memory each request.

Maybe we can allow using with stream in callback.

See

request/request-promise#90

request/request#1990

https://gist.github.com/forivall/6b3f3718b5684e9ea8daa5a4dcd855a6

Version 10 of node.js has been released

Version 10 of Node.js (code name Dubnium) has been released! 🎊

To see what happens to your code in Node.js 10, Greenkeeper has created a branch with the following changes:

• Added the new Node.js version to your .travis.yml

If you’re interested in upgrading this repo to Node.js 10, you can open a PR with these changes. Please note that this issue is just intended as a friendly reminder and the PR as a possible starting point for getting your code running on Node.js 10.

More information on this issue

Greenkeeper has checked the engines key in any package.json file, the .nvmrc file, and the .travis.yml file, if present.

• engines was only updated if it defined a single version, not a range.
• .nvmrc was updated to Node.js 10
• .travis.yml was only changed if there was a root-level node_js that didn’t already include Node.js 10, such as node or lts/*. In this case, the new version was appended to the list. We didn’t touch job or matrix configurations because these tend to be quite specific and complex, and it’s difficult to infer what the intentions were.

For many simpler .travis.yml configurations, this PR should suffice as-is, but depending on what you’re doing it may require additional work or may not be applicable at all. We’re also aware that you may have good reasons to not update to Node.js 10, which is why this was sent as an issue and not a pull request. Feel free to delete it without comment, I’m a humble robot and won’t feel rejected 🤖

---

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.

---

Your Greenkeeper Bot 🌴

Examples

• Daemon

Eslint to lint the code

Eslint to lint the code

An in-range update of eslint-plugin-jsx-a11y is breaking the build 🚨

The devDependency eslint-plugin-jsx-a11y was updated from 6.1.1 to 6.1.2.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

eslint-plugin-jsx-a11y is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Status Details

• ❌ continuous-integration/travis-ci/push: The Travis CI build failed (Details).

---

Release Notes for Minor fixes and documentation updates

6.1.2 / 2018-10-05
[fix] Add link-type styling recommendation to anchor-is-valid #486
[fix] label-has-for: textareas are inputs too #470

Commits

The new version differs by 12 commits.

• c538d35 6.1.2
• ea964b1 Update changelog
• 43d1d94 Add link-type styling recommendation to anchor-is-valid
• 71819a0 [Docs] accessible-emoji: plural of emoji is emoji
• 4a0608b [Docs] label-has-associated-control: update label id
• 9957219 [Tests] temporarily allow node 4 + eslint 4 to fail
• 969ce6d [Dev Deps] add missing dev dep
• 1be1744 [Tests] use nvm install-latest-npm
• 9103c7a [Tests] add additional test case for #474
• bbad58a Merge pull request #471 from ljharb/textarea
• 47633fb [Fix] label-has-for: textareas are inputs too
• c1bf3c1 Update aria-props.md to link to the correct doc

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.

---

Your Greenkeeper Bot 🌴

An in-range update of eslint-plugin-import is breaking the build 🚨

The devDependency eslint-plugin-import was updated from 2.16.0 to 2.17.0.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

eslint-plugin-import is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Status Details

• ❌ continuous-integration/travis-ci/push: The Travis CI build failed (Details).

---

Commits

The new version differs by 61 commits.

• 0499050 bump to v2.17.0
• f479635 [webpack] v0.11.1
• 8a4226d Merge pull request #1320 from bradzacher/export-ts-namespaces
• 988e12b fix(export): Support typescript namespaces
• 70c3679 [docs] make rule names consistent
• 6ab25ea [Tests] skip a TS test in eslint < 4
• 405900e [Tests] fix tests from #1319
• 2098797 [fix] export: false positives for typescript type + value export
• 70a59fe [fix] Fix overwriting of dynamic import() CallExpression
• e4850df [ExportMap] fix condition for checking if block comment
• 918567d [fix] namespace: add check for null ExportMap
• 2d21c4c Merge pull request #1297 from echenley/ech/fix-isBuiltIn-local-aliases
• 0ff1c83 [dev deps] lock typescript to ~, since it doesn’t follow semver
• 40bf40a [*] [deps] update resolve
• 28dd614 Merge pull request #1304 from bradennapier/feature/typescript-export-type

There are 61 commits in total.

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.

---

Your Greenkeeper Bot 🌴

An in-range update of eslint-plugin-react is breaking the build 🚨

The devDependency eslint-plugin-react was updated from 7.11.1 to 7.12.0.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

eslint-plugin-react is a devDependency of this project. It might not break your production code or affect downstream projects, but probably breaks your build or test tools, which may prevent deploying or publishing.

Status Details

• ❌ continuous-integration/travis-ci/push: The Travis CI build failed (Details).

---

Release Notes for v7.12.0

Added

• no-typos: Support createClass (#1828, @alexzherdev)
• Support detecting React.forwardRef/React.memo (#2089, @jomasti)
• jsx-indent: add checkAttributes option for JSX attribute indentation (#2086, @jomasti)
• Change allowed propWrapperFunctions setting values (#2065, @jomasti)
• add jsx-fragments rule to enforce fragment syntax (#1994, @alexzherdev)
• Support "detect" option for React version setting (#1978, @alexzherdev)
• Support shorthand fragment syntax in many rules (#1956, @alexzherdev)
• jsx-no-literals: print node value in warning message (#2008, @jlgonzalezdev)

Fixed

• jsx-max-depth: Fix depth of JSX siblings in a JSXEpressionContainer (#1824, @alexzherdev)
• no-array-index-key: fix in React.Children methods (#2085, @himynameisdave)
• no-unused-state: handle functional setState (#2084, @jomasti)
• version errors should log to stderr, not stdout (#2082, @ljharb)
• no-deprecated: Disable legacy lifecycle methods linting for now (#2069, @sergei-startsev)
• ensure that react and flow versions can be numbers (#2056, @ljharb)
• forbid-foreign-prop-types: ensure allowInPropTypes option applies to class fields (#2040, @Sheile)
• jsx-wrap-multilines: catch single missing newlines (#1984, @MrHen)
• jsx-first-prop-new-line: Fix for parsers (like TypeScript) (#2026, @HauptmannEck)
• [jsx-sort-comp][]: Fix fixer in case of more than 10 props (#2012, @tihonove)
• no-unused-state Don't depend on state parameter name ([#1829][], @alexzherdev)
• no-this-in-sfc fix for class properties (#1995, @sergei-startsev)
• no-this-in-sfc fix rule behavior for arrow functions inside a class field (#1989, @sergei-startsev)
• destructuring-assignment: handle nested props usage (#1983, @alexzherdev)
• sort-prop-types: fix string property order (#1977, @metreniuk)
• jsx-no-target-blank: don’t crash when there’s no value (#1949, @ljharb)
• prop-types, no-unused-prop-types: better handle object spread (#1939, @alexzherdev)

Changed

• jsx-fragments: improve message text (#2032, @alexzherdev)
• no-unsafe: handle all unsafe life-cycle methods (#2075, @sergei-startsev)
• require-default-props: Change error message naming from singular defaultProp to plural defaultProps (#2064, @jseminck)
• [Refactor] Extract used propTypes detection (#1946, @alexzherdev)
• [Refactor] Extract defaultProps detection (#1942, @alexzherdev)
• [Refactor] Extract required propTypes detection (#2001, @alexzherdev)
• [Docs] no-did-mount-set-state, no-did-update-set-state, no-will-update-set-state: fix docs URLs (#2090, @JBallin)
• [Docs] Remove statement on GC in jsx-no-bind (#2067, @rickhanlonii)
• [Docs] jsx-sort-props: Fix small mistake (#2044, @dimitarnestorov)
• [Docs] no-unescaped-entities: add more escape examples (#2015, @stevemao)
• [Docs] display-name: mention default ignoreTranspilerName value (#2002, @OliverJAsh)
• [Docs] jsx-no-target-blank: Add full example (#1988, @atomcorp)
• [Docs] Update jsx-no-target-blank.md (#1953, @brunocoelho)
• [Changelog] fix "Ignore class properties" contributor (#1941, @alexzherdev)
• [Tests] Remove redundant require('babel-eslint') from tests (#2004, @sergei-startsev)
• [Tests] prop-types: Add tests for prop-types destructuring (#2029, @sstern6)
• [Tests] display-name: add false positive component detection for destructured createElement (#1098, @arian)

Commits

The new version differs by 127 commits.

• a83d65c Update CHANGELOG and bump version
• 0c6160e [Dev Deps] update coveralls, babel-eslint, typescript
• cd6c8e6 [Deps] update resolve
• d2b5b73 Merge pull request #1828 from alexzherdev/1721-no-typos-create-class
• e747459 [Tests] remove node 5/7/9, add node 11
• b33ae84 Merge pull request #1098 from arian/createElement-display-test
• 84be80a Merge pull request #1824 from alexzherdev/1762-jsx-max-depth-false-positive
• a442067 Merge pull request #2029 from sstern6/issue1422
• 7d07c37 Merge pull request #2032 from alexzherdev/jsx-fragments-message
• 8c6a8e2 Merge pull request #2089 from jomasti/feature/support-react-forwardref-memo
• 14451d4 Merge pull request #2085 from himynameisdave/issues/2083
• 8be52c7 📝 Addresses CR comments
• f7e3121 Merge branch 'master' of https://github.com/yannickcr/eslint-plugin-react into issues/2083
• 77e3fd0 Merge pull request #2090 from JBallin/no-method-set-state-docs-url
• 7da9e0d Fix noMethodSetState docsUrl's

There are 127 commits in total.

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.

---

Your Greenkeeper Bot 🌴

An in-range update of request-promise is breaking the build 🚨

The dependency request-promise was updated from 4.2.2 to 4.2.3.

🚨 View failing branch.

This version is covered by your current version range and after updating it in your project the build failed.

request-promise is a direct dependency of this project, and it is very likely causing it to break. If other packages depend on yours, this update is probably also breaking those in turn.

Status Details

• ❌ continuous-integration/travis-ci/push: The Travis CI build failed (Details).

---

Commits

The new version differs by 17 commits.

• d831905 Version 4.2.3 (now really)
• 37e8773 fix: updated indirect lodash dependency to fix security vulnerability
• 229225e Version 4.2.3
• 4f27097 chore: fix ci build for node v8+
• c535eb6 Merge pull request #299 from aomdoa/fixToughCookieDep
• 488947b Merge branch 'master' into fixToughCookieDep
• 131abd7 fix: breaking change in tough-cookie v3
• b454ddc chore: added node 8 and 10 to ci build
• 6d11ddc fix: typo
• 3a136ea Merge pull request #303 from lexjacobs/patch-1
• 5e32191 docs: mention of wrapped request errors
• 2ac4f3e Update rp.js
• 1457338 Workaround on the cookie processing test.
• 062d5f8 Limit the tough-cookie to 2.5 as 3.x removes node 0.10 support
• 18c838a docs: cheat sheet for POST like HTML forms do

There are 17 commits in total.

See the full diff

FAQ and help

There is a collection of frequently asked questions. If those don’t help, you can always ask the humans behind Greenkeeper.

---

Your Greenkeeper Bot 🌴