Horseman Article Parser

A web page article parser which returns an object containing the article's formatted text and other attributes including sentiment, keyphrases, people, places, organisations, spelling suggestions, in-article links, meta data & lighthouse audit results.

Prerequisites

Node.js & NPM

Install

npm install horseman-article-parser --save

Usage

parseArticle(options, socket) ⇒ `Object`

Param	Type	Description
options	`Object`	the options object
socket	`Object`	the optional socket

Returns: Object - article parser results object

Usage Example

var parser = require('horseman-article-parser');

var options = {
  url: "https://www.theguardian.com/politics/2018/sep/24/theresa-may-calls-for-immigration-based-on-skills-and-wealth",
  enabled: ['lighthouse', 'screenshot', 'links', 'sentiment', 'entities', 'spelling', 'keywords']
}

parser.parseArticle(options)
  .then(function (article) {

    var response = {
      title: article.title.text,
      excerpt: article.excerpt,
      metadescription: article.meta.description.text,
      url: article.url,
      sentiment: { score: article.sentiment.score, comparative: article.sentiment.comparative },
      keyphrases: article.processed.keyphrases,
      keywords: article.processed.keywords,
      people: article.people,
      orgs: article.orgs,
      places: article.places,
      text: {
        raw: article.processed.text.raw,
        formatted: article.processed.text.formatted,
        html: article.processed.text.html
      },
      spelling: article.spelling,
      meta: article.meta,
      links: article.links,
      lighthouse: article.lighthouse
    }

    console.log(response);
  })
  .catch(function (error) {
    console.log(error.message)
    console.log(error.stack);
  })

parseArticle(options, <socket>) accepts an optional socket for pipeing the response object, status messages and errors to a front end UI.

See horseman-article-parser-ui as an example.

Options

The options below are set by default

var options = {
  // puppeteer options (https://github.com/GoogleChrome/puppeteer)
  puppeteer: {
    // puppeteer launch options (https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#puppeteerlaunchoptions)
    launch: {
      headless: true,
      defaultViewport: null
    },
    // puppeteer goto options (https://github.com/GoogleChrome/puppeteer/blob/master/docs/api.md#pagegotourl-options)
    goto: {
      waitUntil: 'domcontentloaded'
    },
    // Ignore content security policy
    setBypassCSP: true 
  },
  // clean-html options (https://ghub.io/clean-html)
  cleanhtml: {
    'add-remove-tags': ['blockquote', 'span'],
    'remove-empty-tags': ['span'],
    'replace-nbsp': true
  },
  // html-to-text options (https://ghub.io/html-to-text)
  htmltotext: {
    wordwrap: 100,
    noLinkBrackets: true,
    ignoreHref: true,
    tables: true,
    uppercaseHeadings: true
  },
  // retext-keywords options (https://ghub.io/retext-keywords)
  retextkeywords: { maximum: 10 }
}

At a minimum you should pass a url

var options = {
  url: "https://www.theguardian.com/politics/2018/sep/24/theresa-may-calls-for-immigration-based-on-skills-and-wealth"
}

If you want to enable the advanced features you should pass the following

var options = {
  url: "https://www.theguardian.com/politics/2018/sep/24/theresa-may-calls-for-immigration-based-on-skills-and-wealth",
  enabled: ['lighthouse', 'screenshot', 'links', 'sentiment', 'entities', 'spelling', 'keywords']
}

You may pass rules for returning an articles title & contents. This is useful in a case where the parser is unable to return the desired title or content e.g.

rules: [
  {
    host: 'www.bbc.co.uk',
    content: () => {
      var j = window.$
      j('article section, article figure, article header').remove()
      return j('article').html()
    }
  },
  {
    host: 'www.youtube.com',
    title: () => {
      return window.ytInitialData.contents.twoColumnWatchNextResults.results.results.contents[0].videoPrimaryInfoRenderer.title.runs[0].text
    },
    content: () => {
      return window.ytInitialData.contents.twoColumnWatchNextResults.results.results.contents[1].videoSecondaryInfoRenderer.description.runs[0].text
    }
  }
]

If you want to pass cookies to puppeteer use the following

var options = {
  puppeteer: {
    cookies: [{ name: 'cookie1', value: 'val1', domain: '.domain1' },{ name: 'cookie2', value: 'val2', domain: '.domain2' }]
  }
}

To strip tags before processing use the following

var options = {
  striptags: ['.something', '#somethingelse']
}

If you need to dismiss any popups e.g. a privacy popup use the following

var options = {
  clickelements: ['#button1', '#button2']
}

there are some additional "complex" options available

var options = {

  // array of html elements to stip before analysis
  striptags: [],

  // array of resource types to block e.g. ['image' ]
  blockedResourceTypes: [],

  // array of resource source names (all resources from 
  // these sources are skipped) e.g. [ 'google', 'facebook' ]
  skippedResources: [],

  // readability options (https://ghub.io/node-readability)
  readability: {},

  // retext spell options (https://ghub.io/retext-spell)
  retextspell: {}

  // compromise nlp options
  nlp: { plugins: [ myPlugin, anotherPlugin ] }

}

Using Compromise plugins to improve results

Compromise is the natural language processor that allows horseman-article-parser to return topics e.g. people, places & organisations. You can now pass custom plugins to compromise to modify or add to the word lists like so:

/** add some names
let testPlugin = function(Doc, world) {
  world.addWords({
    'rishi': 'FirstName',
    'sunak': 'LastName',
  })
}

const options = {
  url: 'https://www.theguardian.com/commentisfree/2020/jul/08/the-guardian-view-on-rishi-sunak-right-words-right-focus-wrong-policies',
  enabled: ['lighthouse', 'screenshot', 'links', 'sentiment', 'entities', 'spelling', 'keywords'],
  nlp: {
    plugins: [testPlugin]
  }
}

This allows us to match - for example - names which are not in the base compromise word lists.

Check out the compromise plugin docs for more info.

Development

Please feel free to fork the repo or open pull requests to the development branch. I've used eslint for linting.

Module API Docs

Build the dependencies with:

npm install

Lint the project files with:

npm run lint

Test the package with:

npm run test

Update API docs with:

npm run docs

Dependencies

Puppeteer: High-level API to control Chrome or Chromium over the DevTools Protocol
compromise: natural language processing in the browser
retext: Natural language processor powered by plugins
retext-pos: Plugin to add part-of-speech (POS) tags
retext-keywords: Keyword extraction with Retext
retext-spell: Spelling checker for retext
sentiment: AFINN-based sentiment analysis for Node.js
jquery: JavaScript library for DOM operations
jsdom: A JavaScript implementation of many web standards
lodash: Lodash modular utilities.
absolutify: Relative to Absolute URL Replacer
clean-html: HTML cleaner and beautifier
dictionary-en-gb: English (United Kingdom) spelling dictionary in UTF-8
html-to-text: Advanced html to plain text converter
nlcst-to-string: Stringify NLCST
vfile-reporter-json: JSON reporter for virtual files

Dev Dependencies

eslint: An AST-based pattern checker for JavaScript.
eslint-config-standard: JavaScript Standard Style - ESLint Shareable Config
eslint-plugin-import: Import with sanity.
eslint-plugin-json: Lint JSON files
eslint-plugin-node: Additional ESLint's rules for Node.js
eslint-plugin-promise: Enforce best practices for JavaScript promises
eslint-plugin-standard: ESlint Plugin for the Standard Linter
jsdoc-to-markdown: Generates markdown API documentation from jsdoc annotated source code.

License

This project is licensed under the GNU GENERAL PUBLIC LICENSE Version 3 - see the LICENSE file for details

Notes

Due to node-readability being stale I have imported the relevent functions into this project and refactored it so it doesn't use request and therfor has no vulnrabilities.

fmacpro / horseman-article-parser Goto Github PK