bndr / node-read

Get Readable Content from any page. Based on Arc90's readability project using cheerio engine.

Node-read

Get Readable Content from any page. Based on Arc90's readability project.

Features

1. Blazingly Fast. This project is based on Cheerio engine, which is 8x times faster than JSDOM.

Why not Node-readability

Before starting this project I used Node-readability, but the dependencies of that project plus the slowness of JSDOM made it very frustrating to work with. The compiling of contextify module (dependency of JSDOM) failed 9/10 times. And if you wanted to use node-readability with node-webkit you had to manually rebuild contextify with nw-gyp, which is not the optimal solution.

So I decided to write my own version of Arc90's Readability using the fast Cheerio engine with the least number of dependencies.

The Usage of this module is similiar to node-readability, so it's easy to switch.

Install

npm install node-read

Usage

read(html [, options], callback)

Where

• html url or html code.
• options is an optional options object
• callback is the callback to run - callback(error, article, meta)

Example

var read = require('node-read');

read('http://howtonode.org/really-simple-file-uploads', function(err, article, res) {

  // Main Article.
  console.log(article.content);
  
  // Title
  console.log(article.title);

  // HTML 
  console.log(article.html);
  
  // DOM
  console.log(article.dom);
  
});

TODO

• Examples, Docs
• Get Comments with articles
• Get the Author of the article
• Better removal of unnecessary nodes
• Better scoring of content:
  • Based on siblings
  • Based on content length, common words
  • Link density, Image density, other common elements density

node-read's Issues

Many elements in article was cleaned

I parse this web page http://techdaily.cn/ios-android-pay-wear-zhiwenshibie.html.
I found the content of article was cleaned, like image tag.
Why content is not the origin html. it should't be cleaned or modified

Could it read github readme?

I tested https://github.com/bndr/node-read, it only until

so it's easy to switch

the rest is invalid? what happens?

Links inside article content removed

First off, node-read is an awesome project and has been working great! Thanks a ton.
This issue has come up a few times -- it seems like links inside article content get removed, which can be jarring when the link is wrapping a sentence mid-paragraph.
An example is this Vox article: http://www.vox.com/2015/7/27/9044485/rush-limbaugh-donald-trump
See in-browser:

and in reader-mode:

I think the issue is this comparison not using the length of the link's inner-text: https://github.com/bndr/node-read/blob/master/lib/utils.js#L100

I'll make a PR shortly.

Check the type of node, and get its Weight

/**

• Check the type of node, and get its Weight
  **/
  function initializeNode(node) {
  if (!node || node.length == 0) return 0;
  var tag = node.get(0).name;
  if (nodeTypes['mostPositive'].indexOf(tag) != -1) return 5 + getClassWeight(node);
  if (nodeTypes['positive'].indexOf(tag)) != -1) return 3 + getClassWeight(node);
  if (nodeTypes['negative'].indexOf(tag)) != -1) return -3 + getClassWeight(node);
  if (nodeTypes['mostNegative'].indexOf(tag)) != -1) return -5 + getClassWeight(node);
  return -1;
  }

main image,videos and favicon?

i'm just curious, any chances of providing an API to fetch main image, videos and favicon of the article soon?

ref:
https://github.com/ageitgey/node-unfluff

Links that are not working correctly. [post them here]

If you find any links that node-read cannot correctly parse, please post them here.

Why not len=80 & What is the use of linkDensity

  var linkDensity = getLinkDensity(node, $);
  var len = node.text().length;
  if (len < 3) return;

  if (len > 80 && linkDensity < 0.25) {
    append = true;
  } else if (len < 80 && linkDensity == 0 && node.text().replace(regexps.trimRe, "").length > 0) {
    append = true;
  }

Content not extracted correctly

Parsing this page I get unrelated content from another block, which identifies related content.

read('http://www.tvnet.lv/zinas/arvalstis/507357-krievija_pie_ukrainas_robezas_sakoncentrejusi_200_tanku', function(err, article, res){ console.log(article.content);})

Output (formatted to fit):

<div id="article" class="article">
<a href="http://www.tvnet.lv/zinas/arvalstis/507296-krievija_draud_ukraina_ievest_miera_uzturetajus" 
class="thumb330_4-3"><img src="http://itvnet.lv/article/zinas/507296_330x248.jpg" alt=""></a>
<p>Ukrainā attīstoties sliktākajam scenārijam, Maskava «atcerēsies» par Krievijas parlamenta 
augšpalātas doto atļauju ievest kaimiņvalstī armiju, paziņojis Krievijas vēstnieks ANO Vitālijs
Čurkins. Viņš piebildis, ka gadījumā, ja vardarbība Ukrainas dienvidaustrumos nerimsies, tad 
Krievija sasaukšot ANO Drošības padomes ārkārtas sēdi.</p> </div>

Many Title tag problem

In case there are more then one title tag, it sum up all of them.
For example a page like this one: http://news.nationalgeographic.com/2017/06/faceless-fish-deep-sea-voyage-australia/

Error when doing require('node-read')

I'm getting:

Uncaught TypeError: Cannot convert undefined or null to object

Seems to happen here:

function getInverseObj(obj){
    return Object.keys(obj).sort().reduce(function(inverse, name){
        inverse[obj[name]] = "&" + name + ";";
        return inverse;
    }, {});
}

Without the require my project runs without an issue...
Am I missing some dependency or anything?

Not getting article content

node-readability is able to get the main content from this website (http://www.theguardian.com/world/2014/apr/27/ukraine-kidnapped-observers-slavyansk-vyacheslav-ponomarev), but node-read returns an empty string.

Not sure if this is really an 'issue', but I got the idea from the description that node-read would be able to return content for sites that node-readability could?