zicog / parallax-scrape

Download a forum page or multiple pages from forums.parallax.com and extract posts as plain text.

parallax-scrape

node.js program to Download a single or multiple forum pages from forums.parallax.com and extract posts as plain text.

• The url of the page(s) and any page range to download are commandline arguments
• The text of posts is reformatted to 80 column width.
• Quote blocks within posts are indented for clarity.
• Code blocks within posts are output as is, no reformatting.
• Links to attachments to posts are extracted.
• Thread text goes to standard output.
• All attachedments are downloaded.

Install

First install node.js. Linux, Mac and Windows node.js downloads are available here: http://nodejs.org/ You can get parallax-scrape from a zip file on github or fetch it as a git repo:

$ git clone https://github.com/ZiCog/parallax-scrape.git

A few node.js modules are required:

$ cd parallax-scrape
$ npm install request
$ npm install htmlparser2
$ npm install ent

Run

Fetch individual pages by just giving the url:

$ node parallax-scrape http://forums.parallax.com/showthread.php/149173-Forum-scraping/page2

Missing the page number off the end of the URL will get the first page.

Fetch multiple pages by adding a first page and last page parameters;

$ node parallax-scrape http://forums.parallax.com/showthread.php/149173-Forum-scraping 3 9

In this case the page parameters override any "/pagex" in the URL.

TODO

0) Better output format.

parallax-scrape's Issues

Separate out the HTML tag finding engine out into it's own module.

The HTML tag finder could be useful for other scraping tasks if it was a reusable module.

bold, colour and other tags in post text cause a new line to generated messing up the output.

Should not include post's after_content in output.

We don' want to see:

"Last edited by ForumTools...."

everywhere.

Moderator user names are mossing from posts.

User that are moderators have slightly different HTML around their user names and are hence currently not detected and output.

Should allow for automatic download of all attachments.