medialab / artoo

artoo.js - the client-side scraping companion.

artoo.js is a piece of JavaScript code meant to be run in your browser's console to provide you with some scraping utilities.

The library's full documentation is available on github pages.

Contribution

Contributions are more than welcome. Feel free to submit any pull request as long as you added unit tests if relevant and passed them all.

To install the development environment, clone your fork and use the following commands:

# Install dependencies
npm install

# Testing
npm test

# Compiling dev & prod bookmarklets
gulp bookmarklets

# Running a test server hosting the concatenated file
npm start

# Running a https server hosting the concatenated file
# Note that you'll need some ssl keys (instructions to come...)
npm run https

Authors

artoo.js is being developed by Guillaume Plique @ SciencesPo - médialab.

Logo by Daniele Guido.

R2D2 ascii logo by Joan Stark aka jgs.

artoo's Issues

add val to artoo.scrape

Rework artoo's settings

Especially store to store.engine to have autoTypes.

How to setup a local https server to inject some code

Pagination helper (autoExpand variant)

saveHtml

Navigation history

Chrome extension

If one wants to change page and keeps some persistent state, it will be necessary to create a chrome extension injecting artoo in every page.

Scrape subsyntax

//text

Spider

A client-side spider crawling through several urls thanks to artoo's context.

Automatic typing for localStorage

Dropping this in artoo.save methods

Scope issues when using the function as callback

artoo.help

Remove Math.random from prod bookmarklet

Possibility to convert array of objects into a CSV

Plus polymorphism for saveCsv method.

Document $.simulate

Include some async utilities

Remains for series parallel.

Add some verbose indications

Fix toCSVString

No escaping for the time being.

saveStore

Save helpers from storage.

Make artoo easy to use as a framework

Load settings in dom json

Limit option for artoo.scrape?

Get command line history

Change verbose color

ajaxSpider

Function taking a list of urls and doing something on it. Useful to scrape a bunch of ajax accessible things. Asynchronous or not.

Possibility to get aggregated html from multiple jQuery selection

Investigate webdb, websql...

Drop script tag after script injection

Session storage variant for store?

Investigate chrome fs

Remote database

A remote database accessible through a server (localhost for instance), to collect some data that localstorage or downloads could not handle correctly.

Unit tests

Toolbelt --> companion

artoo.execute from bookmarklet

Possibility to load artoo if not loaded before.

Content Security Policy

Adding some shorteners to scrape

artoo.scrape('iterator', {url: 'href', title: 'text'});

Scrape polymorphism without sel object attribute

autoScroll (autoExpand variant)

scrapOne

saveImage or Document

State

Keep a persistent state for artoo enabling it to keep a script working through different pages.

Can be loaded from local Storage or remote.

Bookmarklet link in doc quick start

Navigation

Navigation Queue

Node.js version?

With cheerio and request.

Make artoo.scrape recursive

Masks

Possibility to load masks from remote servers.

For instance:

artoo.loadMask('facebook', cb);

would load selectors and helpers already coded by others and helping to scrape the content of the site.

Difference between bookmarklet & extension

• Changing pages and crawling with context
• Overcoming CSP

Chrome download settings

Document how not to be crippled by chrome download safeguards.

Change page

artoo.to(url)