bibanon / basc-warc

Library for creating and managing WARC files. Currently in planning / pre-alpha stage.

The Bibliotheca Anonoma

The Bibliotheca Anonoma is a wiki designed to collect, document, and safeguard the products and history of internet culture; which constitutes the shared experience of humanity on a network that defines our lives.

The Wiki

This is the source code viewer for the Bibliotheca Anonoma Wiki.
To actually view and edit the Wiki follow one of the links below:

• Wiki Homepage

  • All Stories on the Wiki
  • Internet History
  • Books on the Wiki
  • Culturally Important Treasures
  • Collections on the Wiki

• About the Bibliotheca Anonoma

  • How to Edit the Wiki
  • All the Current projects

basc-warc's Issues

Add function(s) to create basic Record from existing Record

As with #2, but this function should set the WARC-Concurrent-To header (and duplicate the WARC-Date of the existing Record) in the newly-created Record.

Simplified RecordHeader field access

Allow for simplified / dict-like record access on RecordHeader objects, similar to what the IA warc library allows.

Allow simple loading of WARC files

Write a function that loads in a WARC file and presents a WarcFile object. Does not have to be streaming yet – we can work on that part of it later.

Possible uses in Website Reconstruction

One important alternate application of this library would be to export data from the WARC files, to output HTML and other metadata.

For example, the Internet Archive has the only snapshots of 4chanarchive/Chanarchive around. We could download the warc.gz files and try to programmatically reconstruct the site as best we can using the BASC-WARC library, with further HTML extraction tools (such as BeautifulSoup) used to export the threads into a database.

On a related note, there's the good ol' warc Python Library to get us started from the Internet Archive: http://warc.readthedocs.org/en/latest/

Add function(s) to create basic Record from WarcFile

As it says, add function(s) to create basic Records from WarcFile objects. Has an option to automagically create the WARC-Record-ID and the WARC-Date header fields.

Write Gzipped Records

Finish up the Gzipped records option for WarcFile.bytes(), confirm it works properly by parsing the output with a few different WARC tools.