docnow / storified

archive Storify stories

On December 12, 2017 Storify announced that they were going to shut down completely on May 16, 2018. Users have until then to download their stories at which point they will disappear from the web.

storified is a little utility for downloading your stories as HTML, XML and JSON--which can be handy if you have a bunch of stories. You can take these files and mount them on your own website and point your links to them instead of Storify.

storified will also rewrite the index.html to use local images, css and javascript that are downloaded. The originally downloaded index.html is stored as index-original.html.

Install

1. Install Python
2. pip install storified

Run

% storified.py <storify username>

This will create a directory named after your username, which contains a sub-directory for each story, which in turn contains the HTML, JSON and XML export files for the story. For example, here is a partial directory structure created for the digdialog Storify user:

digdialog/
├── a-woman-s-touch-manual-labor-pink-collar-workers-a
│   ├── css
│   ├── images
│   ├── js
│   ├── index.html
│   ├── index-original.html
│   ├── index.json
│   └── index.xml
├── alberto-campagnolo-digital-dialogue-november-1-201
│   ├── css
│   ├── images
│   ├── js
│   ├── index.html
│   ├── index-original.html
│   ├── index.json
│   └── index.xml

...

If you want to control where the downloaded files go use the --download-dir command line option:

storified.py --download-dir /path/to/my/stories

Docker

If you'd rather not install storified with pip you can also run it using Docker.

% docker run -v <PATH>:/storified docnow/storified <STORIFY_USERNAME>

You will need to replace <PATH> with the full path to the directory you would like the files to be downloaded to, and <STORIFY_USERNAME> to be the Storify username who created the stories.

For example if you wanted to download digdialog's stories to /home/ed/storified:

% docker run -v /home/ed/storified:/storified/downloads docnow/storified digdialog

storified's Issues

Max retries exceeded

Looks like some of my stories reference URLs which don't exist any more. Storified crashes with this error about halfway through downloading my stories.

$ ./storified.py vitorio
Traceback (most recent call last):
  File "./storified.py", line 163, in <module>
    storified(args.account_name, args.download_dir)
  File "./storified.py", line 25, in storified
    archive_story(story, archive_dir)
  File "./storified.py", line 48, in archive_story
    rewrite_html(story, story_dir)
  File "./storified.py", line 120, in rewrite_html
    path = localize(src, story_dir, 'images')
  File "./storified.py", line 148, in localize
    path = download_file(url, path)
  File "./storified.py", line 69, in download_file
    resp = requests.get(url)
  File "/Users/vitorio/Library/Python/2.7/lib/python/site-packages/requests/api.py", line 72, in get
    return request('get', url, params=params, **kwargs)
  File "/Users/vitorio/Library/Python/2.7/lib/python/site-packages/requests/api.py", line 58, in request
    return session.request(method=method, url=url, **kwargs)
  File "/Users/vitorio/Library/Python/2.7/lib/python/site-packages/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/Users/vitorio/Library/Python/2.7/lib/python/site-packages/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/Users/vitorio/Library/Python/2.7/lib/python/site-packages/requests/adapters.py", line 508, in send
    raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='a0.twimg.com', port=80): Max retries exceeded with url: /profile_images/369647011/crych_normal.png (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x10ceb27d0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known',))

Video

How are video embeds represented in the HTML archive? Can we fetch them to be present as static assets like the images?

Support for unpublished stories?

Any plans for this?

Download external resources

It should be possible to post-process the HTML export to download any referenced images, CSS and JavaScript and then rewrite the HTML to use the local references. This is important because these links will break once Storify is shut down.

Each story directory can have an images, css and js directory added. There may be some redundancy in css and js files across stories, but duplicating them can mean the story directories can be moved around as bundles without things breaking.

Is this pip-able yet?

Add a top-level downloads directory

Currently, the data is downloaded under $REPODIR/$USERID which may cause some issues in certain situations:

• There is no good way to exclude that dynamic directory from being tracked by git using .gitignore
• Pollutes the repo when more that one user IDs are called from the same local repo as it would be difficult to identify which directories belong to the code and which ones are data directories
• Difficult to generalized the bind mount in containerized environment

I propose to change it to $REPODIR/downloads/$USERID. I can create a PR, if desired.

Unicode encoding errors on write

Different from the previous two unicode issues, tested with 0.0.8 in pip and also downloading master from here:

$ ./storified.py vitorio
Traceback (most recent call last):
  File "./storified.py", line 163, in <module>
    storified(args.account_name, args.download_dir)
  File "./storified.py", line 25, in storified
    archive_story(story, archive_dir)
  File "./storified.py", line 48, in archive_story
    rewrite_html(story, story_dir)
  File "./storified.py", line 128, in rewrite_html
    open(html_file, 'w').write(soup.prettify())
UnicodeEncodeError: 'ascii' codec can't encode character u'\u2026' in position 48744: ordinal not in range(128)

Add a LICENSE file

It would be nice to have a license file included in the repo.

Unicode Encode Error

When trying to download mine, I got this error:

UnicodeEncodeError: 'ascii' codec can't encode character u'\u2019' in position 17858: ordinal not in range(128)

I'm using Python 2.7.10, so I assume that could be the problem ...

Unicode issues

Hey! Ran this but initially received an error (an apparently classic! Python error) related to Unicode conversion issues.

The trace:

  File "/usr/local/bin/storified.py", line 65, in download_file
    open(path, "w").write(resp.text)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xe1' in position 2927: ordinal not in range(128)

I fixed this locally but in a hacky-kind-of-way and ran it to get the job done, but I wanted to note in case someone with more expertise with the codebase would like to fix it properly.

I'm running macOS Sierra and this ran with Python 2.7.