tempname1024 / crane

Repository moved to https://git.jordan.im/crane/

Crane

Crane is a literature download and organization web service. Paper and metadata download is possible for (nearly) every open-access journal which satisfies citation HTML <meta> tags, journals supported by sci-hub, and direct download links.

Goals of the project are to be minimal, support data portability (no databases or app-proprietary formats), and secure. Paper "categories" are simply directories created on the filesystem, and paper metadata is derived from doi.org or <meta> tags and written to disk as XML.

A read-only demo instance can be found here.

Installation

Crane can be compiled with make or go build, and installed system-wide by running make install with root-level permissions.

Usage

Crane can be run locally or on a server. The index ("/") endpoint lists papers but does not permits modification to the set. The admin ("/admin/") endpoint supports optional authentication and permits paper download, deletion, and moving between categories, as well as category addition, deletion, and rename.

Usage of ./crane:
  -host string
        IP address to listen on (default "127.0.0.1")
  -port uint
        Port to listen on (default 9090)
  -path string
        Absolute or relative path to papers folder (default "./papers")
  -sci-hub string
        Sci-Hub URL (default "https://sci-hub.hkvisa.net/")
  -user string
        Username for /admin/ endpoints (optional)
  -pass string
        Password for /admin/ endpoints (optional)

By default, crane listens on 127.0.0.1:9090 but this is configurable with the --host and --port parameters. Authentication is optional but can be enabled with --user and --pass parameters; the index is always publicly accessible.

Papers are written to --path, stored in directories which serve as paper categories.

crane's Issues

Defeat "are you a human" captcha pages

We apologize for the inconvenience...
...but your activity and behavior on this site made us think that you are a bot.

We satisfy cookie checks and look like a browser, but some journals (e.g. IOP) really hate it when a request originates from non-residential ISP which makes running crane on a server less pleasant.

1. We could support passing a socks proxy to our custom dialer and rotate proxies whenever a known captcha page is discovered until we reach MAX_ATTEMTS.
2. We could pass captcha tokens over to anti-captcha, but sounds overkill. Might be fun though.

Alternatively, journals could stop being so hostile.

:)