jdotpz / node-hubble

Node Web API for discovering content types for URLs

node-hubble

This is an node.js, Express-based REST API for discovering the content type of URLs.

Installation

$ npm install

Configuration

The server is configured by setting environment variables. These settings include:

• PORT - The Port number to use for the server. Defaults to 8888.
• REDIS_URL, REDISCLOUD_URL, or REDISTOGO_URL - The URL of the Redis server used for caching. Defaults to none.
• MEMCACHED_URL - The URL of the Memcached server used for caching. Defaults to none.
• CACHE_EXPIRE - The number of seconds to cache responses. Defaults to 3600 seconds (i.e., 1 hour).

How to use

$ node server.js

API

The node-hubble server expects a URL in the following form:

http://localhost:<PORT>/url/<url>

The server takes the URL and attempts to do two things. First, it gets the actual URL in the case that a URL shortener has been used, or other redirects. Second, it requests the HTTP headers for this resource, and determins the content-type that is being used. Both pieces of information are returned as JSON.

Assuming a node-hubble server running on http://localhost:8888, the following are valid API calls:

Example 1: simple url

http://localhost:8888/url/http://google.com

{
  "href": "http://www.google.ca/",
  "contentType": "text/html; charset=ISO-8859-1"
}

Example 2: shortened url

http://localhost:8888/url/http://bit.ly/900913

{
  "href": "http://www.google.ca/",
  "contentType": "text/html; charset=ISO-8859-1"
}

Example 3: image resource

http://localhost:8888/url/http://i.imgur.com/syPS3rG.jpg

{
  "href": "http://i.imgur.com/syPS3rG.jpg",
  "contentType": "image/jpeg"
}

Example 4: video resource

http://localhost:8888/url/http://archive.org/download/PET1018_R-2_LA/PET1018_R-2_LA.mp4

{
  "href": "http://ia700805.us.archive.org/2/items/PET1018_R-2_LA/PET1018_R-2_LA.mp4",
  "contentType": "video/mp4"
}

JSON vs. JSONP

If the caller provides a callback=<callbackFn> query string parameter, the result will be JSONP instead of pure JSON:

http://localhost:8888/url/http://google.com?callback=foo

foo && foo({
  "href": "http://www.google.ca/",
  "contentType": "text/html; charset=ISO-8859-1"
});

Note: if the URL being passed to the API also includes ?callback=..., remember to encode the URL such that ? becomes %3F and is not interpreted as part of the API call. Consider:

// URL with ?callback param used as part of API call (JSONP)
http://localhost:8888/url/http://foo.com?callback=bar

// URL containing ?callback param (not part of API call)
http://localhost:8888/url/http%3A%2F%2Ffoo.com%3Fcallback%3Dfn

// URL containing ?callback param with API using JSONP callback param bar
http://localhost:8888/url/http%3A%2F%2Ffoo.com%3Fcallback%3Dfn?callback=bar

Errors

A number of situations can cause errors. In all such cases, the API will return a 500 result code, and JSON of the following form:

{
  "error": "Error message..."
}

Current errors include:

• Expected url param, found none. -- no url was specified to the API call.
• Unable to determine content type. -- the url was unusable, or the resource's content type could not be determined for some reason.

If no error message is given, an unknown error occurred.

Running the Tests

You can run the tests locally by doing the following from the project's root directory:

$ npm test

The tests depend on a network connection being available (some real URLs are used).

In order to test the server's cache support, you have to run the tests with some extra environment variables. First, install and start redis-server and/or memcached locally, then run the tests like so:

$ EXPECT_CACHED=1 REDIS_URL=127.0.0.1 npm test
...
$ EXPECT_CACHED=1 MEMCACHED_URL=127.0.0.1 npm test

License

Copyright 2013 David Humphrey [email protected]

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.