codeforamerica / naics-api

Basic API to return NAICS codes and information

NAICS-API

NAICS (North American Industry Classification System) is maintained by the United States Bureau of Labor Statistics to classify business types. It is used for aggregating, presenting, and analyzing data and trends in the US economy.

The classification system is currently hosted by the Census Bureau and provided in various Excel and PDF documents, with some rudimentary HTML output and a not-so-great search tool. Our goal is to improve on the Census Bureau's offerings by providing an API to make information machine-readable, with better search functionality, to assist with developing applications that depend on understanding or collecting information about businesses.

The Product

NAICS API is currently a Node.js server that returns NAICS data in a JSON format. Information stored on the server has been scraped or collected from files on the Census.gov web site. Most of the information for 2007 and 2012 has now been scraped thanks to the addition of a python scraper by Mike Migurski (see ./data/scrape-examples-xrefs).

API documentation

Latest API documentation is hosted at Apiary.io.

API example requests

Example request

http://api.naics.us/v0/q?year=2012&code=519120

To get NAICS codes above a given code

http://api.naics.us/v0/q?year=2012&code=519120&above=1

To get NAICS codes below a given code

http://api.naics.us/v0/q?year=2012&code=51&below=1

To get all NAICS codes for a given years codes (only 2007 and 2012 are available right now)

http://api.naics.us/v0/q?year=2012

To get all NAICS codes for given search terms (searches only title and index right now)

http://api.naics.us/v0/s?year=2012&terms=libraries

Warning! The URL (server and/or structure) is likely to change in the very near future. Do not use for production (yet).

Usage

• A simple example demo search interface for NAICS codes site and repository

• Work in progress real-world application site and repository

Additional information

• An early NAICS scraper at daguar/naics-scraper)
• Further discussion at dobtco/NAICS
• Ongoing progress report at codeforamerica/hack-requests

Development setup (on Mac OS X 10.8)

First-time setup

1. Download and install Node.js.

2. Clone this repository to a folder on your computer. The rest of this document will refer to this folder as $PROJECT_ROOT.

3. Install project dependencies.

   cd $PROJECT_ROOT npm install

Every time you sync $PROJECT_ROOT with the remote GitHub repo

1. Update the project dependencies.

   cd $PROJECT_ROOT npm install

To start the REST API server

1. Start the REST API server.

   cd $PROJECT_ROOT npm start

Contributing

Help Needed

There are other data that can be included in the API. Not all of these are within the scope of the scraper however.

• Illustrative examples from 2007 NAICS
• Information from NAICS prior to 2007 (2002, 1997 - low priority)
• Data for converting between different NAICS codes and other systems, like SIC or NIGP

On the API side:

• The API should perform searches on all the available data and return relevant results from the requester (e.g. a business type lookup application)
• Close existing issues.

Submitting an Issue

We use the GitHub issue tracker to track bugs and features. Before submitting a bug report or feature request, check to make sure it hasn't already been submitted. When submitting a bug report, please include a Gist that includes a stack trace and any details that may be necessary to reproduce the bug, including your gem version, Ruby version, and operating system. Ideally, a bug report should include a pull request with failing specs.

Submitting a Pull Request

1. Fork the repository.
2. Create a topic branch.
3. Add specs for your unimplemented feature or bug fix.
4. Implement your feature or bug fix.
5. Add, commit, and push your changes.
6. Submit a pull request.

naics-api's Issues

apostrophes in NAICS data showing up as ’

See the title for http://naics-api.herokuapp.com/v0/q?year=2012&code=315240

Provide a way for a full year query to be collapsed only to codes that have descriptions

Erasing out "blank" entries and ones that say "See industry description for..."

Add CORS Header support for 3rd party requesters

Add Jasmine tests to help document features

Jasmine is a nice little Javascript behavior-testing framework (similar to RSpec in Ruby) that lets you describe your code in very human-readable terms.

http://pivotal.github.io/jasmine/

Given a lot of the structure of issues I've seen so far, I think it'd be a great enhancement to add tests that cover these features using Jasmine.

NAICS 2-digit codes that are ranges will break above & below retrieval

Examples:

http://naics-api.herokuapp.com/v0/q?year=2012&code=446199&above=1 (44-45 is the 2-digit, top level code, which is returned as null)

http://naics-api.herokuapp.com/v0/q?year=2012&code=31-33&above=1 (does not work)

http://naics-api.herokuapp.com/v0/q?year=2012&code=31&above=1 (works, but also returns the 31-33 entry)

Search results should include entries that contain search terms that are separated by other words

(10/5 EDIT: Reworded for clarity, with some more examples.)

Currently, a search like "space research" returns only results where the two words are not separated by other words. In our case, this functionality is good for returning more relevant results (e.g. NAICS code 927110 “Space Research and Technology”) that the official Census search result does not have. (This can be checked on this search form).

In comparison, the official Census search result also returns 541712, “Research and Development in the Physical, Engineering, and Life Sciences (except Biotechnology)”, because the words "space research" appear, separated by other words, in the list of index entries (aka, the list of alternate titles). We should do the same.

Another situation that might be more prevalent is if someone searches for "pizza shop." The current search returns no results. The official Census search will properly return the result of 722513 (pizza delivery shops)

cc @rclosner

Allow for queries or searches to be performed on a digit limit range.

e.g. only 3-digit codes, or min of 4 digits / max of 5.

Service Unavailable

I previously used this API by hitting the address http://naics.codeforamerica.org/. Trying to use it today, I see the address changed at some point to http://api.naics.us/. In any case, neither of those addresses are accessible at this moment. Is this a temporary issue, or is the service no longer available? If the latter, is there a suggested replacement?

Thanks for your help!

Getting Census website data on NAICS

@louh -- Could you describe this a bit more in this issue? (Perhaps with links and what you want to achieve?)

Happy to help out if it's quick.

Consider being helpful by defaulting to last NAICS code set when a user tries to retrieve codes for non-NAICS years (e.g. 2008)

From @ycombinator: Trying to get 2008 codes will return 2007, or trying to get 2013 codes will return 2012, with a helpful (?) warning that NAICS codes don't actually exist for the requested year.

My comment: feature usefulness might depend on actual behavior. The hypothesis is that most people who use NAICS just know what years are available, and will not bother looking for a 2013 code.

Incorporate index entries

Remove search results that contain the word "except"

e.g. the top results for "Internet" include titles that have the words "(except Internet)"

Reverse priority of NAICS digits in search results

6-digit codes should have more priority (because they are more specific) than 2-digit codes

2007 Illustrative Examples are not included

They are in the PDF definition file, but not on the census.gov online search tool, so they haven't been scraped. (Maybe they can never be scraped without accessing the PDF?)

Update the API for current year

Currently the API only supports year - 2007 to 2012. Can we make this api for recent years?

search terms should take into account plurals

e.g. "bar" vs "bars", "restaurant" vs "restaurants"

Implement CORS support

I left some thoughts in Campfire, but this post shows how to implement CORS support on a NodeJS server:
http://senecacd.wordpress.com/2013/02/15/enabling-cors-on-a-node-js-server-same-origin-policy-issue/

CORS is all about serving with the appropriate headers; nothing is required on the client side beyond making a normal AJAX request across domains to test.

Add descriptions from two-digit ranged codes (eg. 44-45)

Search terms that are separate words should match text separately

Provide a way to convert between codes of different years.

What should the request look like? what does it need to return to be useful? How to store the concordance data? (see: http://www.census.gov/eos/www/naics/concordances/concordances.html)

Improve search functionality (goal is to improve how it returns results for lv-dof)

Some things @rclosner and I discussed:

• fuzzy/partial matches
• search terms with other words between them
• ranked relevance (terms matching title or illustrative examples should rank higher than in description)

cc @migurski

Store data in a database format (not flat-file JSONs)

pinging @daguar @rclosner @ycombinator for recommendations.

Search function should accept NAICS codes as search terms

It duplicates the functionality as querying / retrieving a single number but the benefit is that the response comes back in a similar format to the search functionality. Predictability is nice.

Searches on index entries or some way of indexing synonyms