octeufer / geoprocessing-tools-for-hadoop

The Hadoop GP Toolbox provides tools to exchange features between a Geodatabase and Hadoop and run Hadoop workflow jobs.

geoprocessing-tools-for-hadoop

The Geoprocessing Tools for Hadoop provides tools to help integrate ArcGIS with Hadoop. More specifically, tools are provided that:

• Enable the exchange of data between an [ArcGIS Geodatabase] (http://resources.arcgis.com/en/help/main/10.1/index.html#/What_is_a_geodatabase/003n00000001000000/) and a Hadoop system, and
• Allow ArcGIS users to run Hadoop workflow jobs.

See these tools in action as part of the samples in GIS Tools for Hadoop.

Features

• Tools to convert between Feature Classes in a Geodatabase and JSON formatted files.
• Tools that copy data files from ArcGIS to Hadoop, and copy files from Hadoop to ArcGIS.
• Tools to run an Oozie workflow in Hadoop, and to check the status of a submitted workflow.
• Wiki

Instructions

1. Download this repository as a .zip file and unzip to a suitable location or clone the repository with a git tool.
2. WebHDFS and Requests libraries in the tool folder are provided for convenience. Remove 'webhdfs' or 'requests' folders in case there is a need to use libraries installed in /site-packages folder.
3. In the ‘ArcToolbox’ pane of ArcGIS Desktop, use the ‘Add Toolbox…’ command to add the Hadoop Tools toolbox (the HadoopTools.pyt file you saved in step 1) file into ArcGIS Desktop.
4. Use the tools individually, or use them in models and scripts, such as the examples in: GIS Tools for Hadoop.

Requirements

• ArcGIS 10.1 or later.
• A Hadoop system with WebHDFS support.

Dependencies

• WebHDFS and Requests are bundled-in and reside in the tool folder.
• A Python library webhdfs-py is required for WebHDFS support. Source is located at webhdfs-py.
• The Requests python library is required for OozieUtils.py (installation doc is located at http://docs.python-requests.org/en/latest/user/install/#install).

Resources

• ArcGIS Geodata Resource Center
• ArcGIS Blog
• twitter@esri

Issues

Find a bug or want to request a new feature? Please let us know by submitting an issue.

Contributing

Esri welcomes contributions from anyone and everyone. Please see our guidelines for contributing

Licensing

Copyright 2013 Esri

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at:

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

A copy of the license is available in the repository's license.txt file.

[](Esri Tags: ArcGIS, Geoprocessing, GP, Hadoop, Spatial, Python) [](Esri Language: Python)