isabella232 / graphbuilder Goto Github PK

Data relationships play a vital role in various data analytics, structured machine 
learning, and data mining applications. These relationships can be expressed as 
graphs. GraphBuilder is a Java library for preparing graphs from large volumes of 
unstructured and semi-structured data. 

Processing large graphs in a distributed computing environment presents multiple 
system-level challenges, from parallelizing the graph construction algorithms to 
achieving balanced system utilization. In addition, the resultant graph must be 
carefully partitioned to ensure that subsequent application processing is properly 
balanced. GraphBuilder addresses these challenges with a carefully-designed 
multi-stage pipeline that leverages the Apache Hadoop framework to achieve 
scalability. 

How to build the GraphBuilder?
------------------------------
The GraphBuilder is using Maven for building the package, ensure maven is 
installed on the system. To build package type:
$> mvn package


How to use the GraphBuilder? 
----------------------------
The GraphBuilder library can be used in multiple ways. Library groups multiple 
MapReduce jobs in four major groups: a) GraphCreation b) ComputeNetworkInfo 
c) GraphNormalization d) GraphPartitioning. These group can be chained to form 
graph construction pipeline depending on an application requirements (e.g. Please 
refer construction pipeline for link and doc-word graph in demoapps dir).  

Library can be used as tool to construct link and doc-word graph for Wikipedia
using following command:

* Link graph construction - $> hadoop jar target/graphbuilder-0.0.1-SNAPSHOT-hadoop-job.jar com.intel.hadoop.graphbuilder.demoapps.wikipedia.linkgraph.LinkGraphEnd2End <# of partition> <hdfs input directory> <hdfs output directory>
* Doc-word graph construction - $> hadoop jar target/graphbuilder-0.0.1-SNAPSHOT-hadoop-job.jar com.intel.hadoop.graphbuilder.demoapps.wikipedia.docwordgraph.TFIDFGraphEnd2End <# of partition> <hdfs input directory> <hdfs output directory> <optional stop word filter list>

Sample application code
----------------------- 
Early stage in graph building pipeline is data dependent, hence we abstracted 
out the data dependent code from the library and provided interface to plug-in 
data-specific code. Sample code for those interface can be found in demoapps 
directory. It contains code for generating link-graphs and doc-word 
graph out of Wikipedia XML dump. Interfaces are documented, please refer
a Javadoc in the doc directory.

Contact
-------
More information visit www.01.org/graphbuilder