intel / graphbuilder Goto Github PK
View Code? Open in Web Editor NEWThe GraphBuilder library provides functions to construct large scale graphs. It is implemented on Apache Hadoop.
License: Apache License 2.0
The GraphBuilder library provides functions to construct large scale graphs. It is implemented on Apache Hadoop.
License: Apache License 2.0
Data relationships play a vital role in various data analytics, structured machine learning, and data mining applications. These relationships can be expressed as graphs. GraphBuilder is a Java library for preparing graphs from large volumes of unstructured and semi-structured data. Processing large graphs in a distributed computing environment presents multiple system-level challenges, from parallelizing the graph construction algorithms to achieving balanced system utilization. In addition, the resultant graph must be carefully partitioned to ensure that subsequent application processing is properly balanced. GraphBuilder addresses these challenges with a carefully-designed multi-stage pipeline that leverages the Apache Hadoop framework to achieve scalability. How to build the GraphBuilder? ------------------------------ The GraphBuilder is using Maven for building the package, ensure maven is installed on the system. To build package type: $> mvn package How to use the GraphBuilder? ---------------------------- The GraphBuilder library can be used in multiple ways. Library groups multiple MapReduce jobs in four major groups: a) GraphCreation b) ComputeNetworkInfo c) GraphNormalization d) GraphPartitioning. These group can be chained to form graph construction pipeline depending on an application requirements (e.g. Please refer construction pipeline for link and doc-word graph in demoapps dir). Library can be used as tool to construct link and doc-word graph for Wikipedia using following command: * Link graph construction - $> hadoop jar target/graphbuilder-0.0.1-SNAPSHOT-hadoop-job.jar com.intel.hadoop.graphbuilder.demoapps.wikipedia.linkgraph.LinkGraphEnd2End <# of partition> <hdfs input directory> <hdfs output directory> * Doc-word graph construction - $> hadoop jar target/graphbuilder-0.0.1-SNAPSHOT-hadoop-job.jar com.intel.hadoop.graphbuilder.demoapps.wikipedia.docwordgraph.TFIDFGraphEnd2End <# of partition> <hdfs input directory> <hdfs output directory> <optional stop word filter list> Sample application code ----------------------- Early stage in graph building pipeline is data dependent, hence we abstracted out the data dependent code from the library and provided interface to plug-in data-specific code. Sample code for those interface can be found in demoapps directory. It contains code for generating link-graphs and doc-word graph out of Wikipedia XML dump. Interfaces are documented, please refer a Javadoc in the doc directory. Contact ------- More information visit www.01.org/graphbuilder
Building package with mvn package
gives following error:
Tests in error:
modifySelfTest(com.intel.hadoop.graphbuilder.test.experimental.javassist.RuntimeKeyValueTest): by java.lang.LinkageError: loader (instance of sun/misc/Launcher$AppClassLoader): attempted duplicate class definition for name: "com/intel/hadoop/graphbuilder/test/experimental/javassist/Foo"
java version "1.7.0_07"
Java(TM) SE Runtime Environment (build 1.7.0_07-b10)
Java HotSpot(TM) 64-Bit Server VM (build 23.3-b01, mixed mode)
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.