load hgraphdb into gremlin server about hgraphdb

[rayokota]

You'll want to set

gremlin.graph=io.hgraphdb.HBaseGraph

[rayokota]

If things are working, it's probably not a problem, but you can try adding hadoop-hdfs.jar to the CLASSPATH if you want the warning to go away.

[99ttz]

hi,

now i'm getting:

[WARN] GraphManager - Graph [graph] configured at [conf/tinkergraph-empty.properties] could not be instantiated and will not be available in Gremlin Server. GraphFactory message: GraphFactory could not instantiate this Graph implementation [class io.hgraphdb.HBaseGraph]
java.lang.RuntimeException: GraphFactory could not instantiate this Graph implementation [class io.hgraphdb.HBaseGraph]
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:82)
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:70)
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:104)
at org.apache.tinkerpop.gremlin.server.GraphManager.lambda$new$0(GraphManager.java:55)
at java.util.LinkedHashMap$LinkedEntrySet.forEach(LinkedHashMap.java:671)
at org.apache.tinkerpop.gremlin.server.GraphManager.(GraphManager.java:53)
at org.apache.tinkerpop.gremlin.server.util.ServerGremlinExecutor.(ServerGremlinExecutor.java:83)
at org.apache.tinkerpop.gremlin.server.GremlinServer.(GremlinServer.java:110)
at org.apache.tinkerpop.gremlin.server.GremlinServer.main(GremlinServer.java:344)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:78)
... 8 more
Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/HBaseIOException
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
at io.hgraphdb.HBaseGraph.(HBaseGraph.java:157)
at io.hgraphdb.HBaseGraph.(HBaseGraph.java:136)
at io.hgraphdb.HBaseGraph.open(HBaseGraph.java:124)
... 13 more
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.HBaseIOException
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 40 more

also i have modified the variables used when stating the gremlin server to:

HADOOP_CONF_DIR=/etc/hadoop/conf
CLASSPATH=$HADOOP_CONF_DIR:/opt/gremlin-console323/ext/hgraphdb/lib:/opt/gremlin-console323/ext/hgraphdb/lib/hgraphdb-0.4.14.jar:/opt/gremlin-console323/ext/hgraphdb/lib/hbase-protocol-1.2.1.jar:/opt/gremlin-console323/ext/hgraphdb/lib/hbase-annotations-1.2.1.jar:/opt/gremlin-console323/ext/hgraphdb/lib/hbase-protocol-1.2.1.jar
HADOOP_GREMLIN_LIBS=/opt/gremlin-console-323/ext/hadoop-gremlin/lib:/opt/gremlin-console323/ext/hgraphdb/lib

any pointers?

also, how do i specify in the config what tables to use?

best
T

[99ttz]

later edit:

after editing the enviroment variables some more:

HADOOP_CONF_DIR=/etc/hadoop/conf
CLASSPATH=$HADOOP_CONF_DIR:/opt/gremlin-console323/ext/hgraphdb/lib:/opt/gremlin-console323/ext/hgraphdb/lib/:/opt/gremlin-console323/ext/hbase-client/lib/:/opt/gremlin-console323/ext/hbase-common/lib/:/opt/gremlin-console323/ext/hbase-client/plugin/:/opt/gremlin-console323/ext/hbase-common/plugin/:/opt/gremlin-console323/ext/hgraphdb/plugin/
HADOOP_GREMLIN_LIBS=/opt/gremlin-console-323/ext/hadoop-gremlin/lib:/opt/gremlin-console323/ext/hgraphdb/lib

i'm getting:

[WARN] GraphManager - Graph [graph] configured at [conf/tinkergraph-empty.properties] could not be instantiated and will not be available in Gremlin Server. GraphFactory message: GraphFactory could not instantiate this Graph implementation [class io.hgraphdb.HBaseGraph]
java.lang.RuntimeException: GraphFactory could not instantiate this Graph implementation [class io.hgraphdb.HBaseGraph]
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:82)
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:70)
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:104)
at org.apache.tinkerpop.gremlin.server.GraphManager.lambda$new$0(GraphManager.java:55)
at java.util.LinkedHashMap$LinkedEntrySet.forEach(LinkedHashMap.java:671)
at org.apache.tinkerpop.gremlin.server.GraphManager.(GraphManager.java:53)
at org.apache.tinkerpop.gremlin.server.util.ServerGremlinExecutor.(ServerGremlinExecutor.java:83)
at org.apache.tinkerpop.gremlin.server.GremlinServer.(GremlinServer.java:110)
at org.apache.tinkerpop.gremlin.server.GremlinServer.main(GremlinServer.java:344)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:78)
... 8 more
Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration
at io.hgraphdb.HBaseGraphConfiguration.toHBaseConfiguration(HBaseGraphConfiguration.java:299)
at io.hgraphdb.HBaseGraphUtils.getConnection(HBaseGraphUtils.java:36)
at io.hgraphdb.HBaseGraph.(HBaseGraph.java:157)
at io.hgraphdb.HBaseGraph.(HBaseGraph.java:136)
at io.hgraphdb.HBaseGraph.open(HBaseGraph.java:124)
... 13 more
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.conf.Configuration
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 18 more

any ideas?

best
T

[99ttz]

later edit:

after some more variable changes:

HADOOP_CONF_DIR=/etc/hadoop/conf
CLASSPATH=$HADOOP_CONF_DIR:/opt/gremlin-console323/ext/hgraphdb/lib:/opt/gremlin-console323/ext/hgraphdb/plugin/:/opt/gremlin-console323/ext/hgraphdb/lib/:/opt/gremlin-console323/ext/hbase-client/plugin/:/opt/gremlin-console323/ext/hbase-common/plugin/:/opt/gremlin-console323/ext/hadoop-common/plugin/:/opt/gremlin-console323/ext/hbase-common/lib/:/opt/gremlin-console323/ext/hbase-client/lib/:/opt/gremlin-console323/ext/hadoop-common/lib/
HADOOP_GREMLIN_LIBS=/opt/gremlin-console-323/ext/hadoop-gremlin/lib:/opt/gremlin-console323/ext/hgraphdb/lib

*there are wildcard after each folder in the classpath variable as to get all available jars.

i get:

[WARN] GraphManager - Graph [graph] configured at [conf/tinkergraph-empty.properties] could not be instantiated and will not be available in Gremlin Server. GraphFactory message: GraphFactory could not instantiate this Graph implementation [class io.hgraphdb.HBaseGraph]
java.lang.RuntimeException: GraphFactory could not instantiate this Graph implementation [class io.hgraphdb.HBaseGraph]
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:82)
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:70)
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:104)
at org.apache.tinkerpop.gremlin.server.GraphManager.lambda$new$0(GraphManager.java:55)
at java.util.LinkedHashMap$LinkedEntrySet.forEach(LinkedHashMap.java:671)
at org.apache.tinkerpop.gremlin.server.GraphManager.(GraphManager.java:53)
at org.apache.tinkerpop.gremlin.server.util.ServerGremlinExecutor.(ServerGremlinExecutor.java:83)
at org.apache.tinkerpop.gremlin.server.GremlinServer.(GremlinServer.java:110)
at org.apache.tinkerpop.gremlin.server.GremlinServer.main(GremlinServer.java:344)
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:78)
... 8 more
Caused by: io.hgraphdb.HBaseGraphException: java.io.IOException: java.lang.reflect.InvocationTargetException
at io.hgraphdb.HBaseGraphUtils.getConnection(HBaseGraphUtils.java:66)
at io.hgraphdb.HBaseGraph.(HBaseGraph.java:157)
at io.hgraphdb.HBaseGraph.(HBaseGraph.java:136)
at io.hgraphdb.HBaseGraph.open(HBaseGraph.java:124)
... 13 more
Caused by: java.io.IOException: java.lang.reflect.InvocationTargetException
at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:240)
at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:218)
at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:119)
at io.hgraphdb.HBaseGraphUtils.getConnection(HBaseGraphUtils.java:62)
... 16 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:238)
... 19 more
Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/types/CopyOnWriteArrayMap
at org.apache.hadoop.hbase.client.MetaCache.(MetaCache.java:52)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.(ConnectionManager.java:714)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.(ConnectionManager.java:643)
... 24 more
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.types.CopyOnWriteArrayMap
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 27 more

what i'm doing wrong?

best
T

[rayokota]

Ok, you can try the following. Clone the repo, then run

mvn -Pfatjar package -DskipTests

This will create a fatjar with most dependencies. Then you can set

CLASSPATH=hgraphdb/target/hgraphdb-0.4.15-SNAPSHOT.jar:/path-to/hadoop-common-2.5.1.jar:/path-to/hadoop-auth-2.5.1.jar

Don't forget to set the following properties

gremlin.graph=io.hgraphdb.HBaseGraph
gremlin.hbase.namespace=[your namespace]
hbase.zookeeper.quorum=[your zookeepers]
zookeeper.znode.parent=[your znode parent, such as /hbase]

[99ttz]

thank you very much! data is being loaded as we speak into hbase.

one more thing: upon start-up of the gremlin server, the following error still show:

[WARN] DynamicClassLoader - Failed to identify the fs of dir /tmp/hbase-tnoadmin/hbase/lib, ignored
java.io.IOException: No FileSystem for scheme: hdfs
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2660)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2667)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:172)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:357)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
at org.apache.hadoop.hbase.util.DynamicClassLoader.initTempDir(DynamicClassLoader.java:120)
at org.apache.hadoop.hbase.util.DynamicClassLoader.(DynamicClassLoader.java:98)
at org.apache.hadoop.hbase.protobuf.ProtobufUtil.(ProtobufUtil.java:243)
at org.apache.hadoop.hbase.ClusterId.parseFrom(ClusterId.java:64)
at org.apache.hadoop.hbase.zookeeper.ZKClusterId.readClusterIdZNode(ZKClusterId.java:75)
at org.apache.hadoop.hbase.client.ZooKeeperRegistry.getClusterId(ZooKeeperRegistry.java:105)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.retrieveClusterId(ConnectionManager.java:905)
at org.apache.hadoop.hbase.client.ConnectionManager$HConnectionImplementation.(ConnectionManager.java:648)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:238)
at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:218)
at org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:119)
at io.hgraphdb.HBaseGraphUtils.getConnection(HBaseGraphUtils.java:62)
at io.hgraphdb.HBaseGraph.(HBaseGraph.java:157)
at io.hgraphdb.HBaseGraph.(HBaseGraph.java:136)
at io.hgraphdb.HBaseGraph.open(HBaseGraph.java:124)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:78)
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:70)
at org.apache.tinkerpop.gremlin.structure.util.GraphFactory.open(GraphFactory.java:104)
at org.apache.tinkerpop.gremlin.server.GraphManager.lambda$new$0(GraphManager.java:55)
at java.util.LinkedHashMap$LinkedEntrySet.forEach(LinkedHashMap.java:671)
at org.apache.tinkerpop.gremlin.server.GraphManager.(GraphManager.java:53)
at org.apache.tinkerpop.gremlin.server.util.ServerGremlinExecutor.(ServerGremlinExecutor.java:83)
at org.apache.tinkerpop.gremlin.server.GremlinServer.(GremlinServer.java:110)
at org.apache.tinkerpop.gremlin.server.GremlinServer.main(GremlinServer.java:344)
[INFO] GraphManager - Graph [graph] was successfully configured via [conf/tinkergraph-empty.properties].

is there something additionaly that needs to be done?

best
T

[99ttz]

many thanks again!

T