knowbi / knowbi-pentaho-pdi-neo4j-output

Pentaho Data Integration output step for Neo4J

knowbi

know.bi is a team of data engineers and consultants with a very strong Apache Hop affinity as developers, implementers and sponsors since the project’s inception. know.bi has a long track record as one of Europe’s most experienced Pentaho implementors.

In addition to Apache Hop, know.bi specializes in Neo4j and Vertica.

Know.bi primary focus is to add value to our customer’s data, with technology as a means to an end. We have an "open source first" approach, since we strongly believe open source and an open mindset are prerequisites to build solutions that are as innovative as robust. Our main platforms are Apache Hop, since all data needs orchestration, and Neo4j, because all data is connected.

We are based in Belgium but operate globally.

knowbi-pentaho-pdi-neo4j-output's Issues

Neo4j Cypher - Unable to list Neo4j connections

Dear Know.bi Team,

I just tried the transformation Neo4j Cypher and it throws an exception "Unable to list Neo4j connections" . it means that I cannot configure this module to be connected with Neo4j. I declared 2 connections on my project (One from Generic JDBC Driver and the other, using the Neo4j connection already declared into Connections list) and both are working well according to the test. But the Cypher module doesn't show the list of available connectors.

Also, I tried to create a connection from the New connection button inside of the Cypher module, but it throws this exception when I tried to save it.

java.lang.NullPointerException
	at org.pentaho.metastore.stores.delegate.DelegatingMetaStore.getElements(DelegatingMetaStore.java:275)
	at org.pentaho.metastore.stores.delegate.DelegatingMetaStore.getElementByName(DelegatingMetaStore.java:314)
	at org.pentaho.metastore.persist.MetaStoreFactory.loadElement(MetaStoreFactory.java:77)
	at bi.know.kettle.neo4j.shared.NeoConnectionUtils.newConnection(NeoConnectionUtils.java:38)
	at bi.know.kettle.neo4j.steps.cypher.CypherDialog.newConnection(CypherDialog.java:562)
	at bi.know.kettle.neo4j.steps.cypher.CypherDialog$4.widgetSelected(CypherDialog.java:423)
	at org.eclipse.swt.widgets.TypedListener.handleEvent(Unknown Source)
	at org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source)
	at org.eclipse.swt.widgets.Display.sendEvent(Unknown Source)
	at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source)
	at org.eclipse.swt.widgets.Display.runDeferredEvents(Unknown Source)
	at org.eclipse.swt.widgets.Display.readAndDispatch(Unknown Source)
	at bi.know.kettle.neo4j.steps.cypher.CypherDialog.open(CypherDialog.java:447)
	at org.pentaho.di.ui.spoon.delegates.SpoonStepsDelegate.editStep(SpoonStepsDelegate.java:120)
	at org.pentaho.di.ui.spoon.Spoon.editStep(Spoon.java:8949)
	at org.pentaho.di.ui.spoon.trans.TransGraph.editStep(TransGraph.java:3291)
	at org.pentaho.di.ui.spoon.trans.TransGraph.mouseDoubleClick(TransGraph.java:785)
	at org.eclipse.swt.widgets.TypedListener.handleEvent(Unknown Source)
	at org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source)
	at org.eclipse.swt.widgets.Display.sendEvent(Unknown Source)
	at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source)
	at org.eclipse.swt.widgets.Display.runDeferredEvents(Unknown Source)
	at org.eclipse.swt.widgets.Display.readAndDispatch(Unknown Source)
	at org.pentaho.di.ui.spoon.Spoon.readAndDispatch(Spoon.java:1375)
	at org.pentaho.di.ui.spoon.Spoon.waitForDispose(Spoon.java:8104)
	at org.pentaho.di.ui.spoon.Spoon.start(Spoon.java:9466)
	at org.pentaho.di.ui.spoon.Spoon.main(Spoon.java:701)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.pentaho.commons.launcher.Launcher.main(Launcher.java:92)

Any idea how to fixed?
Thanks in advance,
Antonio

Graph Output : Support UNWIND

At first glance it's a hard thing to do since every executed Cypher statement is different from the previous one. However, with "some extra code" we could group identical Cypher statements (using parameters) and execute those in blocks as well. If you for example have 10 nodes and 9 relationships in the graph model you can have anything from 10/9 to 1/0 nodes/relationships. So we wait until we have the batch size number of rows and accumulate all the data in memory, then push the groups out in a number of UNWIND statements.

Update plugin functionality

Have a "Check for updates" or "Update plugin" functionality in the Neo4j perspective

Setting NULL as last value in a list causes error in Cypher

It's missing a space.

org.neo4j.driver.v1.exceptions.ClientException: Invalid input ')': expected whitespace or a relationship pattern (line 2, column 625 (offset: 665))
"SET node1.listingUrl = {param2} , node1.name = {param3} , node1.summary = NULL, node1.space = {param5} , node1.description = {param6} , node1.experiencesOffered = {param7} , node1.neighborhoodOverview = NULL, node1.notes = NULL, node1.transit = NULL, node1.access = NULL, node1.interaction = NULL, node1.houseRules = NULL, node1.latitude = {param14} , node1.longitude = {param15} , node1.isLocationExact = {param16} , node1.propertyType = {param17} , node1.roomType = {param18} , node1.accommodates = {param19} , node1.bathrooms = NULL, node1.bedrooms = {param21} , node1.beds = NULLMERGE (node2:City { zipcode : {param24} }) "
2018/11/23 09:21:54 - Update Airbnb model.0 - ^

Notice the NULLMERGE statement at the end of that statement.
We'd better add spaces after all the NULL values.

Is there available documentation about this step?

Allow editing of graph models in the Neo4j perspective

When a Neo4j Output step is connected but gets no input fields you get an error

java.lang.IllegalArgumentException: Argument cannot be null
at org.eclipse.swt.SWT.error(Unknown Source)
at org.eclipse.swt.SWT.error(Unknown Source)
at org.eclipse.swt.SWT.error(Unknown Source)
at org.eclipse.swt.widgets.Widget.error(Unknown Source)
at org.eclipse.swt.widgets.Combo.setItems(Unknown Source)
at bi.know.kettle.neo4j.steps.output.Neo4JOutputDialog.open(Neo4JOutputDialog.java:497)
at org.pentaho.di.ui.spoon.delegates.SpoonStepsDelegate.editStep(SpoonStepsDelegate.java:120)
at org.pentaho.di.ui.spoon.Spoon.editStep(Spoon.java:8949)
at org.pentaho.di.ui.spoon.trans.TransGraph.editStep(TransGraph.java:3291)
at org.pentaho.di.ui.spoon.trans.TransGraph.mouseDoubleClick(TransGraph.java:785)
at org.eclipse.swt.widgets.TypedListener.handleEvent(Unknown Source)
at org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source)
at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source)
at org.eclipse.swt.widgets.Display.runDeferredEvents(Unknown Source)
at org.eclipse.swt.widgets.Display.readAndDispatch(Unknown Source)
at org.pentaho.di.ui.spoon.Spoon.readAndDispatch(Spoon.java:1375)
at org.pentaho.di.ui.spoon.Spoon.waitForDispose(Spoon.java:8104)
at org.pentaho.di.ui.spoon.Spoon.start(Spoon.java:9466)
at org.pentaho.di.ui.spoon.Spoon.main(Spoon.java:701)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.pentaho.commons.launcher.Launcher.main(Launcher.java:92)

Neo4j Perspective crashes WebSpoon

WebSpoon has an issue with Tabbed interfaces apparently

WebSpoon Neo4j Perspective.pdf

3.1.0 : Lazy conversion leads to B[ strings in relationship labels

When reading from a CSV file using lazy conversion, version 3.1 leads to B[ strings in relationship labels

Cypher step: allow for row collection to support UNWIND

Consider the following example: https://neo4j.com/docs/developer-manual/current/cypher/clauses/unwind/#unwind-creating-nodes-from-a-list-parameter

We can collect the data for the step in blocks of and pass it to the statement given by the user if we see the "Collect rows for UNWIND?" option enabled.

Cypher step : add option to reconnect on ServiceUnavailableException

This could fix certain time-out issues on the server (LDAP et al.)

Validate variable usage and pickup in Neo4j Connection Dialog and Runtime

Testing and fixing issues case

Add scrollbar to Cypher Textbox in Neo4j Cypher step

Enhance Get Fields button to reformat Neo4j Property names to the standard

When using the Get Fields button the fields are brought over exactly as in the source.
Is it possible to reformat the fields to
1)Capitalize the all words after a space or _
2)Make first character lower case (if not a number)
3)Remove spaces and _ concatenate the words together.
e.g Text Area 1 becomes textArea1
My_Silly_Column becomes mySillyColumn

Import Properties in Neo4j Graph Output should apply same property style rules as Neo4j Output

When the properties get added with the Import Properties button in the Graph Model Editor - Node Tab, it would be great to use the same style and case rules as adding properties with (Get Fields) button in Neo4j Output.

String return values from cypher queries are quoted

Incorrect API usage: need to use asString() i.s.o. toString() which places double quotes around values.

Move old CCombo SWT widgets to Combo

The Custom Combo widgets are acting up on Linux.

Add a Preview button

Similar to other input steps, add a Preview button between the OK and Cancel in the Neo4j Cypher step.

Cypher component enhancement

Hi,

Firstly thanks for your plugin, it's a great one.

By using it, I have some suggestion to enhance the cypher component :

• Have a Get Fields button on the Parameters section, to fulfil it with all the input fields.
• Have a Compute return fields button, to fulfil the return section

Yes, devs are lazy ^^

The second point is not an easy one because you need to have the result format without executing the query. There is no functionnality to do that in Neo4j, but by adding a LIMIT 0 to the query and rollback the transaction, it should work.

Cheers.

Add ConfigBuilder connection options

See: https://neo4j.com/docs/api/java-driver/1.7/org/neo4j/driver/v1/Config.ConfigBuilder.html

Add support for ETL Metadata Injection

Now that the core functionality of the steps has stabilized a bit we should add support for more dynamic scenarios

For Graph Output have Relationship Names based on a Property

Take the case of employee we want the relationship to be IS_CURRENT if they are an employee today, and IS_PREVIOUS if they are not. A property can be set with this in a previous step.

The other use case is if the RELATIONSHIP name is data driven, e.g. in the input.

Wrong Pentaho Repisotry

knowbi-pentaho-pdi-neo4j-output/pom.xml

Line 93 in 678c80e

<url>http://repository.pentaho.org/artifactory/repo/</url>

Change to http://nexus.pentaho.org/content/groups/omni

Unable to send messages to server: 远程主机强迫关闭了一个现有的连接(The remote host forcibly closed an existing connection)

2018/12/10 22:08:49 - MAX_OVERDUE_DAYS nodes.0 - ERROR (version 6.0.0.0-353, build 1 from 2015-10-07 13.27.43 by buildguy) : Error executing statement: MERGE (n:MAX_OVERDUE_DAYS { 最大逾期天数: {最大逾期天数}});
2018/12/10 22:08:49 - MAX_OVERDUE_DAYS nodes.0 - ERROR (version 6.0.0.0-353, build 1 from 2015-10-07 13.27.43 by buildguy) : Unable to send messages to server: 远程主机强迫关闭了一个现有的连接。
2018/12/10 22:08:49 - 阻xudai 2.0 - 完成处理 (I=0, O=0, R=4957, W=4957, U=0, E=0
2018/12/10 22:08:49 - Add MAX_OVERDUE_DAYS Label.0 - 完成处理 (I=0, O=0, R=9908, W=9908, U=0, E=0
2018/12/10 22:08:49 - 建立base表中节点关系 - ERROR (version 6.0.0.0-353, build 1 from 2015-10-07 13.27.43 by buildguy) : 错误被检测到!
2018/12/10 22:08:49 - MAX_OVERDUE_DAYS nodes.0 - ERROR (version 6.0.0.0-353, build 1 from 2015-10-07 13.27.43 by buildguy) : Error adding nodes
2018/12/10 22:08:49 - MAX_OVERDUE_DAYS nodes.0 - Unable to send messages to server: 远程主机强迫关闭了一个现有的连接。
2018/12/10 22:08:49 - APPLY_PASS_COUNT nodes.0 - ERROR (version 6.0.0.0-353, build 1 from 2015-10-07 13.27.43 by buildguy) : Unexpected error
2018/12/10 22:08:49 - APPLY_PASS_COUNT nodes.0 - ERROR (version 6.0.0.0-353, build 1 from 2015-10-07 13.27.43 by buildguy) : java.lang.NullPointerException
2018/12/10 22:08:49 - APPLY_PASS_COUNT nodes.0 - at bi.know.kettle.neo4j.output.Neo4JOutput.processRow(Neo4JOutput.java:62)
2018/12/10 22:08:49 - APPLY_PASS_COUNT nodes.0 - at org.pentaho.di.trans.step.RunThread.run(RunThread.java:62)
2018/12/10 22:08:49 - APPLY_PASS_COUNT nodes.0 - at java.lang.Thread.run(Thread.java:745)

Browser port is not stores in the Neo4j Connection Dialog

Performance: re-use Driver

Neo4j Driver objects are closed and re-opened all the time and this takes time.
It's better to re-uses those and keep them around. This way, the driver can use connection pooling.
For small transformations in a loop this makes a lot of difference.

Add LICENSE file

Can you add LICENSE (preferably permissive one) file? Without it, I'm not allowed to make a contribution.

README file with version information.

Please provide a README file with version information / usage instructions if possible. Is this code actually functional with current Pentaho?

Neo4j Graph Output gets Stuck in Halting with Missing Parameter Field

When this error occurs the Transform won't completely stop.
2018/11/14 11:49:42 - CSV file input.0 - Header row skipped in file '/Users/davidrosenblum/Documents/kettle/hr_poc2/files/cp2position.csv'
2018/11/14 11:49:42 - Neo4j Graph Output.0 - ERROR (version 8.1.0.0-365, build 8.1.0.0-365 from 2018-04-30 09.42.24 by buildguy) : Unexpected error
2018/11/14 11:49:42 - Neo4j Graph Output.0 - ERROR (version 8.1.0.0-365, build 8.1.0.0-365 from 2018-04-30 09.42.24 by buildguy) : org.pentaho.di.core.exception.KettleStepException:
2018/11/14 11:49:42 - Neo4j Graph Output.0 - Unable to find parameter field 'PerArea
2018/11/14 11:49:42 - Neo4j Graph Output.0 -
2018/11/14 11:49:42 - Neo4j Graph Output.0 - at bi.know.kettle.neo4j.steps.graph.GraphOutput.processRow(GraphOutput.java:155)
2018/11/14 11:49:42 - Neo4j Graph Output.0 - at org.pentaho.di.trans.step.RunThread.run(RunThread.java:62)
2018/11/14 11:49:42 - Neo4j Graph Output.0 - at java.lang.Thread.run(Thread.java:748)

Create a Neo4j perspective in Spoon

• Show defined connections
• Allow CRUD of connections
• Allow user to open Browser connections to defined connections

Neo4j Output: have an option for fixed labels in nodes and relationships

Graph Output : support relationship properties

Right now only node properties are supported by the Graph Output step

The Cypher dialog is too high

It's cutting off the Return values section on smaller displays

Store a standard map in Trans which will contain all affected nodes

For steps like Neo4j Output and Graph Output we can keep track of a list of nodes we've updated. With a simple map in Trans we can do this without noticeable performance degradation.
This information can in turn be used by the kettle-neo4j-logging plugin to link from the metadata of the step to the actual data node(s) with a UPDATED_BY_STEP lineage relationship.

the newest release version can not be used in 6.0.0.0

when i download the Neo4JOutput-3.16.0.zip ,unzip and put it in plugins\steps and delete the older Neo4JOutput folder. but the plugin don't appear in the sidebar when i restart kettle

Add functionality to check Neo4j connections

We would love to have a job entry that connects to one or more Neo4j databases to see if they're up and running or not, logs issues.

Create a job entry to execute a Cypher Script

Integer Kettle data type is incorrectly mapped to String

3.0.7无法编译

Graph Output : setting NULL values failed because of missing space

2018/10/03 14:20:14 - Neo4j Graph Employee.0 - MERGE (node5:Address { city : {param28} , address : {param29} , state : {param31} , postalCode : {param32} })
2018/10/03 14:20:14 - Neo4j Graph Employee.0 - SET node5.address1Line2 = {param30} , node5.address1Line3 = NULLMERGE (node1)-[rel4:HAS]->(node5)

Add Neo4j to the Spoon Tools menu

Add options for managing Neo4j connections and opening browser sessions.

3.2.0 : Graph Output only updates properties ON MATCH, not ON CREATE

needs to move to simply SET

Code cleanup: get rid of Cloneable

use constructors instead

Error encoding to XML

when it is used in my remote server , there will be a error :Error encoding to XML
java.lang.NullPointerException
at bi.know.kettle.neo4j.output.Neo4JOutputMeta.getXML(Neo4JOutputMeta.java:75)
at org.pentaho.di.trans.step.StepMeta.getXML(StepMeta.java:221)
at org.pentaho.di.trans.step.StepMeta.getXML(StepMeta.java:197)
at org.pentaho.di.ui.spoon.Spoon.copySelected(Spoon.java:3472)
at org.pentaho.di.ui.spoon.Spoon.copySteps(Spoon.java:1423)
at org.pentaho.di.ui.spoon.Spoon.copy(Spoon.java:1451)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.pentaho.ui.xul.impl.AbstractXulDomContainer.invoke(AbstractXulDomContainer.java:313)
at org.pentaho.ui.xul.impl.AbstractXulComponent.invoke(AbstractXulComponent.java:157)
at org.pentaho.ui.xul.impl.AbstractXulComponent.invoke(AbstractXulComponent.java:141)
at org.pentaho.ui.xul.jface.tags.JfaceMenuitem.access$100(JfaceMenuitem.java:43)
at org.pentaho.ui.xul.jface.tags.JfaceMenuitem$1.run(JfaceMenuitem.java:106)
at org.eclipse.jface.action.Action.runWithEvent(Action.java:498)
at org.eclipse.jface.action.ActionContributionItem.handleWidgetSelection(ActionContributionItem.java:545)
at org.eclipse.jface.action.ActionContributionItem.access$2(ActionContributionItem.java:490)
at org.eclipse.jface.action.ActionContributionItem$5.handleEvent(ActionContributionItem.java:402)
at org.eclipse.swt.widgets.EventTable.sendEvent(Unknown Source)
at org.eclipse.swt.widgets.Widget.sendEvent(Unknown Source)
at org.eclipse.swt.widgets.Display.runDeferredEvents(Unknown Source)
at org.eclipse.swt.widgets.Display.readAndDispatch(Unknown Source)
at org.pentaho.di.ui.spoon.Spoon.readAndDispatch(Spoon.java:1339)
at org.pentaho.di.ui.spoon.Spoon.waitForDispose(Spoon.java:7939)
at org.pentaho.di.ui.spoon.Spoon.start(Spoon.java:9214)
at org.pentaho.di.ui.spoon.Spoon.main(Spoon.java:653)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.pentaho.commons.launcher.Launcher.main(Launcher.java:92)

Neo4j Output : create option to only create relationships

This way we can use UNWIND and have optimal performance

Graph Output : several issues with editing Nodes and Relationships

You risk losing data when editing nodes and relationships.

• Relationship names should be renamed in-line like nodes
• New nodes need to be added to drop-down lists in the relationship tab
• ...

Show graph representation in the Graph Model editor

Don't use UNWIND in the prompts on the Neo4j Cypher UI

The UNWIND key-word is not correctly used there.
“Use UNWIND to collect parameter values?” should be
Collect parameter value maps?
and
“Name of UNWIND values map”
should be
“Name of values map list”
Then in cypher you have to do the UNWIND in the Cypher box, as it is now.

You could use “Name of UNWIND values map” but then the "UNWIND $__list as xx should be automatically prepend the cypher statement. Where "xx" is the name of the UNWIND values map and "$__list" an internal name for the values map list.

Add support for relationship properties in Graph Output

Neo4j Output Step won't load relationship properties

knowbi-pentaho-pdi-neo4j-output/src/main/java/bi/know/kettle/neo4j/steps/output/Neo4JOutputMeta.java

Line 249 in c3211b7

List<Node> relPropNodes = XMLHandler.getNodes( toPropsNode, STRING_RELPROP );

When opening a saved ktr, the relationship properties won't be loaded into the transformation, the line referenced above should point to relPropsNode instead of toPropsNode