cloudprivacylabs / lsa-neo4j Goto Github PK

The existing linking algorithm does not support merge. It tried to re-link the nodes just inserted.

We should use a custom linking algorithm that will link existing nodes to the newly inserted ones, and that will link the inserted node to existing non-linked nodes.

Small issue. When running go build received an error message.

M:\lsaneo> go build
go: github.com/cloudprivacylabs/[email protected] requires
github.com/bserdar/[email protected]: missing go.sum entry; to add it:
go mod download github.com/bserdar/jsonom

After running below command "go build" worked.
go mod download github.com/bserdar/jsonom

We need to add support for valuesets (nodesets) -- a set of concepts that are considered equivalent.

We will represent a nodeset as a central node that defines the nodeset, with all concepts in that nodeset as adjacent nodes to that central node (hub-spoke pattern. Hub is the nodeset root, concepts are connected to the hub node)

Ex:

vital_signs: nodeset
blood pressure, heart rate, etc. : concepts

  vital_signs <-->blood_pressure
                    <-->heart  rate
                    <-->....

The commands should be:

lsaneo nodeset delete <nodesetFile>    // Delete the nodeset from DB
lsaneo nodeset apply <nodesetFile>    // Change the DB to reflect nodesetFile (create/update nodes)

The nodeset file is a spreadsheet.

nodeset_id, nodeset_labels, node_id, node_labels,optional_col1, optional_col2, ...

Process columns by column header, not column order.

nodeset_id: ID of the node that defines the nodeset. All concept nodes should be linked to this node
nodeset_labels (optional): Labels of the nodeset, space-separated list
node_id: ID of a concept node
node_labels (optional): Labels of the node, space-separated list
all other optional columns are properties of the concept node. Column header is property key, cell value is property value.

nodeset_id, nodeset_labels, node_id, node_labels, prop1, prop2
vital_signs, VITAL_SIGN,  heart_rate, CONCEPT, value1, value2

Read the input via an interface:

type NodesetInput interface {
    ColumNames() []string
    Next() ([]string,error) // Next should skip header row
    Reset() error
}

A single input file can have multiple nodesets. Work on each nodeset one by one.

Algorithm sketch:

• Read rows until you see a nodeset_id you haven't processed yet
• Load all the rows for that nodeset_id
• Process all nodes of that nodeset_id (write to db)
• Seek to the beginning, and rescan to find another nodeset_id, and continue
• Finish when no rows are processed

When updating an existing nodeset, you need to load all the nodes of that nodeset.

match (root:`NODESET` {nodesetId:<theId>})-[:theId]->(m) return root,m

I think you can use MATCH OPTIONAL to test if the m node has connections other that :theId. If node m has other connections delete the edge, Otherwise, detach delete the node if the node is to be deleted.

node: (:VITAL_SIGN, { entityId: vital_signs })
node: (:CONCEPT, {entityId:heart_rate, prop1:value1, prop2:value2})

When deleting a nodeset from a db, it should delete the valueset node, and all nodes that are attached to it that are not also attached to other valueset nodes.

Currently we are saving all values as strings. This should be changes to save values using native type. For this:

1. When saving nodes, check if a node has ls.AttributeTypeValue as label. If so, it has ls.NodeValueTerm. It may or may not have ls.ValueTypeTerm. If so, use ls.GetNodeValue(node) to get the native value. Then using type-assertions, convert that value to neo4j native value.
2. Use config to assign types to property values. For instance:
   Node: (:label {prop1:1, prop2: true, prop3:str})
   Currently, we store all properties as strings. In the neo4j config, have a section for type mappings:

propertyTypeMappings:
    prop1: json:number
    prop2: ls:boolean

Based on these mappings, use ls/types facilities to interpret the value, and then store the value using that type.
* For instance, while saving prop1, lookup the type mapping: json.number. It will return a value accessor. Use the value accessor to get the native value, and then use the type mapping in #1 to store it.
* Note that a property may be a slice

Finding the accessor for a type: ls.GetValueAccessor (typeName) like ls.GetValueAccessor(xsd:date)

Some functions under lsa/pkg/types cannot deal with GetNativeValue(str,nil). We need to fix those.

For Measure, use float64 for measure.Value

When you load graph from the db, convert all native values back to str.

This should be a function:

CreateGraph(nodes []graph.Node) {...}

It should create:

• All nodes in nodes
• If a node A in nodes is connected to another node B in nodes via edge E, then the edge E between those nodes

If a node A is connected to another node C that is node in nodes, the edges between them should not be created in the db.

A --label1-->B
A --label2-->C

CreateGraph([]{A,B}) should create nodes A and B, with edge label1