odpi / egeria Goto Github PK
View Code? Open in Web Editor NEWEgeria core
Home Page: https://egeria-project.org
License: Apache License 2.0
Egeria core
Home Page: https://egeria-project.org
License: Apache License 2.0
remove spring-boot-starter-web dependency where is not used
replace spring-boot-starter-web dependency with spring-web where is used only for rest client
add sl4j capability
The open metadata archive has placeholders for the type definitions for annotations and request for action. These are part of Area 6 and are documented here.
https://github.com/odpi/egeria/blob/master/open-metadata-publication/website/open-metadata-types/Area-6-models.md
The following WARNing is produced many times when launching a service instance with a configured event bus.
If this is normal, but harmless, at best it should be downgraded to an INFO rather than WARN.
It may also indicate a bug?
2018-09-13 09:04:39.965 WARN 70428 --- [nio-8080-exec-6] o.a.k.clients.producer.ProducerConfig : The configuration kafka.omrs.topic.id = cocoCohort was supplied but isn't a known config.
If a cohort is operating properly then when the enterprise repository requests an existing entity then it should be returned. There is a possiblity that due to an outage or a logic error, that only entity proxies can be found. There is a todo in method throwCapturedEntityProxyOnlyException to audit log that only entity proxies are found since this may be an indication of a logic or operational error.
See throwCapturedEntityProxyOnlyException() method in
EnterpriseOMRSMetadataCollection.java
The open metadata archive has placeholders for the type definitions for logic specifications and models. These are part of Area 5 but haven ot been fully spec'ed yet.
I have some postman scripts with variables (URL, user, server, cohort) which can be used to configure the server chassis from scratch for use with an in-memory repository. These would form a useful example for anyone trying to use egeria -- as I found out when I included kafka configuration body in the wrong rest call.
I propose to add these with a README
When looking at adding unit tests for the Governance Engine OMAS Client
Proposing for inclusion
Update:
While testing the pom I encountered many errors:
[WARNING] Some problems were encountered while building the effective model for org.odpi.egeria:developer-resources:jar:0.1-SNAPSHOT
[WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-javadoc-plugin is missing. @ org.odpi.egeria:egeria:0.1-SNAPSHOT, /Users/jonesn/IdeaProjects/egeria/pom.xml, line 193, column 21
This was caused by the maven-javadoc-plugin not having a version specified. I've fixed this
Finally my previous patch to add the maven enforcer plugin used a literal version. I've updated to use a property instead.
In the top level POM we have a set of properties such as
<log4j.version>2.4.6</log4j.version>
Then when we have a flag in either this pom, or any child we can use:
${log4j.version}
This would seem to work, but I wondered whether it was simpler to use the standard maven mechanism for expressing the use of consistent versions, where we would use
flags in the top level pom to declare our default versions, and then use as today where each plugin needs to be used
We would do the same for to ensure consistent use of plugin versions across the project (in exceptional cases overriding can be done at any level).
Atlas did use the explicit property mechanism, but I would think going with standard maven is simpler,
Happy to do a patch if the community feels this is a better approach
I've added a few additional git related hints and tips in the community guide
The OMRSRepositoryEventMapperConnector class implements the OpenMetadataTopicListener interface. This is not needed. A repository event mapper may use the open metadata topic connector to receive event from the network but it is not required. This fix changes OMRSRepositoryEventMapperConnector, adds the OpenMetadataTopicListener to the interface of the local graph connector interface and removed the unimplemented processEvents() method from the IGCEventMapper as it is no longer needed.
Using Coco Pharmaceuticals persona to drive the scenario, provide Egeria capability with supporting resources to enable Callie Quartile (Data Scientist) to access all of the known information about a data source she is attempting to use.
Using Coco Pharmaceuticals persona to drive the scenario, provide Egeria capability with supporting resources to support the governance processes required to process a request to bring an external data set into an organization's data stores.
This scenario will show the creation of the onboarding request and how the approval reviews are driven along with the provisioning request and notification back to the requester.
Currently asset catalog omas gives the following if the metadata collection is not initialized
ie from:
{{baseURL}}/open-metadata/access-services/asset-catalog/users/rux/get-typedefs
{
"timestamp": "2018-09-11T09:21:35.077+0000",
"status": 500,
"error": "Internal Server Error",
"message": "No message available",
"path": "/open-metadata/access-services/asset-catalog/users/rux/get-typedefs"
}
Instead a useful exception should be raised - for example subject area returns
{{baseURL}}/open-metadata/access-services/subject-area/users/rux/glossaries/123
{
"class": "MetadataServerUncontactableExceptionResponse",
"relatedHTTPCode": 200,
"responseCategory": "MetadataServerUncontactableException",
"exceptionClassName": "org.odpi.openmetadata.accessservices.subjectarea.utilities.OMRSAPIHelper",
"exceptionErrorMessage": "OMAS-SUBJECTAREA-503-003 The access service has not been initialized and can not support REST API call ",
"exceptionSystemAction": "The org.odpi.openmetadata.accessservices.subjectarea.server has received a call to one of its open metadata access services but is unable to process it because the access service is not active.",
"exceptionUserAction": "If the org.odpi.openmetadata.accessservices.subjectarea.server is supposed to have this access service activated, correct the org.odpi.openmetadata.accessservices.subjectarea.server configuration and restart the org.odpi.openmetadata.accessservices.subjectarea.server."
}
Opening up GI issue to track implementation of Governance Engine OMAS
(being migrated from a patch using apache atlas base to the egeria base)
The Enterprise OMRS Repository Connector has multiple TODOs identified around the way results are compared, collated and sequenced. This needs some focus to ensure its results processing is bulletproof.
When attempting to configure my egeria install I hit a code path where I get the following exception:
java.lang.NullPointerException: null
at org.odpi.openmetadata.frameworks.connectors.ConnectorBroker.getConnection(ConnectorBroker.java:260) ~[classes/:na]
at org.odpi.openmetadata.frameworks.connectors.ConnectorBroker.getConnector(ConnectorBroker.java:471) ~[classes/:na]
at org.odpi.openmetadata.frameworks.connectors.ConnectorBroker.getConnector(ConnectorBroker.java:331) ~[classes/:na]
at org.odpi.openmetadata.repositoryservices.metadatahighway.OMRSMetadataHighwayManager.getTopicConnector(OMRSMetadataHighwayManager.java:394) ~[classes/:na]
This appears to be caused in getConnection:
/**
* Extract the connection from the embedded connection and push any arguments into the
* AdditionalProperties for the connection.
*
* @param embeddedConnection embedded connection object.
* @return connection object augmented with the arguments from the embedded connection.
*/
private ConnectionProperties getConnection(EmbeddedConnectionProperties embeddedConnection)
{
ConnectionProperties connection = null;
if (embeddedConnection != null)
{
AccessibleConnection accessibleConnection = new AccessibleConnection(embeddedConnection.getConnectionProperties());
Connection connectionBean = accessibleConnection.getConnectionBean();
AdditionalProperties arguments = embeddedConnection.getArguments();
if (arguments != null)
{
Map<String, Object> additionalProperties = connectionBean.getAdditionalProperties();
Iterator<String> argumentNames = arguments.getPropertyNames();
while (argumentNames.hasNext())
{
String argumentName = argumentNames.next();
additionalProperties.put(argumentName, arguments.getProperty(argumentName));
}
connectionBean.setAdditionalProperties(additionalProperties);
}
It occurs as additionalProperties is null, even though the connectionBean is ok
It may be worth validating this & capturing information for the returned exception, though I've not yet identified why in particular this NPE is hit during the configuration of my server
Did a clean maven build with latest branch master. But it failed with following messages:
~/Documents/GitHub/egeria$ mvn clean install -DskipTests -DskipCheck=true -Drat.skip=true
[INFO] Scanning for projects...
[ERROR] [ERROR] Some problems were encountered while processing the POMs:
[FATAL] Non-resolvable parent POM for org.odpi.egeria:open-metadata-subject-area-client-samples:[unknown-version]: Could not find artifact org.odpi.egeria:egeria:pom:0.1-SNAPSHOT and 'parent.relativePath' points at wrong local POM @ line 8, column 13
[WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: org.odpi.egeria:repository-services-implementation:jar -> duplicate declaration of version ${open-metadata.version} @ org.odpi.egeria:subject-area-tools:[unknown-version], /Users/chen.yang10ibm.com/Documents/GitHub/egeria/open-metadata-implementation/access-services/subject-area/subject-area-tools/pom.xml, line 24, column 21
[WARNING] 'build.plugins.plugin.version' for org.apache.maven.plugins:maven-jar-plugin is missing. @ org.odpi.egeria:server-chassis-spring:[unknown-version], /Users/chen.yang10ibm.com/Documents/GitHub/egeria/open-metadata-implementation/governance-servers/server-chassis/server-chassis-spring/pom.xml, line 99, column 21
[WARNING] 'build.plugins.plugin.version' for org.springframework.boot:spring-boot-maven-plugin is missing. @ org.odpi.egeria:server-chassis-spring:[unknown-version], /Users/chen.yang10ibm.com/Documents/GitHub/egeria/open-metadata-implementation/governance-servers/server-chassis/server-chassis-spring/pom.xml, line 103, column 21
@
[ERROR] The build could not read 1 project -> [Help 1]
[ERROR]
[ERROR] The project org.odpi.egeria:open-metadata-subject-area-client-samples:[unknown-version] (/Users/chen.yang10ibm.com/Documents/GitHub/egeria/open-metadata-resources/open-metadata-samples/open-metadata-subjectarea-client-samples/pom.xml) has 1 error
[ERROR] Non-resolvable parent POM for org.odpi.egeria:open-metadata-subject-area-client-samples:[unknown-version]: Could not find artifact org.odpi.egeria:egeria:pom:0.1-SNAPSHOT and 'parent.relativePath' points at wrong local POM @ line 8, column 13 -> [Help 2]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ProjectBuildingException
[ERROR] [Help 2] http://cwiki.apache.org/confluence/display/MAVEN/UnresolvableModelException
Issue to track documentation cleanups:
OMRS repository connectors have the ability to store entity proxies to support relationships to entities that are stored/managed in a different repository.
Today the OMASs are responsible for creating the entity proxies but this is not optimal. It would be an easy extention to the Enterprise OMRS Repository Connector implementation of addRelationship() to automatically create the entity proxies in a repository if needed.
Opening issue to checkin the first pass of the Governance Engine OMAS Server code which has been migrated from a Atlas patch base.
Depends on Issue 33 code being merged first
The OMRSReposiotryContentManager is responsible for processing inbound TypeDef events. However, there are TODOs requesting auditlog messages and some empty methods.
Fix build WARNINGS and ERRORS:
[WARNING]
[WARNING] Some problems were encountered while building the effective model for org.odpi.egeria:governance-engine-client:jar:0.1-SNAPSHOT
[WARNING] 'dependencies.dependency.version' for org.junit.jupiter:junit-jupiter-api:jar is either LATEST or RELEASE (both of them are being deprecated) @ org.odpi.egeria:go
vernance-engine-client:[unknown-version], C:\Users\NO53JV.AD\Projects\egeria10\open-metadata-implementation\access-services\governance-engine\governance-engine-client\pom.x
ml, line 42, column 22
[WARNING]
[WARNING] Some problems were encountered while building the effective model for org.odpi.egeria:governance-engine:pom:0.1-SNAPSHOT
[WARNING] 'dependencies.dependency.version' for org.junit.jupiter:junit-jupiter-api:jar is either LATEST or RELEASE (both of them are being deprecated) @ org.odpi.egeria:go
vernance-engine:[unknown-version], C:\Users\NO53JV.AD\Projects\egeria10\open-metadata-implementation\access-services\governance-engine\pom.xml, line 78, column 22
[WARNING] 'dependencies.dependency.(groupId:artifactId:type:classifier)' must be unique: org.junit.jupiter:junit-jupiter-api:jar -> version ${junit.jupiter.version} vs RELE
ASE @ org.odpi.egeria:governance-engine:[unknown-version], C:\Users\NO53JV.AD\Projects\egeria10\open-metadata-implementation\access-services\governance-engine\pom.xml, line
75, column 21
[WARNING]
[WARNING] It is highly recommended to fix these problems because they threaten the stability of your build.
[WARNING]
[WARNING] For this reason, future Maven versions might no longer support building such malformed projects.
[WARNING]
[ERROR] [ERROR] The projects in the reactor contain a cyclic reference: Edge between 'Vertex{label='org.odpi.egeria:connector-configuration-factory:0.1-SNAPSHOT'}' and 'Ver
tex{label='org.odpi.egeria:cohort-registry-file-store-connector:0.1-SNAPSHOT'}' introduces to cycle in the graph org.odpi.egeria:cohort-registry-file-store-connector:0.1-SN
APSHOT --> org.odpi.egeria:connector-configuration-factory:0.1-SNAPSHOT --> org.odpi.egeria:cohort-registry-file-store-connector:0.1-SNAPSHOT @
[ERROR] The projects in the reactor contain a cyclic reference: Edge between 'Vertex{label='org.odpi.egeria:connector-configuration-factory:0.1-SNAPSHOT'}' and 'Vertex{labe
l='org.odpi.egeria:cohort-registry-file-store-connector:0.1-SNAPSHOT'}' introduces to cycle in the graph org.odpi.egeria:cohort-registry-file-store-connector:0.1-SNAPSHOT -
-> org.odpi.egeria:connector-configuration-factory:0.1-SNAPSHOT --> org.odpi.egeria:cohort-registry-file-store-connector:0.1-SNAPSHOT -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/ProjectCycleException
Checking cross-project versions
log4j - 1.2.17 - this is current
jackson - 2.9.2 - this is 2017 (2.9.5 is current). May be worth considering, with testing
spring - 1.5.7. Actually this is the springboot version. We should rename the label. Currently on 2.0.2, but that's a significant change, but as we're just getting started worth talking about the pros/cons. 1.5.13 is the current version in the 1.x stream
commons-io - 2.4 - 2.6 is current, whilst 2.4 is from 2012. Unless we have a good reason not to I strongly feel we should be more current (security fixes etc)
commons-collections - 3.2.2. Current in the 3.x stream, though we have 4.1 equivs available. May need some porting but now may be a good time
testng - 6.9.4 - This is from May 2015, with current versions at 6.14.3, I would think we should uipdate this
enunciate - 2.10.1 (oct 2017) - fairly current though 2.11.1 is available
Now may be a good time to uplift some of these. Opening for discussion
When attempting to invoke {{baseURL}}/open-metadata/repository-services/users/{{user}}/instances/entity
where {{}} are relevant variables
it appears the code in OMRSRestRepository.java is not invoked (was trying to debug) but there is an issue hit earlier on in the spring framework to do with the structure of the request body
2018-09-17 15:50:53.310 WARN 6144 --- [nio-8080-exec-2] .w.s.m.s.DefaultHandlerExceptionResolver : Failed to read HTTP message: org.springframework.http.converter.HttpMessageNotReadableException: JSON parse error: Missing type id when trying to resolve subtype of [simple type, class org.odpi.openmetadata.repositoryservices.rest.properties.EntityCreateRequest]: missing type id property 'class'; nested exception is com.fasterxml.jackson.databind.exc.InvalidTypeIdException: Missing type id when trying to resolve subtype of [simple type, class org.odpi.openmetadata.repositoryservices.rest.properties.EntityCreateRequest]: missing type id property 'class'
As per EGERIA-6
See https://jira.odpi.org/browse/EGERIA-2 for full description.
Temporarily loosening java version check to 1.8 instead of 1.8.0_151 to allow openjdk builds to be used
Currently 'classificationdefs' will return all typedefs in the category CLASSIFICATION_DEF
However this includes all kinds of classifications - such as WebServer, Application server... not just governance classifications
Opening this issue as a reminder that this needs refining
This may require API changes. For now all classification defs are returned
Adding spdx identifier
When an omag server is configured a file
omag.server.SERVERNAME.config
is created in the directory where the OMAGApplication is run
Until/ unless any work to change the default location of the config file etc is done, I propose adding to .gitignore as there should be no need to have these files under SCM
[ERROR] Failed to execute goal on project asset-catalog-server: Could not resolve dependencies for project org.odpi.egeria:asset-catalog-server:jar:0.1-SNAPSHOT: Could not find artifact org.apache.atlas:omrs:jar:2.0.0-SNAPSHOT -> [Help 1]
[ERROR]
This is due to pom fragment
org.apache.atlas
omrs
2.0.0-SNAPSHOT
This does not exist in maven central
Furthermore open metadata code should not be directly dependent on atlas -- the dependency should be the other way around - atlas plugins may depend on open metadata modules.
The open metadata archives allow changes to be made to TypeDefs through a patching mechanism. The support is available but the processing in OMRS Repository Content helper to applyPatch() just has the code structure but no implementation.
See applyPatch() method in OMRSRepositoryContentHelper.java
Using Coco Pharmaceuticals persona to drive the scenario, provide Egeria capability with supporting resources to explain how Jules Keeper, the new CDO, builds a team to beef up their governance programs.
This scenario needs to show how Egeria helps teams focused on different governance domains to collaborate to create an integrated governance ecosystem for the business.
Highlighted portion below isn't clear:
"Administration Services drive the initialization of the OMRS at server startup plus access to the OMRS's internal status. It relies on the OMAG Server to supply it with the configuration information to initialize the connectors it should use and connectors with other open metadata servers."
not certain if this should be:
"... It relies on the OMAG Server to supply it with the configuration information to initialize the connectors it should use and connects with other open metadata servers."
or:
"...It relies on the OMAG Server to supply it with the configuration information to initialize the connectors it should use and connectors to other open metadata servers."
(open-metadata-implementation/repository-services/docs/README.md
)
Every OCF connector returns properties about the resource it represents. The EnterpriseOMRSRepositoryConnector should return information about the cohort iti s connecting to. This information is current null and needs to be filled out through queries to the connector manager.
Is this statement still accurate?
"C - Other metadata repositories that have their own API can create a connector to translate OMRS calls to call to their API. The IGC OMRS Repository Connector is an example of this type of OMRS connector written to support a specific metadata repository that is not Apache Atlas. This OMRS Connector will be included in the Apache Atlas build so it can be called directly from the Enterprise OMRS Connector rather than via a remote REST call."
(open-metadata-implementation/repository-services/docs/component-descriptions/connectors/repository-connector.md
)
Missed root path component (likely as written before other services were included and mapping decided).
Will add back
See https://jira.odpi.org/browse/EGERIA-2 for full description.
Re-introducing tighter java version check once openjdk fixed
The open metadata archive has placeholders for the type definitions for logic specifications and models. These are part of Area 7 but haven not been fully spec'ed yet.
Reported by IntelliJ .gitignore plugin:
# in directory: /Users/jonesn/IdeaProjects/egeria
git rm --cached --force "open-metadata-implementation/access-services/subject-area/subject-area-server/src/test/java/org/odpi/openmetadata/accessservices/subjectarea/generated/server/TestSubjectAreaBeansToAccessOMRS.java"
Actioned.
Verified build still clean, Unit test runs clean
Missing license
Issue to track progress in GE server implementation
If the license is not present, build fails.
In the maven fail message is the path of the rat.txt file with details about what files don't have the proper license.
Currently in the top level pom we have:
3.0.4
However this is deprecated in maven 3 (there's references at https://maven.apache.org/ref/3.5.2/maven-model/apidocs/org/apache/maven/model/Prerequisites.html & mojohaus/versions#210 - in fact it simply doesn't work any more for non plugin projects
In AtIas this was addressed by changing to use:
<!-- Validates maven & java versions -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-enforcer-plugin</artifactId>
<version>1.4.1</version>
<executions>
<execution>
<id>enforce-versions</id>
<goals>
<goal>enforce</goal>
</goals>
<configuration>
<rules>
<requireMavenVersion>
<version>[3.5.0,)</version>
<message>** MAVEN VERSION ERROR ** Maven 3.5.0 or above is required. See
https://maven.apache.org/install.html
</message>
</requireMavenVersion>
<requireJavaVersion>
<level>ERROR</level>
<version>[1.8.0-151,)</version>
<message>** JAVA VERSION ERROR ** Java 8 (Update 151) or above is required.
</message>
</requireJavaVersion>
<requireJavaVersion>
<level>WARN</level>
<version>(,1.9]</version>
<message>** JAVA VERSION WARNING ** Java 9 and above has not been tested.
</message>
</requireJavaVersion>
</rules>
</configuration>
</execution>
</executions>
</plugin>
I propose to submit a patch request with a similar change for egeria.
I will leave the versions as-is for the initial patch as we have been ok with this change in Atlas. However we could consider if we want to move forward to
java - 171 (current default service release, 172 is latest)
maven 3.5.3
Please raise a new issue to make these changes if required
At the VDC meeting in Amsterdam, Aug 2018 we identified the following changes
When developing GE-omas I used 'getPropertiesDefinition' on TypeDef to get a list of optional attributes & iterate (to pass to ranger)
I missed out a check for null - so NPE, but I wonder if we would be better returning an empty collection
in these cases, rather than a null? Collections such as lists are often used with iterators, and unless there's a particular reason to use null, an empty list seems easier?
Open for discussion.
INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 3.108 s
[INFO] Finished at: 2018-07-25T11:01:23+01:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.rat:apache-rat-plugin:0.12:check (rat-check) on project egeria: Too many files with unapproved license: 1 See RAT report in: /Users/jonesn/IdeaProjects/egeria/target/rat.txt -> [Help 1]
Printing headers for text files without a valid license header...
All members of this project agree to adhere to the ODPi Code of Conduct listed at https://github.com/odpi/specs/wiki/ODPi-Code-of-Conduct
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.