marklogic-community / grove-ml-gradle Goto Github PK

Some people have been asking for this, and had code that is covering a fair amount of it already. Opening this ticket and a PR to share and discuss it.

Should perhaps go into a separate repo, or perhaps in a samples branch. To be discussed.

• check and resolve pending PR's
• import from internal bitbucket to github
• add deprecation message to bitbucket
• migrate unresolved PR's
• prune unnecessary branches
• fix README, package*, NOTICE/LICENSE
• publish under @marklogic-community

I could not load the sample data with 10.0-2. Adding a placeholder ticket and assigning to me to follow back up on this.

For supporting app-level auth with custom login code

In the Grove and DHF documentation at https://marklogic-community.github.io/grove/guides/data-hub-framework/ one of the recommended steps for using Grove against a DHF environment is remove the configuration for all the databases except for your custom application's modules database so that they aren't created and you can just use the DHF Final database:

Delete content-database.json, schemas-database.json, and triggers-database.json from marklogic/ml-config/databases/.

This is fine for the content and triggers databases but there is a custom grove-ml-gradle task that runs when you run mlDeploy, setSchemasPermissions, that will fail as it always assumes that a schemas database has been created for your app (it's likely looking for appname-schemas or similar). I would recommend detecting if a schemas database config exists in ml-config and only running the task if it does.

At the moment, you can obviously work around this by only removing content-database.json and triggers-database.json. You will deploy an app-specific schemas database that will never get used but mlDeploy will complete.

<preferred-elements><element ns="" name="body"/></preferred-elements> <preferred-matches> <element ns="http://marklogic.com/entity-services" name="instance"/> <json-property>instance</json-property>

Default snippeting does not work with a complex object. instance in the DHF schema is always a complex object. Consider alternatives for how to snippet meaningfully.

Originally posted by @mariannemyers in #22

We should either support the Data Hub approach for Grove by default, or possibly have two mostly parallel branches: one standard Grove and one Grove for a Data Hub.

This could be a choice as part of Grove CLI, or perhaps as part of specific Gradle tasks/properties to support both types.

we should push out some common useful tasks into a grove.gradle file that can be easily included in grove-ml-gradle, as well as in dhf/dhs projects. That way we can greatly simplify the task of copying relevant gradle stuff from grove to dhf. It is currently typically omitted, but the mlcp related tasks, as well as the setting of ping properties is useful for dhf as well, even if it is just as example.

Using latest grove build on Mac OS, following command errors

// from inside marklogic dir
./gradlew mlDeploy

Error:

Unable to copy file to string from path: /Users/sjordan/.gradle/daemon/4.8/./ml-config/security/roles/app-role.json; cause: /Users/sjordan/.gradle/daemon/4.8/./ml-config/security/roles/app-role.json (No such file or directory)

Work around is to use --no-daemon flag
./gradlew --no-daemon mlDeploy

Provide an out of the box endpoint that can support the ping endpoint api in the middle-tier.

To match up with Datahub, and get some default patterns in place that will help DHF users out of the box.. (Type facet, instance extraction, etc)

Allows dropping mlLoadContent task as well. Use collections.properties to assign a config collection to those docs..

I just created a new project using Grove and tried to create a new document using the from on the UI but ran into the following TDE error after submitting the form:

{
  "errorResponse": {
    "statusCode": 500,
    "status": "Internal Server Error",
    "messageCode": "INTERNAL ERROR",
    "message": "TDE-INDEX: Error applying template /tde/people.json to document /all/15023029385779580941.json: TDE-EVALFAILED: Eval for Column _id='_id' returns an empty sequence for a non-nullable column . See the MarkLogic server error log for further detail."
  }
}

To prevent conflicts when moving over configs to DHF/DHS

You might see something like this using Java 10 or 12:

Task :mlLoadContent
20/04/16 10:58:29 INFO contentpump.LocalJobRunner: Content type is set to MIXED. The format of the inserted documents will be determined by the MIME type specification configured on MarkLogic Server.
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.apache.hadoop.security.authentication.util.KerberosUtil (file:/Users/poneill/.gradle/caches/modules-2/files-2.1/org.apache.hadoop/hadoop-auth/2.6.0/b0b8dec23a84ac8a0d00fbd69a87d320724ae34a/hadoop-auth-2.6.0.jar) to method sun.security.krb5.Config.getInstance()
WARNING: Please consider reporting this to the maintainers of org.apache.hadoop.security.authentication.util.KerberosUtil
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
20/04/16 10:58:30 INFO contentpump.ContentPump: Job name: local_1708497191_1
20/04/16 10:58:30 INFO contentpump.FileAndDirectoryInputFormat: Total input paths to process : 2
20/04/16 10:58:30 ERROR contentpump.LocalJobRunner: Error checking output specification:
20/04/16 10:58:30 ERROR contentpump.LocalJobRunner: java.lang.IllegalArgumentException: No enum: segment