lightblue-platform / lightblue-client Goto Github PK

The recommendation I have is to model license stuff off of https://github.com/manageiq/manageiq
What I see in this repo is:

• LICENSE.txt - blanket license / copyright informaiton
• LICENSE.AL - Apache License 2.0 text
• LICENSE.GPL - GPL text

Let's follow this instead of peppering all source with copyright info. Once this is put in place in this repo, I say we replicate in all our repos.

Note, the party for the copyright should be "Lightblue Authors".

This not really a bug or an enhancement, more of just a "code smell" IMHO. It seems odd that the top level interface LightblueRequest has setters because it...

• Forces mutability at the interface level
• Forces subtypes to deal with a setBody API that could potentially overwrite the request body the subtype has already set up

It seems that we now have two exceptions that could occur for a lightblue response from the lightblue-client, they seem to mean similar things: LightblueException & LightblueErrorResponseException. I don't feel that we need both.

LightblueErrorResponseException could probably be eliminated as the DefaultLightblueResponse's constructor now throws LightblueException is an error is on the response.

That said, LightblueException has a laundry list of error codes that I am not sure why they are there. I am concerned that this list will quickly become unmaintained and I question the value.

What uses the constant error codes and are there any good reason why they should not be removed?

Reported BLOCKER issue by sonar on this line.

Correctness - Impossible downcast of toArray() result

This code is casting the result of calling toArray() on a collection to a type more specific than Object[], as in:
  String[] getAsArray(Collection c) {
    return (String[]) c.toArray();
  }
This will usually fail by throwing a ClassCastException. The toArray() of almost all collections return an Object[]. They can't really do anything else, since the Collection object has no reference to the declared generic type of the collection.
The correct way to do get an array of a specific type from a collection is to use c.toArray(new String[]); or c.toArray(new String[c.size()]); (the latter is slightly more efficient).
There is one common/known exception exception to this. The toArray() method of lists returned by Arrays.asList(...) will return a covariantly typed array. For example, Arrays.asArray(new String[] { "a" }).toArray() will return a String []. FindBugs attempts to detect and suppress such cases, but may miss some.

Packages are a logical organization of source code based on responsibilities. Putting all enum classes in a separate package is inconvenient, wrong, and ugly. Following the same paradigm, all applications can have three packages: enums, interfaces, concrete classes.

Put the enum in the package it is used.

From https://hc.apache.org/httpcomponents-client-ga/tutorial/html/connmgmt.html

2.1. Connection persistence
The process of establishing a connection from one host to another is quite complex and involves multiple packet exchanges between two endpoints, which can be quite time consuming. The overhead of connection handshaking can be significant, especially for small HTTP messages. One can achieve a much higher data throughput if open connections can be re-used to execute multiple requests.

Sure, it makes it easy for clients to use because they don't have to worry about handling an HttpClient in their code, but this seems lazy to me. There should be a better way for clients to manage the life cycle of an HttpClient in various service and web frameworks that they use, rather than have the service client make a new connection for every service call. Currently the lightblue client gives no such option; it must manage the HttpClient, and gives no interface for a consumer to close it at some point, so it must close it after every request.

Bug found in lightblue-platform/lightblue-migrator#162. The response is valid json, so this will need to be investigated and fixed.

Caused by: com.redhat.lightblue.client.http.LightblueHttpClientException: Error sending lightblue request: {"query":{"field":"whenAvailableDate","op":">=","rvalue":"20150409T22:39:14.769+0000"},"projection":[{"field":"*","include":true,"recursive":true}],"sort":[{"whenAvailableDate":"$asc"}], "range": [0,0]}
        at com.redhat.lightblue.client.http.LightblueHttpClient.data(LightblueHttpClient.java:137)
        at com.redhat.lightblue.client.hystrix.LightblueHystrixClient$DataTypeHystrixCommand.run(LightblueHystrixClient.java:69)
        at com.netflix.hystrix.HystrixCommand$1.call(HystrixCommand.java:294)
        ... 48 more
Caused by: com.redhat.lightblue.client.response.LightblueResponseParseException: Error parsing lightblue response: {"status":"COMPLETE","modifiedCount":0,"matchCount":30351,"processed":[{"creationDate":"20150402T13:54:05.000+0000","expectedExecutionMilliseconds":30000,"startDate":"20150409T22:00:00.000+0000","createdBy":"nmalik","_id":"termsJob_321","lastUpdatedBy":"nmalik","jobExecutions":[],"endDate":"20150409T23:00:00.000+0000","whenAvailableDate":"20150409T23:01:00.000+0000","configurationName":"terms","objectType":"migrationJob","lastUpdateDate":"20150402T13:54:05.000+0000","jobExecutions#":0}]}

        at com.redhat.lightblue.client.response.LightblueResponse.parseProcessed(LightblueResponse.java:107)
        at com.redhat.lightblue.client.http.LightblueHttpClient.data(LightblueHttpClient.java:135)
        ... 50 more
Caused by: com.fasterxml.jackson.databind.JsonMappingException: Can not deserialize instance of com.redhat.lightblue.migrator.consistency.MigrationJob out of START_ARRAY token
 at [Source: N/A; line: -1, column: -1]
        at com.fasterxml.jackson.databind.JsonMappingException.from(JsonMappingException.java:164)
        at com.fasterxml.jackson.databind.DeserializationContext.mappingException(DeserializationContext.java:575)
        at com.fasterxml.jackson.databind.DeserializationContext.mappingException(DeserializationContext.java:569)
        at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.deserializeFromArray(BeanDeserializerBase.java:1121)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer._deserializeOther(BeanDeserializer.java:148)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:123)
        at com.fasterxml.jackson.databind.ObjectMapper._readValue(ObjectMapper.java:2860)
        at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:1569)
        at com.redhat.lightblue.client.response.LightblueResponse.parseProcessed(LightblueResponse.java:105)
        ... 51 more

The client itself logs all requests and responses using slf4j. I think this responsibility should be left up to consumers of the client. Isn't it a security issue to be logging all request and response bodies? I imagine different consumers will be storing data in Lightblue that may require masking before logging, for example.

Instead of SSL and client cert auth being hard coded (see LightblueHttpClientCertAuth) these should be options. And probably break this client out into another module or git repo (http://dev.forum.lightblue.io/Client-http-library-td124.html)

This issue was moved from lightblue-platform/lightblue-applications#14

For example:

new MetadataGetEntityNamesRequest("user", "1.0.0").getRestURI("http://lightblue.com/metadata/rest");

will return: http://lightblue.com/metadata/rest/user/1.0.0

Which is get metadata request

Likewise, if you pass only entity, it is a get versions request.

Expressions do not require quotation marks around string fields.

I know this is true for ValueQuery expressions, but we should check other expressions as well to ensure consistency.

A static DEFAULT_MAPPER was on LightblueHttpClient until #64 which moved it to LightblueResponse. There is also one on a JSON class. Given DEFAULT_MAPPER sets up some things like date format and it can be reused it would be good to use one instance by default.

The lightblue endpoint I was using was down for a bit (due to OpenShift maintenance). I received an HTML based response, however the stacktrace showed a Jackson parse exception. Even though this made sense after the fact I feel that it could be handled more gracefully. Below is an example stacktrace received:

com.fasterxml.jackson.core.JsonParseException: Unexpected character ('<' (code 60)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')
 at [Source: java.io.StringReader@2504577; line: 1, column: 2]
    com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1369)
    com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:532)
    com.fasterxml.jackson.core.base.ParserMinimalBase._reportUnexpectedChar(ParserMinimalBase.java:453)
    com.fasterxml.jackson.core.json.ReaderBasedJsonParser._handleUnexpectedValue(ReaderBasedJsonParser.java:1386)
    com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:669)
    com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:2926)
    com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:2873)
    com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:1750)
    com.redhat.lightblue.client.response.LightblueResponse.<init>(LightblueResponse.java:21)
    com.redhat.lightblue.client.http.LightblueHttpClient.callService(LightblueHttpClient.java:119)
    com.redhat.lightblue.client.http.LightblueHttpClient.data(LightblueHttpClient.java:96)
    com.redhat.lightblue.client.http.LightblueHttpClient.data(LightblueHttpClient.java:100)

lightblue-cilent.properties should be lightblue-client.properties

See https://github.com/lightblue-platform/lightblue-client/blob/master/http/src/main/java/com/redhat/lightblue/client/http/LightblueProxyServlet.java#L41

Few options:

• Accept a QueryExpression and Projection(s) in constructor. This prevents the DataFindRequest from being allow to be in an invalid state which is a bad practice for constructors (allowing objects which will never actually work without more configuration).
• Throw something other than NPE if either are omitted (but this is still failing at runtime which is bad IMHO)
• Use a default QueryExpression and Projection if either is omitted (then you could also implement the first suggestion)

As an aside, this is why the Builder pattern is much prefered than mutable objects:

• Immutable objects are always +1
• You prevent objects being in invalid state
• You get "named parameters" from chainable methods in a Builder (some call this "fluent") so the code is more readable

So I guess I'm suggesting IMHO moving all of these requests to a builder pattern where the final object is immutable and guaranteed valid, but ultimately anything other than throwing an NPE would be nice.

Pulled in from https://github.com/lightblue-platform/secant/issues/13

To configure the project (and sub-projects) to use: travis, coveralls.io, jenkins (making sure to trigger the proper dependencies) and sonar.

Expression classes are building JSON as strings in toJson() method. This is error prone, you're not escaping special characters, and creating situations where invalid json can be created. Instead, a json framework (lightblue uses jackson databind) should be used to build a JSON document tree, and that should be converted to string. This pattern is implemented as follows in lightblue:

class SomeClass {
public JsonNode toJson() {
// build a json node and return
}

public String toString() {
return toJson().toString();
}
}

No results matching a query provided is not an exceptional situation. Return empty list instead.

To follow standard: https://github.com/lightblue-platform/lightblue-docs-user/blob/master/standards/versioning.md

(gitbook.io is down right now..)

Everything under .../client (no subdirs) would make it much easier to use and maintain this package.

"clientconfig.properties" isn't very specific. Config files for lightblue metadata and crud are prefixed with "ligthblue-", maybe this could be used here? Though we do have some generic file names like "datasources.json" but those are definitely deployed in a module.

There should be a better way to do

{ $append : { someArray : {} } }

other than

new AppendUpdate("someArray", new ObjectRValue(new HashMap<>()))

The only dependency I saw was DateType, used to get the date format. I think there is some risk to not doing this in that metadata could change the date format (very unlikely!) but, if client didn't update the dependency version it would use the wrong format. So it's not much different than defining the format in the client code.

Am I missing another dependency? Thoughts on removing this dependency?

If you want to load certificates from a system path, you need to use file:// prefix in the lightblue client configuration file. Omitting the prefix will result in java.security.cert.CertificateException: Missing input stream exceptions, which will make you check the paths, filesystem permissions, etc. and generally waste your time. The exception message needs to be updated with a short explanation of this behavior.

Related to 4a6932b.

This issue was originally opened in the secant repo (https://github.com/lightblue-platform/secant/issues/1)

From @jewzaam

A thought I had a while back about aggregation was what if we did it in the client? Downsides are:

each client implementation has to implement this logic if they want it
loss of server side performance (network, direct join for example in RDBMS)
But the up-side is clients can get this feature sooner than it's ready on the server.

What is aggregation? Think of a join in SQL. Aggregation is bringing together data from multiple entities into a single result set.

MetadataCreateSchemaRequest does not have a way to define schema path param (required)
MetadataGetEntityNamesRequest does not have a way to define status (s) path param (optional)

Compression can improve response times considerably. Lightblue server already supports it, now is the time to implement this functionality in the java client.

We're relying on this project: https://github.com/svanoort/rest-compress/

From a cursory glance at LightblueHttpClient, it looks like each call to callService creates a new instance of CloseableHttpClient. I imagine this forces a new SSL/TLS handshake on each call (somewhat significant overhead when using HTTPS). Would it be possible to reuse instances of CloseableHttpClient here?

As more applications start using lightblue and the client, I could easily see a demand for having the option to public hystrix metrics. I know this is something that has already been mentioned as a nice-to-have for the migrator.

The Query hierarchy sometimes has static factory methods for constructors, but not always. I think we should be consistent. Either all constructors also have a factory method, or none of them do.

In general, I find myself asking if I can directly call the constructor, then why do I need a static factory method? But I don't really care, so long as we are consistent.

There does not appear to be a way to perform an $in or $nin through any of the Query implementations. It seems to me that ValueQuery would make the most sense.

The only difference is that they read their url from a different property. There could just be one with a generic property like, "lightblue service uri." Not sure why I didn't realize this at the time.

Http is like:

com.redhat.lightblue.client
http
1.2.0-SNAPSHOT

And it should be like:

com.redhat.lightblue.rest
lightblue-rest-metadata
1.3.0-SNAPSHOT
classes

Projection is an optional element, but it is added to the request string unconditionally, without checking if it is null.

Just discovered the code doesn't work if the cert alias doesn't match the name, so let's just compute the alias from the name and have one less thing to configure

Appears that ApacheHttpClients has been preferred over HttpClientNoAuth

To reproduce, add the client as a dependency to a maven project, and then look at the imported JARs.

The jar names would be core-{version}.jar and http-{version}.jar.

I think the names should be changed to include lightblue-client as prefixes. This would fit the convention most projects follow (ex. commons-lang, commons-io, jackson-core-asl).

If the change were made, then the new jar names would make more sense: lightblue-client-core-{version}.jar, lightblue-client-http-{version}.jar.

From spec: PUT /metadata/{entityName}/{version}/{status}?comment={Change comment}

Not sure what PUT/metadata/{entityName}/{version}/ will do, but that's the only(ish) request that this class supports currently

Once lightblue-platform/lightblue-core#297 is merged, the code should handle the new query types: NaryValueRelationalExpression and NaryFieldRelationalExpression instead of NaryRelationalExpression.

All migrated documents contain data as following:

"createdBy":"lightblue-consistency-checker",
"lastUpdatedBy":"lightblue-consistency-checker",
"migrationDate":"20150504T18:08:42.160+0000",
"migratedFrom":"jdbc:oracle:thin:@webdev-scan.corp.redhat.com:20010:WEBDEV1"

The applications don't care about those fields and it would be nice to filter them out at the lightblue client level, without a need to specify the filtering projection for each call.

I'm thinking about a new lightblue-client feature:

lightblueClient.setProjection(excludeField("migrationDate"), excludeField("migratedFrom"))

A client configured like this would append specified projections to every call. This will produce cleaner json and can slightly improve response times.

As a follow up to #50, if there are no further uses of these classes then can they be removed?

Instead of:
withValue("scheduledDate",ExpressionOperation.LTE,ClientConstants.getDateFormat().format(new Date())

I should be able to write:
withValue("scheduledDate",ExpressionOperation.LTE,new Date())

The implementation should check if the value is date, and format it.

This is another side effect of not using a proper json library to construct json:

This generates invalid json:

new SetUpdate(new PathValuePair("executions.-1.status",new LiteralRValue(MigrationJob.STATE_ACTIVE)

The literalRValue does not quote the string.

Use jackson to build json, do not build json as a string.

Currently only a resource path is allowed, but it would be useful to also be able to specify the absolute system path. This is needed for lightblue-migrator which is not running in a jboss container.

Right now the client has it hard-coded to accept all certificates (SSLConnectionSocketFactory.ALLOW_ALL_HOSTNAME_VERIFIER). This should be configurable to allow any of the available values (ALLOW_ALL_HOSTNAME_VERIFIER, BROWSER_COMPATIBLE_HOSTNAME_VERIFIER, STRICT_HOSTNAME_VERIFIER)

Pulled in from https://github.com/lightblue-platform/secant/issues/15