glytching / dragoman Goto Github PK

The build on Travis CI generates Javadocs.

Enabling Travis to publish these to Dragoman's GitHub pages would improve their visibility and, by extension, their usefulness.

The MongoDB datasource is currently configured with the following properties:

• mongo.host
• mongo.port

So, Dragoman assumes open access to that MongoDB instance. This is consistent with a single developer using their own MongoDB instance but it makes sense to extend the properties and the MongoProviderImpl to support an authenticated MongoDB server.

Expected Behavior

In the Dataset view, opening a dataset should close any existing dataset including cancelling any open subscription. Otherwise the user will be unable to cancel the subscription short of reopening the previously opened dataset. It seems more likely that the user will forget or will (reasonably) assume that the system will do the cancellation on the user's behalf.

Actual Behavior

Opening a dataset does not ensure that any dataset which was currently open is properly closed.

Steps to Reproduce the Problem

Please bear in mind that the more detail you provide the more likely it is that the issue will be resolved.

1. Open a dataset
2. Subscribe to it
3. Open another dataset
4. The subscriptionfor the first dataset remains active but (a) there is no way to display its events and (b) the user is unaware that it remains open

Specifications

• Dragoman Version: 1.0.0
• Browser Name: ANY
• Browser Version: ANY

Logs

N/A

Expected Behavior

It should be possible for Dragoman to handle some non JSON responses from HTTP datasets. It's not feasible for every possible response format to be supported but Dragoman should support some minimal response post processing for turning proprietary HTTP response formats into JSON.

Dragoman already has a ResponsePostProcessor wired into the HTTP reader flow, so perhaps we need to (a) provide implementations of this interface which can handle other formats (perhaps something regex based?) and (b) allow the dataset definition to reference or choose a ResponsePostProcessor implementation.

Actual Behavior

Some HTTP providers emit customised or wrapped responses. For example:

• Stackoverflow produces this (Dragoman would be interested in the items):

{
    "items": [ {...}, {...} ],
    "has_more": true,
    "quota_max": 300,
    "quota_remaining": 285
}

• Flickr produces this (Dragoman would be interested in the contents of jsonFlickrFeed(...))

jsonFlickrFeed({
	"title": "Recent Uploads tagged soccer",
	"link": "http:\/\/www.flickr.com\/photos\/tags\/soccer\/",
	"description": "",
	"modified": "2017-11-08T02:22:34Z",
	"generator": "http:\/\/www.flickr.com",
	"items": [ {...}, {...} ]
})

• EventBrite also wraps its results (Dragoman would be interested in the events):

{
    "pagination": { ... },
    "events": [ { ... }, { ... } ],
    "location": { ... }
}

Steps to Reproduce the Problem

1. Create a dataset for https://api.stackexchange.com//2.2/questions/unanswered?order=desc&sort=activity&site=stackoverflow
2. Open the dataset
3. Nothing is displayed
4. Create a dataset for http://www.flickr.com/services/feeds/photos_public.gne?tags=soccer&format=json
5. Open the dataset
6. Boom! ... Failed to deserialise content!

Specifications

• Dragoman Version: 1.0.0
• Browser Name: ANY
• Browser Version: ANY

Logs

2017-11-08 10:51:45,208|[vertx-worker-13]|INFO|o.g.d.web.resource.DatasetResource|Sampling dataset: org.glytching.dragoman.dataset.Dataset@57df8083[id=5a02e159c786a0894c1a81df,name=asdasds,owner=Colm,source=http://www.flickr.com/services/feeds/photos_public.gne?tags=soccer&format=json,subscriptionControlField=,subscriptionControlFieldPattern=]
2017-11-08 10:51:46,342|[vertx-worker-13]|INFO|o.g.d.http.okhttp.OkHttpClient|Get from: http://www.flickr.com/services/feeds/photos_public.gne?tags=soccer&format=json
2017-11-08 10:51:46,730|[vert.x-eventloop-thread-1]|INFO|o.g.d.web.GlobalExceptionHandler|Caught exception at the end of the route chain: {"timestamp":"2017-11-08T10:51:46.728","statusCode":500,"statusMessage":"Internal Server Error","path":"/dragoman/dataset/5a02e159c786a0894c1a81df/sample","message":"Failed to deserialise content!","stackTrace":"org.glytching.dragoman.transform.JsonTransformer.transform(JsonTransformer.java:49)\norg.glytching.dragoman.http.HttpClientAdapterImpl.read(HttpClientAdapterImpl.java:55)\norg.glytching.dragoman.store.http.repository.HttpRepository.find(HttpRepository.java:77)\norg.glytching.dragoman.reader.ReaderImpl.read(ReaderImpl.java:41)\norg.glytching.dragoman.web.resource.DatasetResource.getSample(DatasetResource.java:140)\norg.glytching.dragoman.web.resource.DatasetResource$$Lambda$88/71836497.handle(Unknown Source)\nio.vertx.ext.web.impl.BlockingHandlerDecorator.lambda$handle$0(BlockingHandlerDecorator.java:48)\nio.vertx.ext.web.impl.BlockingHandlerDecorator$$Lambda$146/1968839378.handle(Unknown Source)\nio.vertx.core.impl.ContextImpl.lambda$executeBlocking$1(ContextImpl.java:278)\nio.vertx.core.impl.ContextImpl$$Lambda$110/768715369.run(Unknown Source)\nio.vertx.core.impl.TaskQueue.run(TaskQueue.java:80)\nio.vertx.core.impl.TaskQueue$$Lambda$32/396918327.run(Unknown Source)\njava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)\njava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)\njava.lang.Thread.run(Thread.java:745)"}

Expected Behavior

When running the application in embedded mode a browser instance should be loaded pointing at the application on the dynamically assigned HTTP serve rport.

Actual Behavior

When running Dragoman in embedded mode the HTTP port is dynamically assigned. If a user wishes to use the web application they must look for this in the logs ...

o.g.dragoman.web.WebServerVerticle|Starting HTTP server on port: ...

... to discover the HTTP port. This is awkward.

Steps to Reproduce the Problem

1. Start Dragoman with -Denv=embedded
2. Trawl the logs for this statement: Starting HTTP server on port
3. Copy the port
4. Open a browser and enter: http://<host>:<port>/dragoman/about.hbs where <port> is the value copied from the logs
5. Now (finally :) you can use the application

Specifications

• Dragoman Version: 1.0.0
• Browser Name: ANY
• Browser Version: ANY

Logs

N/A

The core functionality ...

• Modelling Dataset
• Talking to MongoDB sources
• Talking to HTTP sources
• Subscriptions
• The SQL parsers etc

.. could be separated from the web application.

This would...

• Formalise separation of responsibilities (currently web applicaiotn concepts could bleed down through the layers)
• Facilitate the creation of other clients (scripting client, a Dragoman shell, drivers in other languages etc)

Consider a swagger-eque approach for auto generating docs form the Vert.x web router.

See vert-x3/vertx-web#113

Failing that, manually write the docs. The REST API isn't very large and is unlikely to be subject to significant change from multiple devlopers etc etc.

The purpose of Dragoman is:

to provide a sandbox, a place for trying out technologies

So, let's play around with JUnit 5.

See ...

• http://junit.org/junit5/docs/snapshot/user-guide/#migrating-from-junit4

Expected Behavior

When a client subscribes to a dataset they should receives events as those events occur in the undelrying dataset. Although this is complicated for HTTP datasets (by the presence or otherwise of streaming support from the underlying provider) it is possible for MongoDB backed datasets because:

• We are the underlying provider of the MongoDB backed datasets
• MongoDB does provide a event stream describing changes to its collection

So, let's integrate with that event stream and push events through the Dragoman server and out onto the event bus for consumption by the web app.

Actual Behavior

Subscriptions are currently implemented by polling the underlying dataset rather than by consuming an event stream emitted by the provider of the underlying dataset. So, every 15000ms the subscription task wakes up and re-queries the underlying dataset including a derived AsOf predicate to limit the results to those records which have been edited/created since the previous subscription period.

Steps to Reproduce the Problem

1. Subscribe to a MongoDB backed dataset
2. Update the underlying MongoDB collection for this dataset
3. Wait for the current poll period to elapse and then watch the event appear in the UI

The desired flow is:

1. Subscribe to a MongoDB backed dataset
2. Update the underlying MongoDB collection for this dataset
3. Watch the event appear in the UI ~immediately

Specifications

• Dragoman Version: 1.0.0
• Browser Name: ANY
• Browser Version: ANY

Logs

N/A

Expected Behavior

When subscribed to a dataset, new events should appear at the top of the view. It's reasonable to assume that anyone interested in subscribing expects to be presented with the new data rather than having to scroll down to the bottom of the dataset view every time a new event arrives.

Actual Behavior

When subscribed to a dataset, new events are appended to the bottom of the view.

Steps to Reproduce the Problem

Please bear in mind that the more detail you provide the more likely it is that the issue will be resolved.

1. Open a dataset
2. Subscribe to it
3. As changes are made to the underlying dataset watch them register in the dataset view ...
4. All new records are appended to the end of the dataset view, forcing the user to scroll to the bottom of the dataset view.

Specifications

• Dragoman Version: 1.0.0
• Browser Name: ANY
• Browser Version: ANY

Logs

N/A

Expected Behavior

Given a subscription on a dataset, when you attempt to create another - concurrent - subscription on the same dataset the system should prevent the second request.

There may be a follow up issue here to properly support concurrent subscriptions (should that be deemed wise, especially for HTTP sources!) but for starters let's just close this loophole since the current behaviour is clearly defective.

Actual Behavior

The system allows the second subscription ot be created with the result that the second subscription consumes the results from the first subscription.

Steps to Reproduce the Problem

Please bear in mind that the more detail you provide the more likely it is that the issue will be resolved.

1. Create a subscription on a dataset for a given select statement
2. Create a subscription on the same dataset for a different select statement
3. Update the underlying dataset (or wait for it to be updated) and then wait for the poll period to elapse
4. The first subscription receives its expected update
5. The second subscription receives the updates from the first subscription too i.e. the event stream is consistent with the first subscription's select statement rather than the second subscription's select statement

Specifications

• Dragoman Version: 1.0.0
• Browser Name: ANY
• Browser Version: ANY

Logs

N/A

The purpose of Dragoman is:

to provide a sandbox, a place for trying out technologies

So, let's play around with Java 9.

Expected Behavior

Clicking on the Logout icon when in a secured view (such as Datasets or Management) should redirect to the landing page.

Actual Behavior

Clicking on the Logout icon when in a secured view (such as Datasets or Management) results in the following error:

{
    "timestamp": "2017-11-08T09:53:57.368",
    "statusCode": 403,
    "statusMessage": "Forbidden",
    "path": "/dragoman/management/management.hbs",
    "message": "There is no user currently logged in!",
    "stackTrace": "org.glytching.dragoman.web.WebServerVerticle.fromSession(WebServerVerticle.java:204)\norg.glytching.dragoman.web.WebServerVerticle$$Lambda$59/1891170239.handle(Unknown ..."
}

Steps to Reproduce the Problem

1. Log in
2. Browse to the Management view
3. Log out
4. Boom!

Specifications

• Dragoman Version: 1.0.0
• Browser Name: ANY
• Browser Version: ANY

Logs

N/A