marklogic-community / bitemp-explorer

An interactive UI for exploring bitemporal data and queries in MarkLogic.

"# bitemporal-demo Bitemporal + MarkLogic

To clone the project when you have git installed: git clone https://github.com/marklogic/bitemp-explorer.git If not, you can download the project as ZIP file.

INSTALL INSTRUCTIONS

1. Install MarkLogic 8: https://developer.marklogic.com/products and start the server
2. Navigate to http://localhost:8000/qconsole
3. On the right bar click the down arrow and choose "import workspace"
4. Import files in folder "WorkSpace-JS" or "WorkSpace-XQuery" depending on your language preference: Javascript or XQuery
5. Download and install node.js 0.10.29: https://nodejs.org/dist/v0.10.29/
6. Go to command line/terminal, cd under this repo directory, do "npm install"
7. Check to see that there is no ERR
8. Configure connection in env.js or create a local-env.js, if needed. local-env.js takes the same form as env.js, but does not get committed.
9. Deploy three REST extensions by running - "node deploy.js" -in your shell.
10. Run app with "node app.js"
11. go to http://localhost:3000/ to see a blank graph with title
12. go back to Query console and go from workspace 1 to 5, start from tab 1.

APP INSTRUCTIONS

13. Go through the workspaces.
14. It is recommended to go through at least the first two workspaces and create one or two temporal collections before trying the features of the demo.
15. On the home page, choose a temporal collection from the dropdown menu. Click the 'Search' button to get a list of all the documents within that temoral collection. Click one of the blue links to redirect you to the view page.
16. The graph on the view page is a visual representation of bitemporal, with 2 axes representing system and valid times of a temporal document.
17. The view page dropdown menu will drop down the properties of your documents being displayed in the graph. You can select and view new properties.
18. Click boxes (that represent physical documents) in the graph to edit, view, and delete certain documents. Edit will create a new physical document. View to see the contents of a document. The delete button will cut off a system end time to the current real time, or a time that you specify.
19. Click the 'Home' link in the upper lefthand corner to go back to the home page.
20. Back on the home page, you can also choose a system time operator and/or a valid time operator from the dropdown menus and drag the blue bars on the graph to select your ranges. Click the 'Search' button to display the documents specified by the temporal collection and the operators/time ranges.
21. To run another query, click the 'Reset' button and repeat the steps above.
22. Click the 'Create a Document' button to create a new logical document with its first physical document.
23. If a lot is new or confusing, look for highlighted words on the page for hovering over. A definition will pop up to help you cement your knowledge of bitemporal.

Guide to Workspaces:

MLW1-configure&basics.xml - set up temporal axis/collection, basic intro to MarkLogic

MLW2-Ingestion&Query.xml - temporal insert/update/delete examples and exercises, work together with node app on port 3000

MLW3-BitemporalTradeStore.xml - more complex data set (a trade store) exercises

MW4-Semantics.xml - intro to how bitemporal and semantics work together

MLW5-BitemporalLSQT.xml - special use case, proceed when done with 1 to 3

MLW6-IngestYourOwn - some helper code to ingest your own data

Documentation - Temporal Developer's Guide http://docs.marklogic.com/guide/temporal

bitemp-explorer's Issues

getDocColls in bitemp_graph_helper is not working with xml

Eliminate synchronous AJAX calls

To keep the UI responsive, avoid making synchronous AJAX calls. In other words, restructure code so that we can remove "async: false". The way to do this is to have any work that depends on the AJAX call done (or called) by the success function.

Add 'text' search to Search page

• allow a user to do a text search to search through documents

Make search work for any sys/val time properties

Currently the code expects to find properties called "sysStart", "sysEnd", "valStart", and "valEnd". Modify the search code so that it uses the range indexes specified for the axis creation for the specific temporal collection.

Edit a document

Allow the user to modify the contents of a temporal document.

• Add an "Edit" button below the chart
• Click on a physical document box to select a physical document
• When the button is clicked:
  • provide an editor for the content
  • populate the editor with the current contents of the physical document
  • display "Save" and "Cancel" buttons
  • on Save, PUT the updated content to /v1/documents
  • on Cancel, hide the editor and buttons

Add ‘Create a Temporal Collection’ button

• on the search page, add a button that allows a user to create a Temporal Collection
• allow the user to enter their own axes
• allow the user to enter their own system and valid timestamps (ex. “mySystem”, “myValid”, “sysStart”, “sysEnd”, “valStart”, “valEnd”)

Create a search page

Create a new page at /search.html.

• provide a drop list labeled "Temporal Collection"
• populate the Temporal Collection drop list with values from GET /manage/v2/databases/{id|name}/temporal/collections
  • the database will be the default database for the application server that the client is talking to
• provide a search button
  • when clicked, run a query with the specified temporal collection, returning one page of results (10)
  • this will be a call to /v1/search
• for each result, display the following:
  • document URI
  • system time span
  • valid time span

Deleting a temporal document currently assumes the logical document has a json format. Change code to allow for xml documents to work as well.

Revise code such that JSHint is clean

Right now, JSHint is throwing warnings on most of the JavaScript files. Update files so that JSHint does not show warnings.

Modify search results display to show "Infinity"

For any system or valid time where the value is 9999-12-31T23:59:59.99Z, display the dateTime as "Infinity" in the search results. This applies to search.html.

Show bitemp triples

Not sure what this would look like, but it would be useful to illustrate bitemp triples somehow.

Create a document

Provide the ability for the user to create a new temporal document (ie, a document with a new URI, not just editing an existing document). This functionality will be on index.html.

• present the list of available temporal collections to the user
• user selects a temporal collection
• identify the axes required for documents in that collection
• provide an input box for the document URI
• provide a textarea for document content
• pre-populate the text area with the appropriate axis JSON properties or XML elements (sysStart, sysEnd, valStart, valEnd)
• provide Save and Cancel buttons
• after successful Save, update the display to show this new logical document
• after a failed Save, display an error message to the user

Before beginning work, present your ideas for the UI to Dave.

Viewing with xml documents is buggy

Lukas working on this, when viewing a doc and clickin on other xml docs, the content displaying in the doc is also showing json contents

Editing a document works on Mac, but not Linux or Windows

#7 is complete for the user interface and works correctly on the Mac. However, the Node.js middle tier is not passing along the body of the PUT request when running on Windows or Linux. This results in a MarkLogic-level error:

Status 500: XDMP-NOMATCH: No matches for cts:element-reference(fn:QName("","sysStart"),("type=dateTime")) in document{}

Delete a logical document

From the single-document display, allow the user to do a temporal delete of a document.

• Add a "Delete" button below the chart
• When the button is clicked, send a DELETE message to /v1/documents
• On success, send a request for new data and update the display
• On failure, show a message that the delete failed

Move 'Create a Doc' Button to Search.html

Reduce page load time

• use .min versions for jQuery, jQuery UI, d3, moment
• currently loading both jquery.min.js and jquery-1.10.2.js. Eliminate one of these and retest. (jquery.min.js is version 2.1.1, so we should probably keep that one)
• move CSS links from body to head

Display system time and valid times

Dependency: #8

On the Search page, display an area that shows the system time range on the horizontal axis and the valid time range on the vertical axis.

• the range of system and valid times must be based on the range of date/times in the currently specified temporal collection (see #8)
  • this will likely require a REST API extension

Specify a system time search

Dependency: #9

A system time search will be specified using either one or two vertical bars, plus a drop-down to select an Allen or ISO operator.

• Display all operators plus "None" in a drop-down
• If the currently selected operator requires a date/time range, show two vertical bars
• If the currently selected operator requires a single date/time, show one vertical bar
• If the currently selected operator is "None", show no vertical bars
• The user must be able to slide the bar(s) left and right
• Based on the location of the bars and the currently selected operator, display the corresponding cts query.
• On clicking the Search button (#8), include a valid time query if the operator is something other than "None"

Display whether LSQT is enabled

In the UI, indicate whether LSQT is enabled. (Use http://localhost:8002/manage/v2/databases/Documents/temporal/collections/lsqt/properties?collection=myTemporal&format=json.)

Provide a hover-over explanation of what LSQT is.

Have index.html display based on a URL parameter

By default, index.html shows addr.json. Based on #5, the user can type a URL and see that logical document displayed. Add code to drive that functionality from a URL parameter, as "/?collection=foo.json". This will allow the search page (#8) to set up links to logical documents.

Changing textboxes when greater than end time or less than start time

On the search page, when you manually change a date, such as the start date, to a time that is greater than the end date, it will give an error and the bars won't move, but the date that was entered stays the same and doesn't reset to what it was before

Add drop down list with possible properties to change the text in the graph

Rather than having a textbox, create a drop down menu, so that the user only has to choose and click a property rather than think and type and click.

Consolidate to one implementation of parseData()

The parseData function is in public/split.js and public/input_text_load_data.js. Remove one of them.

Make the choosing of properties work for XML documents

Currently, changing the property shown on the graph only works for JSON docs. Make it work for XML also.

Allow selection of a logical document to view

Change the logical document display to let the user specify the URI of a logical document.

• add a text box and a button
• when the button is clicked, send a request to MarkLogic to get the new data
• clear out the old data and display the new
• if there is no data, display a message indicating that the specified URI did not match any data

Add a favicon

https://en.wikipedia.org/wiki/Favicon. Use the standard MarkLogic one (the one used by both www.marklogic.com and developer.marklogic.com appear to be bigger than needed; scale down to 32x32 or 16x16).

Editing a document requires its temporal collection to have use LSQT set

Remove Node.js layer

The current implementation uses a Node.js middle tier. For this visualization, we don't need that. Remove the Node layer and have the client-side JavaScript call the MarkLogic REST API directly.

Update Readmes for install/run of bitemp-explorer

Copied over from bitemp-demo, so has references to that separate repository and those workspaces.

README has a bad link

The README file has a link to http://pubs.marklogic.com:8011/guide/temporal; should be http://docs.marklogic.com/guide/temporal

JavaScript Workspace 2 buffer 10 has a typo

The code in this buffer looks like this:

cts.search(
  cts.orQuery((
  cts.notQuery(cts.collectionQuery("myTemporal")),   // not in temporal collection
  cts.collectionQuery("latest")))                     // in latest collection
);

cts.orQuery's parameter should be in brackets (an array) rather than parentheses.

Specify a valid time search

Dependency: #9

A valid time search will be specified using either one or two horizontal bars, plus a drop-down to select an Allen or ISO operator.

• Display all operators plus "None" in a drop-down
• If the currently selected operator requires a date/time range, show two vertical bars
• If the currently selected operator requires a single date/time, show one vertical bar
• If the currently selected operator is "None", show no vertical bars
• The user must be able to slide the bar(s) left and right
• Based on the location of the bars and the currently selected operator, display the corresponding cts query.
• On clicking the Search button (#8), include a valid time query if the operator is something other than "None"

Make search work for XML documents

Test whether search works with XML documents. If not, fix it so that it does.

Add ticks to graph on index.html

Use d3 to align the ticks
Use d3 to rotate the xaxis ticks instead of css

Add word definitions to search.html and index.html using hover textboxes

Words with Definitions: Bitemporal, URI, Valid Time, System Time, Temporal Collection, Logical Document and Physical Document (when you hover over the names of each one in the bullet list)

Improve layout of search page

• Move reset bars to be closer to graph
• Make graph and cts:Query tabs more apparent
• Move 'Create a Document' button to search page, and delete from (old) index page

Visually indicate the result of clicking on a physical document

In the graph on index.html, provide a visual indication after selecting a physical document. Display the URI of the document above the Edit/View/Delete buttons, with a label that says, "Selected URI:".

Clicking a document should also cause the outline of that document to have thicker lines. (I haven't checked whether d3 allows that, but it probably does.)

Browser is prompting for credentials

The Node.js layer has the MarkLogic credentials, so there shouldn't be any prompt for them.

Problem with deleting a document then viewing/editing.

After a document is deleted and then it is selected to view or edit, the web page gives has a lot of alert windows specifying to select a doc first even though one has been selected. I will look into this.

Add deleting a temporal collection feature

• add a button to search page that allows a user to delete an entire temporal collection

Axes set up for start + 5 or 10 years

setupXAxis() in bitemp_graph.js specifies a range of sysStart through sysStart + 10 years. setupYAxis() specifies valStart through valStart + 5 years. Most likely this is because some of the End years are infinity (year 9999). Fix this so that the range is sysStart through sysEnd, excluding any infinity dates. Fall back on sysStart + 10 years if all sysEnd dates are infinity. Apply the fix to the X and Y axes.

Specify the data to display

On the logical document visualization, each box has a label based on the value of the physical document's JSON property "data". Allow the user to specify what single JSON property or XML element of the physical documents will be displayed in the boxes.

• Add a text input, label ("Document Property"), and "Display" button
• On button click, update the display to show the values of the specified document property.
• If the property/element is not found, show a "No such property/element" message.

Reverse flow of demo

• Make search page default page
• Make (old) index page titled view.html

Create a deploy command

Right now, someone using the bitemp explorer has to use curl to deploy the extensions. Replace this with a node command.

• create a deploy.js (will be run with "node deploy.js")
• have deploy.js properly deploy the extensions in the ext directory
• update the README's instructions
• remove curl.exe (should not be in the repository)

To read a file:

• use Node's fs module; readFileSync would be appropriate here
• use Node's request module to send the extension to MarkLogic. This should be equivalent to the curl command in the README

Modify code to work for different strings representing the timestamps (index page)

Currently, our project is hard-coded to take the specific JSON properties "sysEnd", "valStart" etc.

Modify our code to work (in both XML and JSON) with timestamp property/element names such as "mySystemStart", "startValTime", "myTemporalValidEndTime" etc.

We can do this by making requests to the Management API (ask for the the axes on the current collection; ask for the properties of each axis) or by making an extension to get these in a single request.

Allow for local credentials

Allow people running the app to set up an alternative to env.js, so that they don't need to edit it.

Check all JS buffers for parentheses where brackets are needed

Prevent bars from getting out of order

On the search page, we can drag bars on the graph to represent the system and value times we want to query. Currently it is possible to drag the start-time bar past the end-time bar, and vice versa. This results in an error, because the times are expected to be in order.

Modify the bar dragging such that it does not allow the user to drag a bar past its companion bar.

Remove some functionality from old index page

• Take out ability to change uri
• Take out text area for selecting which document property to view on graph
• Shift edit, view and delete buttons up