icstechsales / dql-explorer

This project is an exploration of data in Domino databases leveraging Domino Query Language.

Quickstart

If you are here to try out DQL Explorer, learn about DQL and just want to get started... then:

Preparation:

1. Download the dqlexplorer.nsf
2. Deploy it to a Domino 10.0.1 Server (into the data directory root, not a sub directory for now)
3. Sign the database with your signer ID, you can use the server id for instance.
4. Set an appropiate ACL - as the database allows to create and store DQL queries, you want to define some users with Author access.
5. Access the database with a browser: https://<domino_server_url>/dqlexplorer.nsf/index.html
6. DQL Explorer is just a gateway to access other databases data. So you now need to have DQL Explorer list some databases you want to use.
7. As of now, DQL requires that you issue these commands on the Domino console:

load updall -e

load updall -d

Now you are ready to create your first DQL Query. As an example we want to use names.nsf

1. Add names.nsf to the listed databases
2. load updall names.nsf -e
3. load updall names.nsf -d
4. Define your query
5. Run your query

Watch the demos

Luis Guirigay and Andrew Manby introduce DQL Explorer at Think 2019: https://youtu.be/OMjSND5cPsE

Scott Good provides a demo that includes some configurations aspects not shown in the stage demo: https://youtu.be/Cfw_6Wvk8c8

Release Notes

Go to the Releases tab for the latest version and update notes.

Requirements

The Domino Query Language (DQL) uses design data extracted from view notes. This information is stored in a special database called a design catalog and named GQFDsgn.cat. You must create a design catalog before a Node.js developer any attempts to use DQL, using DQL Explorer, Node.JS, LotusScript, Java, server console, etc... to query any database. Now that I slightly corrected the documentation, feel free to follow the remainder of the steps as outlined in the DQL documentation to create the design catalog and add your target databases to it.

Configure

Note: Try this on a development server.

• Clone this repository.

• Using the dqlexplorer.ntf template create a new database called dqlexplorer.nsf (currently it needs to be deployed in the root of your data directory).

• If you previously deployed dqlexplorer.nsf to your server, you should use the dqlexplorer.ntf to replace (or refresh) the design on your dqlexplorer.nsf.

• The database runs a number of agents, you will need to sign the database so the agents can run on your server.

• The Web application relies on Domino Access Services (DAS) being enabled on the dqlexplorer.nsf (default), but that means there is a need to be enable DAS on your server. The following example show how to enable the Data service on a server.

  • If internet site documents are enabled, do the following:

    a. Create or edit an internet site document.

    b. On the Configuration tab, under Domino Access Services, edit the Enabled services field. Type 'Data' to enable the data service.

    c. Save the document.

  • If internet site documents are disabled, do the following:

    a. Edit the server document.

    b. On the Internet Protocols tab, click Domino Web Engine.

    c. Under Domino Access Services, select 'Data' for Enabled Services.

    d. Save the document.

• Configure the ACL as you would any Domino database. The roles DQLXAuthors and DQLXReaders will give those that belong to these roles either reader or author access to all queries stored in the database, only give it to administrators or department admins. All users with regular Author access to the database can create/share queries without needing these roles.

• Edit the lookup document in the dqlexplorer.nsf and setup directories or databases that you want to make available via the DQL Explorer (see Scott's demo of that here: https://youtu.be/Cfw_6Wvk8c8?t=440).

• For development, update the package.json in this project to point the proxy (domino_server_url) and homepage (domino_server_url/dqlexplorer.nsf) to your Domino server's url, use http (not https).

Available Scripts

In the project directory, you can run:

npm install

This process may run for less than 2 minutes. It installs all the npm modules required for this project. You only need to run it once or if you add/update the modules defined in the package.json.

npm start

Runs the app in the development mode.

Open http://localhost:3000 to view it in the browser.

The page will reload if you make edits.

You will also see any lint errors in the console.

npm run build

Builds the app for production to the build folder.

It correctly bundles React in production mode and optimizes the build for the best performance.

The build is minified and the filenames include the hashes.

Your updated version of the app is ready to be deployed to the dqlexplorer.nsf domino database under the Resources...Files as shown below, make sure to include the appropriate path for css and js files when the files are added by changing the filenames after import to include the path, i.e. static/js/... or static/css/...

Once deployed the application is accessbile from any web browser accesing https://<domino_server_url>/dqlexplorer.nsf/index.html

dql-explorer's Issues

Wrong number of search results is displayed when no document have been found

If the query returns no results, the number of matching documents should be 0 instead of a random(?) number ( always 143 in my environment )

Server console shows

[018518:000012-00007F2861CA7700] 03/09/2019 06:59:29 AM Agent message: timeType = '1'
[018518:000012-00007F2861CA7700] 03/09/2019 06:59:29 AM Agent 'runDQLQuery' error: Domino Query execution error: Entry not found in index - validation error Error validating field name (field not defined) - [timeType] (Call hint: NIFFindByKeyExtended2, Core call #0) ******************

'DefaultButton' is defined but never used no-unused-vars

Compiled with warnings.

./src/components/querybuilder/ShareDialog.js
Line 3: 'DefaultButton' is defined but never used no-unused-vars

Search for the keywords to learn more about each warning.
To ignore, add // eslint-disable-next-line to the line before.

TODO: remove DefaultButton from line 3 ; it has been used in line 73, but line 73 is commented out.

NSF from OpenNTF: Agent did not complete within configured time limit

I used the pre-built NSF from the OpenNTF download page. When I open the start page it runs to death because the getdatabases agent runs infinite. Interesting: this message occurs already after 10 seconds though the agent execution time on my server is set to 10 minutes.

HTTP Web Server: Couldn't find design note - runDQLExplain2

Agent is missing in dqlexplorer.nsf

Question: What about security of targeted databases?

If I'm understanding correctly, the ACLs setup for the DQL database (say Bob as Author) would then use "Bob" in the query to the targeted database. So, the results given are what "Bob" would be able to see in that target database.

If I make this query available to other users, will they also see what "Bob" would get from the query or what the end-user himself would be able to see from the target database.

This to me has been the fundamental flaw in keeping DQL from being truly useable at this stage in the game. I have not played with it enough to determine how I'd want to manage users of the Node app against their ACL rights in the targeted Domino DBs.

Perhaps I've missed something in more recent updates to DQL and it's been addressed?

Remove hard coded in who you can share query with ..

When you create a query, you have the ability to share the query with other users. Currently the names that appear in the drop down for sharing is hard coded to 4 names:

This enhancement would be to add an agent to the dqlexplorer.nsf (like was done for the getdatabases agent) or a page (like was done for the getuser page) , that will get a list of all individuals in the ACL for dqlexplorer.nsf and return a json object to the app in the form of

  {
    "key": 1,
    "imageInitials": "AM",
    "text": "Andrew Manby",
    "secondaryText": "Director, Product Management",
    "tertiaryText": "CN=Andrew Manby/O=MyOrg"
  },
  {
    "key": 2,
    "imageInitial"s: "DP",
    "text": "Dimitri Prosper",
    "secondaryText": "Developer",
    "tertiaryText": "CN=Dimitri Prosper/O=MyOrg"
  },
  {
    "key": 3,
    "imageInitials": "LG",
    "text": "Luis Guirigay",
    "secondaryText": "Technical Leader",
    "tertiaryText": "CN=Luis Guirigay/O=MyOrg"
  },
  {
    "key": 4,
    "imageInitials": "SG",
    "text": "Scott Good",
    "secondaryText": "Designer",
    "tertiaryText": "CN=Scott Good/O=MyOrg"
  }

This object can then be used in the src/components/querybuilder/PeoplePicker.js component to make the names available in the above drop down.

Query by Form : No fields in dropdown

When I try to build a query using a Form, I'm unable to select fields. Neither in the Query Builder section, nor in the Columns/Fields to display ... section.
When I look at the network traffic in the browser (Firefox), the agent getFieldNamesFromForm is called with the appropriate parameters and I get a valid JSON containing all fields.

[{"name": "$fldSecAuthors","displayName":"$fldSecAuthors","type": "name"},{"name": "StoreId","displayName":"StoreId","type": "text"},{"name": "StorePhone","displayName":"StorePhone","type": "text"},{"name": "StoreCountry","displayName":"StoreCountry","type": "text"},{"name": "StoreRegion","displayName":"StoreRegion","type": "text"},{"name": "StoreCity","displayName":"StoreCity","type": "text"},{"name": "StoreZip","displayName":"StoreZip","type": "text"},{"name": "StoreStreet","displayName":"StoreStreet","type": "text"},{"name": "StoreName","displayName":"StoreName","type": "text"},{"name": "StoreBanner","displayName":"StoreBanner","type": "text"}]

Thus the agent is called and returns values, but the comboboxes are not filled with the data.
I don't see any errors in the browser console.

I'm running the database on a Domino V11 beta 1 server, but I assume this is not a server issue, since the agent returns valid data.

Please advise.

Error parsing timedate string

Resulting query contains wrong timedate string

Domino Query execution error: Release 10.0.1|November 29, 2018 - error during planning and tree generation Error parsing timedate string - 2019-21-13 (ISO massaged string = 20192113)Error filling node for timeTimeFrom = '2019-21-13' (Call hint: OSLocalAllc, Core call #0)******************

dqlexplorer.ntf is encrypted

cannot open .ntf

Access the database with a browser no result

Hi!
Canot Access the database with a browser. In browser i get a blank page without any views...

Timedate values not enter-able manually.

The date selection UI must be used. But dqlexplorer should accept typed-in values

Query by view: Columns remain empty

Do you both mind to join the OpenNTF Slack?

The conversation would be a lot faster and better here ;-) Don't be shy!
https://slackin.openntf.org/

Add operator only if condition is added

I created a plain simple query whre I did not add any condition. This results in

[018518:000012-00007F2861CA7700] 03/09/2019 06:44:53 AM Agent 'runDQLQuery' error: Domino Query execution error: Query is not understandable - syntax error - MUST have at least one operator (Call hint: OSLocalAllc, Core call #0) ^ ******************

which is ok, because the query submitted to the server is

So pls. add the operator only if a condition is configured.

Please document prerequisites for using DQL

Please document how to update the required design catalog in order for DQL to work - or at least state that this is required in order for DQL to work. Thanks.

UI crashes while entering the value

UI shows following error while entering the value in a text field in "Query Builder" Section

Error

2.62e4ba75.chunk.js:1 TypeError: Cannot read property 'type' of undefined
at a.onTermChange (main.23744580.chunk.js:1)
at Object.onChange (main.23744580.chunk.js:1)
at t. (2.62e4ba75.chunk.js:1)
at ia (2.62e4ba75.chunk.js:1)
at ra (2.62e4ba75.chunk.js:1)
at Ya (2.62e4ba75.chunk.js:1)
at Va (2.62e4ba75.chunk.js:1)
at 2.62e4ba75.chunk.js:1
at Object.t.unstable_runWithPriority (2.62e4ba75.chunk.js:1)
at As (2.62e4ba75.chunk.js:1)

Query does not take selected FORM into account

I have created a database with 2 forms ( form1 and form2 ) . Both forms have an item with name 'subject'
I also have created douments for each form.

I have configured a query to get all documents from the database that are of type form1 AND have the word form in 'subject'

DqlExplorer builds the correct query in the Development Options sections, but the submitted qury is missing the Form="form1" part.

The result is 2 documents, but this also includes the document created with form2

Query by view / column creates a query by form / field

Building a query by view and column name, then activating Developer Options: the resulting query has the wrong format.
Example:
configured node-demo.nsf (demo db from appdevpack) in dql Explorer, also made sure the db design is indexed in GQFDsgn.cat.
Using query builder I select "Views" >> "AllContactsByState" >> "State" >> "in" >> "CA" and "FL".
The resulting query should be "'AllContactsByState'.State in ('CA', 'FL')".
However it is listed as "State in ('CA', 'FL').
Running "Explain" from the Developer Options has the following result:

Query Processed:
[State in ('CA','FL')]

0. IN    (childct 2) (totals when complete:) Prep 0.0 msecs, Exec 8.401 msecs, ScannedDocs 986, Entries 0, FoundDocs 70
1.State = 'CA' NSF document search estimated cost = 100  
Prep 1.102 msecs, Exec 8.395 msecs, ScannedDocs 986, Entries 0, FoundDocs 50
1.State = 'FL' NSF document search estimated cost = 100  
Prep 0.691 msecs, Exec 0.0 msecs, ScannedDocs 936, Entries 0, FoundDocs 20

Also see screenshot

JSON contains useless values

It happened to me that the JSON for the fieldnames of a form were empty. You could check that and then omit this element in the generated JSON

Readme.md misleading

There's two stories told in readme.md:

1. For the DQL Newbie to try DQL via DQL explorer. For this we need a crisp intro that simply focuses on getting the nsf on the disk and prereqs set up
2. For building the NTF from scratch

Currently, these stories are mixed - basically it starts with story 1, then deep dives in story 2 and the last line finally is the missing piece of story 1.

Let's make this help wanted and i'm ready to work on this

Please provide the NSF as ODP

This would enable us to contribute to the NSF as well ;-)
I'd then switch to using Java for the Agents or even better: use a custom service bean to create a REST service using the REST service control from the Extlib.

What about using Webcontent instead of file resources?

The build process then also could copy all the stuff in Webcontent folder of the ODP. Dealing with file resources in an NSF and renaming everything is quite a hassle...

Problem with query by form if form's alias is different from it's 'real' name

There's a problem building a query by form if the form's name is different from its alias, as in 'Main form | fmMain':
Drop-down to select forms shows the 'real' form names, but obviously the field selection is looking for the form definition by the alias name: selecting "Main form" from the list leaves the field selection empty.

This can be solved by changing form name / form alias so that they are equal as in 'fmMain | fmMain' (or eliminate alias name altogether).
Then I have the proper field selection

BTW: difference between name and alias of course is case-sensitive: if the form's name is 'Main | main' no fields can be selected; name has to be'main | main'.
Environment info:
node.js 10.15.3 is running locally on a Windows 10 notebook;
database to be queried is hosted by a Domino 10.0.1 server running on CentOS.
DQLExplorer version is 0.2.7

Also see here: https://www.openntf.org/main.nsf/project.xsp?r=project/DQL%20Explorer/discussions/BD568D0F285460CC852583C20041611A

Type error 's is undefinded' if databases are located deeper in File-directory

If the folder or the database is locate deeper in the file directory than you get a s is undefined error.
It is because the json-File has a wrong escape character like :"...folder1/folder2\database.nsf
The agent 'getDatabasesFromServer' only replace the first backslash with the function "replaceSubstring". I put a new function in it:

Public Function ReplaceSubstring3(SourceS As String) As String
	Dim SearchS As String, ReplaceS As String	
	SearchS="\"
	ReplaceS="/"	
	While InStr(SourceS, SearchS) > 0		
		SourceS = Left$(SourceS, InStr(SourceS, SearchS) -1) + ReplaceS + _
		Right$(SourceS, Len(SourceS) -InStr(SourceS, SearchS) - Len(SearchS) + 1)		
	Wend
	ReplaceSubstring3 = SourceS	
End Function

If you use this function in line 66 the json-file is ok und will work.