Start generating more complex reports using data from the database

Create a report template which will be filled with data during report generation

Having some troubles signing and building the frontend app

If there are no emails to send - no schedules or none in the time range, stop the emailing process early

Need to add a couple more variables to the app config: specifically email server/port

Some variables in grafana can be custom, so you have a query which is like:

select * from item_line where ven_category in (${VEN})

Where VEN is a variable with the options defined as V,E,N. Then when viewing a dashboard, there is a drop down to select which character. This character then transforms the query to be:

select * from item_line where ven_category in ('V')

There can be multi-select or non-multi select custom variables.

The Variable definition object you get from grafana looks roughly like:

{
    query: 'V,E,N'
    options: [{selected: true, text: "V", value: "V"}, .. {selected: false, text: "N", value: "N"}]
    name: "VEN"
    label: "VEN"
    multi: true
    includeAll: false // this will give an "ALL" option - setting "V", "E", "N" to true.
}

Grafana takes this definition and creates a drop down which then also enables transforms on the query, according to the format provided. I.e.: ${VEN:sqlstring} transforms it to be "V", "E", "N" where as ${VEN} will transform it to be: V,E,N

Solution:

For each panel, iterate through the variables for the panels dashboard, and determine if the variable is used within the sql. If it is, generate and display the drop down to choose the variable to use. Save this in the msupply sqlite and use it for generating reports.

Saving: Saving it as a JSON stringified object 😢 in ReportContent (A 'panel' record) - because each panel can have multiple variables, and multiple values for each variable:

{
  VEN: ["V", "E"]
  status: ["fn"]
}

Even if they're not multi-selectable, store in an array.

Limitations:

• There are other variable types other than custom. Also other formatting types. JUST going to implement custom, sqlstring formatted.
• The reporter won't be able to keep in sync well enough - i.e. if you set up variables for a panel, then go edit those variables.. 🤷 But this is pretty same same, i.e. if you set up panels and then go delete panels.. 🤷

1. User filter does not work properly. It shows other names not including letters from the input box.
2. Group should not be included in this list.

3. Only user names with active = true in the mSupply user table should show in the list. Currently, there are many duplications in the list if there are the same user names in mSupply.

C:\\Program Files\\mSupply Dashboard\\bin\\data\\template.xlsx: The system cannot find the path specified.

The revert of homePath, _ := filepath.Abs(".") might not have been done properly 🙄

Issue

Number formats for cells in different columns need to be customizable, depending on the report.

Proposed solution

None

Hook up a simple emailing implementation to the scheduler send an email with a simple attachment

In grafana v8.2, App Configuration is not shown.

A request to improve the formatting of the spreadsheet

• column widths should be relevant to content
• dates should be formatted as dates

other suggestions? @joshxg @craigdrown

When user open the report group page and the report schedule page, it shows permission error and does not show available panels if user's role is not admin.

To access to the path api/datasources/XXXX seems denied to viewer.

However, admin user can access the same page properly.

I assume that it's because Grafana organization roles restrict editors and viewers to access to data sources.

If user can not access to the report schedule page because of permissions, it would be better restrict menu icon visibility itself?
https://github.com/openmsupply/msupply-dashboard-app/blob/3be59cd164d659ef04dddb78a80ce2b07d4fe07f/frontend/src/plugin.json#L31-L48

Additional info

Also one thing works different from above is report group page.
Even with viewer user, the list of users shows and we can save it.

Version

Grafana version 7.3.0

scheduler date/time fields are not in the init queries so the db meltsdown when creating a new one

When populating data in the sheet, determine if a value is numeric. If it is, right align it.

Perhaps have this as a configuration option too?

Create reports to all of those registered for the schedule

Currently when a variable is set to refresh as anything but "never", the "options" field is not populated. This field holds an array of the options available in the drop down.

Would need to query for and map the options on loading the panels.

Some thing something Ujwal and backup SQLite

When trying to build the backend code on Windows, getting the following error:

$ go run mage.go 
# github.com/grafana/grafana-plugin-sdk-go/build
..\..\..\..\go\pkg\mod\github.com\grafana\[email protected]\build\common.go:91:10: undefined: syscall.Kill
Error: error compiling magefiles
exit status 1

OS: Microsoft Windows 10 Pro.
Version: 10.0.18362 Build 18362.
Architecture: x86_64.

Downloaded excel file has no data when grafana is version 8

Details

After the fix to #71, when we try to add configuration to Plugin, the validation fails.

Should change the implementation to send, for example, everyday at 7am.

Current implementation is set to daily, weekly, 30 days (monthly) etc is just the interval being the number of seconds in that duration. Then the initial "nextReportTime" is set to 0. Every 10 mins the scheduler checks if the current time is greater than any nextReportTime, if there is, sends that scheduled report and increments the nextReportTime by the unix timestamp in interval.

Need to : Change UI to select an interval: Daily, Weekly, Monthly etc and also a time of day: 7am, 8am etc. and possibly, if it's weekly: What day of the week, if its monthly, what day of the month (Or potentially also "last day of month", "first of the month" etc??)

Not sure exactly what to do as there is quite a lot there. I suppose you would treat them all seperately, and have an initial step: Daily, Weekly, Monthly, then if it's daily: Time of day, otherwise weekly: Number of weeks (to handle fortnightly), day of week, then time of day. Then for monthly, number of months, day of week, time of day?

If it was done like this, I suppose you could map this into a cron scheduler string and then just start a cron job for each schedule.. but I'm not convinced this is a good idea, i'd rather save an exact time of the next report in a persistent db and check against that and increment, I think it seems safer! But determining a way to do that with these changes is more difficult, so I dunno.

enter the grafana url through the config

In the email report, Lookback period does not work even though there are records on the actual dashboard panel.

Add the option to export a table directly to excel and then download the result.
To be used with the msupply-table plugin

Details

t=2022-03-08T23:52:22+1300 lvl=eror msg="Could not start plugin" logger=plugin.manager pluginId=msupplyfoundation-datasource err="Unrecognized remote plugin message: 2022/03/08 23:52:22 notifying bugsnag: Binary was compiled with 'CGO_ENABLED=0', go-sqlite3 requires cgo to work. This is a stub\n\nThis usually means that the plugin is either invalid or simply needs to be recompiled to support the latest protocol."

In mac grafana tries to use the ARM version of the executable which also breaks saying could not find file.

I did recompiled the app in my m1 mac system but it was broken before I tinkered. I forgot to take the error log entry though.

Definitely getting this error because Grafana for some reasons cannot run the backend executable.

Add a feature to download the default template from some API so we don't have to change the default on every instance

Clicking the SEND TEST button always send the first schedule in the list and not the current schedule.

The front end is sending the correct schedule ID and this is parsed correctly in testEmail - but the schedule instantiated is always the first one.

When deleting a report schedule, the prompt asks to confirm deletion of a report group

After having chosen a ReportGroup in the EditReportScheduleForm component, the layout goes all hunchy funchy.

Need to add some more logging for in correct reports being generated as 0kb files

Configured an instance where the user password has a % in the password and the following shows in the logs:

lvl=eror msg="GetEmails: http.Post: parse \"https://[redacted]/api/tsdb/query\": invalid URL escape \"%[some other chars]\"" logger=plugins.backend pluginId=msupplyfoundation-datasource

lvl=eror msg="ReportEmailer.createReport: emailsFromUsers: parse \"https://[redacted]/api/tsdb/query\": invalid URL escape \"%[some other chars]\"" logger=plugins.backend pluginId=msupplyfoundation-datasource

in the UI there is "REporting group" and also "Report Group" - should be "Report Group" consistently

Remove the storeID field from ReportContent and any left over traces to it. Unneeded!

Need these docs:

• supported variables
• supported macros
• template requirements
• examples
• email requirements (smtp)
• sqlite schema

Currently when the plugin boots it tries to send emails, and the scheduler isn't properly running.

Instead: Try send emails instantly, but also start the scheduler

Dunno why, but this line here https://github.com/openmsupply/msupply-dashboard-app/blob/70d273b97dd006e17f3cfd91b5dc08dd1d00ccc2/frontend/src/api.ts#L183

Should check for "table" or "table-old" .. 🙄

Have the scheduler attach the simple generated report to the email sent

Show some sort of feedback to users when there are no users or panels to select:

"No users found!"

"No panels found!"

Create the correct reports during a scheduled report generation

need to add this app to the msupply-grafana fork installer?

Currently, it is a little bit difficult to find users if there are many users on the list.
So this change would be helpful.

• To show the list in alphabetical order. Currently, it seems not to line up in order.
• To show only users who have an email address in mSupply.
  

the link is still going to /a/msupply/.. which now invalid

Start creating a simple excel report

In a query there are sometimes variables.

i.e. [store] or time macros $timeFilter(..

For variables: These are replaced by sub queries within the dashboard definition.

For example:

Select store.name as \"Store\",
 item_department.department as \"Program\",
 item_name as \"Item Name\",
 unit.units as \"Units\",
 pack_size as \"Pack Size\", 
 available as \"Packs Available\",
 pack_size*available as \"units left\",
 batch as \"Batch No.\",
 to_char(expiry_date, 'YYYY/MM/DD') as \"Expiry Date\",
 location.code as \"Location\"
 from item_line join item on item.id = item_line.item_ID
 join store on store.id = item_line.store_ID
 join unit on item.unit_id = unit.id
 join item_department on item.department_ID = item_department.ID
 join location on location.id = item_line.location_ID
 where expiry_date > now() + interval '($monthsFrom) months'
 and expiry_date < now() + interval '($monthsTo) months'
 and available > 0
 and store.name in ([[store]]) 
 and item.item_name in ([[item]])
 and item_department.department in ([[department]])
 order by 1, 2"

The above query has variables and macros.

Variables are for example ([[store]]). This needs to be replaced with the sub query which is in the dashboard definition where the panel lives:

The dashboard has a templating field which has a list of variable objects like:

{
	"0": {
		"allValue": null,
		"current": {
			"selected": true,
			"text": "Caldwell",
			"value": [
				"Caldwell"
			]
		},
		"datasource": "PostgreSQL",
		"definition": "select distinct name from store where disabled=false and store_mode in ('store', 'dispensary');",
		"hide": 0,
		"includeAll": true,
		"label": "Store",
		"multi": true,
		"name": "store",
		"options": [],
		"query": "select distinct name from store where disabled=false and store_mode in ('store', 'dispensary');",
		"refresh": 1,
		"regex": "",
		"skipUrlSync": false,
		"sort": 0,
		"tagValuesQuery": "",
		"tags": [],
		"tagsQuery": "",
		"type": "query",
		"useTags": false
	}
}

So, for variables: Replace the ([[store]]) with the query field.

For macros as above there is interval '($monthsFrom) months' and interval '($monthsTo) months'

In the request body of the POST for executing the query, there are two fields from and to which seem to be used for these macros.

Need to add a bit more defensive code to give more insight into errors as well as stopping the app completely crashing when the db can't be opened

This error happens during setting up newly.
The latest version of codes in repo was used to build.

Also, I tried to submit without any data, alert is only at email textbox

Add a feature to upload a template

Whenever the grafana server changes the postgres datasource ID increments.

I.e. it is set to 10 - reset server, goes to 11.

Ugh.

Could do something like using the default datasource, probably?

Create a simple scheduler and outlining structure of a scheduler to generate reports with a static interval