pepfar-datim / matches Goto Github PK

Please add ability to upload generated questionnaire response set to a url.

For iteration 1, let's do this as:

• add ability for the user to specify the URL against which POST should be used;
• once posted, mAppr should show the response received from the url (success or error);
• following are being assumed:
  • provided url is public, and no authentication/authorization is required;
  • content-type of the POST request is application/json and receiver can handle it;
  • content (a fhir bundle of questionnaire responses in json format) is send as the body of the request.

Rather than just list rows with unmapped values, list the unmapped values themselves.

E.g.
not
unmapped values: 1, 2, 3

but ideally

女: 1,2
男: 3

On first click to upload mappings from another map, the map being populated gets the new headers,

but on the second/subsequent clicks, the matching headers get removed

currently back end looks at path to get relevant table name e.g. knows to interact with maps table when /api/maps/map100 is called...

may want to avoid this because path could be different based on host environment. Could look into passing table name to callback (if possible) or could

front-end may generally need a configuration to know where to go to (e.g. particularly for redirects)

Probably easiest to start with is to make configuration file with the expected base path. For now, it would just be /

Add upload button (redirect to upload data) and make it disabled until map is valid

• Run basic validation on file client-side (is it a file, are there headers "missing" from either CSV or extra files in the map)
• Pass to back end if file is valid for processing
• Conditionally handle errors returned from validation/processing
• Handle display with management of React state (e.g hide if upload=true)

string values are not being passed to QuestionnaireResponse
(This is because there is no validation check that handles a string type)

Build out UI to accept CSV file on Upload page and then trigger handling (#17)

If implementing "properly" a DELETE request on a questionnaire would remove related maps...but maybe don't want to do this as it could cause problems for users...e.g. so DELETE of questionnaire should maybe fail when there are related maps

Probably want maps dashboard to act as the "home page", so make server/ redirect to server/maps

Make CSV file upload consistent across mAppr and handle special processing in a callback function
Maybe also have initial preprocessing separated out

From #24

Consider how to handle interactions with backend if they fail...maybe warning about being offline/update problems?

Current set up for delete request doesn't alert properly when ID does not exist (because query works, it returns as deleted)

Need to filter out current map, and not use MaterialTable because can't get it to display within frame for this box

clean up file structure (aka decide structure for files :-) ) and summarize for others (maybe in README)

sanitize parameters on backend requests to prevent SQL injection

1. validate table names (for now just validate against list)
2. validate column names (maybe for now just specify valid columns for a given table)
3. pass variables as parameters e.g. pool.query('SELECT * FROM maps WHERE uid=$1', ['abcdef'], function (err, res) { //do something })

FYI @vshioshvili

Add checks for other valueTypes (integer, date, dateTime)

Table with list of maps (landing screen) with navigation options

#5 (comment)

add boxes for handling map add

• retrieve/parse/pass available questionnaires
• need to check name to make sure not in use (requires new api call)
• handling of add through api call
• redirect following 200 response to appropriate edit resource

Validation checks need to be expanded to handle more checks: https://github.com/pepfar-datim/mAppr/blob/master/src/services/validateFile.js

Need to also define those checks more concretely: https://github.com/pepfar-datim/PLM/issues/95

per @vshioshvili's suggestion, look into using MongoDB for backend (and storing json maps there)

Output of the mAppr tool needs to be a bundle type structure that has each questionnaire response as an entry.

Note that reference used for structure definition is a placeholder, and I believe we need to create one - I think it will be same for all mAppr exports, and won't vary based on the questionnaire.

Here is what it looks without any QR, followed by sample with two QR entries.

{
	"resourceType": "Bundle",
	"id": "auto-generated",
	"meta": {
		"profile": [
			"http://datim.org/fhir/StructureDefinition/PLM-QuestionnaireResponse-Bundle"
		]
	},
	"type": "message",
	"timestamp": "1970-01-01T00:00:00.001-00:00",
	"entry": []
}

{
	"resourceType": "Bundle",
	"id": "auto-generated",
	"meta": {
		"profile": [
			"http://datim.org/fhir/StructureDefinition/PLM-QuestionnaireResponse-Bundle"
		]
	},
	"type": "message",
	"timestamp": "1970-01-01T00:00:00.001-00:00",
	"entry": [
		{
			"resource": {
				"resourceType": "QuestionnaireResponse",
				"id": "R7eJnv-2019-10-11T13:09:17.752Z-row0",
				"status": "completed",
				"questionnaire": "http://datim.org/fhir/Questionnaire/PLM-FP-Questionnaire",
				"item": [
					{
						"linkId": "/Patient",
						"text": "Patient ID",
						"item": [
							{
								"linkId": "/Patient/id",
								"text": "Patient ID",
								"answer": {
								}
							},
							{
								"linkId": "/Patient/birthDate",
								"text": "Birth Date",
								"answer": {
									"valueDate": "1980-01-01"
								}
							},
							{
								"linkId": "/Patient/gender",
								"text": "Gender",
								"answer": {
									"valueCoding": {
										"system": "http://hl7.org/fhir/ValueSet/administrative-gender",
										"code": "female"
									}
								}
							}
						]
					},
					{
						"linkId": "/Encounter",
						"text": "Encounter",
						"item": [
							{
								"linkId": "/Encounter/location",
								"text": "Location ID",
								"answer": {
									"valueString": "DATIM OU UID"
								}
							}
						]
					},
					{
						"linkId": "/MedicationStatement",
						"text": "MedicationStatement",
						"item": [
							{
								"linkId": "/MedicationStatement/method",
								"text": "Family Planning Modern Method",
								"answer": [
									{
										"valueCoding": {
											"system": "http://snomed.info/sct",
											"code": "LA27903-6"
										}
									}
								]
							},
							{
								"linkId": "/MedicationStatement/startDate",
								"text": "ART Start Date",
								"answer": {
									"valueDateTime": "2010-01-01"
								}
							}
						]
					}
				]
			}
		},
		{
			"resource": {
				"resourceType": "QuestionnaireResponse",
				"id": "R7eJnv-2019-10-11T13:09:17.752Z-row1",
				"status": "completed",
				"questionnaire": "http://datim.org/fhir/Questionnaire/PLM-FP-Questionnaire",
				"item": [
					{
						"linkId": "/Patient",
						"text": "Patient ID",
						"item": [
							{
								"linkId": "/Patient/id",
								"text": "Patient ID",
								"answer": {
									"valueString": "124"
								}
							},
							{
								"linkId": "/Patient/birthDate",
								"text": "Birth Date",
								"answer": {
									"valueDate": "1981-01-01"
								}
							},
							{
								"linkId": "/Patient/gender",
								"text": "Gender",
								"answer": {
									"valueCoding": {
										"system": "http://hl7.org/fhir/ValueSet/administrative-gender",
										"code": "male"
									}
								}
							}
						]
					},
					{
						"linkId": "/Encounter",
						"text": "Encounter",
						"item": [
							{
								"linkId": "/Encounter/location",
								"text": "Location ID",
								"answer": {
									"valueString": "DATIM OU UID"
								}
							}
						]
					},
					{
						"linkId": "/MedicationStatement",
						"text": "MedicationStatement",
						"item": [
							{
								"linkId": "/MedicationStatement/method",
								"text": "Family Planning Modern Method",
								"answer": [
									{
										"valueCoding": {
											"system": "http://snomed.info/sct",
											"code": "LA27903-6"
										}
									}
								]
							},
							{
								"linkId": "/MedicationStatement/startDate",
								"text": "ART Start Date",
								"answer": {
									"valueDateTime": "2011-01-01"
								}
							}
						]
					}
				]
			}
		}
	]
}

Build
-left bar with map (informational)
-upload csv file
-need request to back end to parse map + csv file (basis is here: https://github.com/pepfar-datim/mAppr/blob/master/csvConversion/QuestionnaireResponse/convert.js)
-handle error display
-if success: download array of questionnaireResponse data

here is redirecting to correct URL but not rendering
https://github.com/pepfar-datim/mAppr/issues/new

Have tried to export with withRouter() but that didn't change things. Also tried changing syntax several ways without success. This was originally working without component references, but removing them also does not work

• add questionnaire url from questionnaire into questionnaire response (and update questionnaires to include url as well)
• change structure of valueCoding responses

• confirm with @lduncan-intrah re: DateTime, ValueCoding.System

page for submitting data

Tried a variety of approaches for columns (

• Material-UI (downside is potential awkwardness with interacting between Drawer and main (also violates Material-UI design principles in that Drawer is supposed to be for navigation)
• plain HTML/CSS divs (okay, but looks a bit inconsistent and requires a lot of syling)
• with (also tried . There are some hassles here with Material-UI being finicky, but looks more consistent with cards on dashboard page, and layout lends itself to resizing in various window sizes

After discussion with @vshioshvili, it was decided that we should let the CSV file be processed (with a warning) if there are extra headers in the CSV file. There will be an error if there are extra headers in the map that don't correspond to the CSV (e.g you map Gender in your map, but use Sex in your CSV file...in this case Gender being missing from CSV is an error, but Sex in the CSV file would be a warning)

Requires some backend redo because of logic in when the file should be handled

Get response of PayloadTooLargeError: request entity too large with larger csv uploads (testing at 10,000 lines)

Write api call to run processing of csv file (to be used by #17)

Need to modify this code to run server side and to return useful response.

Response like
{statusCode: 200, message: 'Success', errors: [], questionnaireResponses: [...]

Endpoint of maps/:id/upload or maps/:id?mode=process

@tomzemp having a UUID might be good. But it can be left as the literal text if UUID is not possible.

Originally posted by @vshioshvili in #41 (comment)

To define what we want for UUID before implementing

Currently front end endpoint maps/:id looks "correct" but with empty map/questionnaire

Should make this so that it delivers a message that says there is no ID

Make errors more readable (easiest to change how errors are returned on the back end, so that they are like {"key":{"errorType":[1,2,3]}

• "Row" number is being duplicated.
• the number should be updated to make more sense to user (currently it returns 4 for the 6th row in the file because header row is ignored and then counting begins at 0)
• more detail should be provided on the specific value that is a problem

create basic header bar component to be used throughout app
-mAppr name (+ placeholder for logo)
-organization name (from configuration file)
-icons for "menu" (list of maps), configuration, help

When uploading a file, sometimes a space is added at the end of the last column, which makes the header checks think, for example, that date is not the same as date .

From #24

Allow users to edit map's name from within edit page

Design approach to let users use existing map as the basis for new map.

Front-end approaches are outlined here: https://whimsical.com/QopncHDGwNVsjWoh7iJi1W

Ultimately, Option 1 and Option 2 are desired, but Option 1 is priority.

This can probably run client-side, in which case "flatQuestionnaire" can be leveraged, but that may need to be revisited.

When initially creating a map, it appears that if you create a header but don't associate it with a map, when you navigate away/back, the code to process the questionnaire fails because it expects there to be associations in the map between the header and the questionnaire. Need to add handling in validateMap service

Need to check that everything is mapped when going to upload page before allowing actual upload

From edit screen, it would be better for performance to have the check running as changes are made, but could also be useful as a "final" check when background check alerts that map is completed

Hi @tomzemp,
I like the code! Looks well organized and makes sense when reading!
However, since I was asked to provide feedback I will put a few comments here :)

cc @benguaraldi

Make README

• brief summary of project
• instructions for setting up and running locally
• summarize file structure (after #10)

Return 500 status code and direct to error page when invalid resource is requested.

When you reload a file in upload mode, the missing headers get messed up, e.g. loading this file even though these headers are present in the file:

From #24

Check for completeness/validity as map gets changed
-are all headers/questions mapped
-no "1:many" mappings

• values mapped (if relevant)

create file for configuration that contains name and decide where in backend it should be

"name": "Tanzania Ministry of Health"
}```

Will need to be loaded from Main component

Can eventually contain things like language, etc,. 

Need to add callback to delete icon to actually remove map

Work on building page for this:

-Convert map properties to Chip "Headers" in side bar (with handlers for add/delete)
-In main space, have parsed questionnaire with links to map's headers/properties
-For initial stage, probably have question use a select of the available headers/properties from the map
-api calls to back-end to update map as user makes changes

Need to summarize the map for informational purposes
This will be generated as part of a check on the map's validity/completeness

From #24

Work on handling of value maps...

for ease of use at the moment (for proof of concept), probably want to put code system definition within questionnaires (e.g. don't go fetch from external source as we define requirements/vision for questionnaire storage)

Might be useful to prepopulate the value map with the definitions expected by the code system definition and then allow users to update