avniproject / integration-service

Service for integration Avni with other systems

Module order

1. Bahmni
2. Goonj
3. Amrit
4. Power
5. Lahi

integration-service's Issues

[LAHI] Pull Student registration data from Gliffic to Avni

Use case:
Once a student is sent a message over whatsapp from LAHI to express and confirm their interest in program(s), student will share their basic information such as Name, Gender, Age, School, etc. in response.
When the details are available in Gliffic, Avni needs to bring those details and store for student registration.

Subject (Beneficiary) Process

• Convert response from Avni to amrit request for entity(Beneficiary) Only for all fields that are in common for both systems
• Sync create/update of Beneficiaries from Avni to Amrit
  a. Check for amrit id for avni id,
  b. if api doesnot throw error sync, if error, add an error record
• After create/update, check if entity was created by invoking Amrit ID generation, if error, add as error_record
• Sync error records from Avni to Amrit after attempting Amrit ID generation, for all those that get success response

Goonj Integration with Avni

Synchronise Activity from Avni to SF

• The Activity encounter should be Synced to SF using corresponding Demand Or Distribution with appropriate - voiding status, last modified date time, and sourceId. For the remaining fields use the mapping metadata approach.
• Save error record for missing mapping for a field, its coded answers, or missing Activity in Avni.
• The sync should resume from the previous point if an uncaught exception occurs.
• Make it ready for QA
• Deploy to Staging

Make AvniHttpClient work for different implementations using implementation specific scope, session or threadLocal context

1. Extract 4 fields to separate bean - AvniImplementationClientConfig
2. Make them scope, session or threadLocal context based
3. Ensure the scheduled job starts with implementation specific scope, session or threadLocal context
4. Init the AvniImplementationClientConfig bean for that implementation before starting scheduled job within the scope, session or threadLocal context
5. If using ThreadLocal, config a separate single-thread threadPool for each implementation

Deploy and define systemctl service for running the integrator service

Some helpful scripts can be found here for defining the systemctl service in CentOS. We can use these or do it from scratch - depending on how we want to do it.

https://github.com/avniproject/avni-integrator-host

Acceptance Criteria

1. Integrator service is running in production and/or staging environment.
2. There is an automated way to deploying the new build to the same server

Metadata migration

Acceptance criteria

There are mechanisms to automate pulling of metadata for an organisation on integration-service. and pushing them to another organisation/implementation. This will be used to migrate changes from one organisation to another

Tech notes

• Start using uuids

Improve monitoring capability of Integration service

To improve monitoring capability of Integration service:

• 0. write tests to create a suite to ensure proper classification of errors
• 1. Segregate and stop only specific task group
• 2. Let the task stop if there are uncaught errors
• 3. Build logic for processing error and segregating between Business errors and other errors
• 4. Store Business ErrorType with matching Exception msg, to determine if an exception is not a business error
• 5. Create separate bugsnag
• 6. Create internal group for avni-int
• 7. Store error message in ErrorRecordLog
• 8. Show error message in int-web-app, on clicking Edit button with proper fields keys
• 9. Use metabase for alerts for new errors occurred since lastCutOffTime, schedule periodic trigger
• 10. Show correct date time string in int-web-app error records screen
• 11. Filter on integration system

Synchronise Dispatch from SF to Avni

• The dispatch encounter should be created in Avni under corresponding demand with - voiding status, last modified date time, and external id. For the remaining fields use the mapping metadata approach.
• Reuse the two scheduled jobs for demand to run dispatch sync also
• Dispatch response now has "TypeOfMaterial"  :"Contributed, Kit, Purchase" specified to ease Avni Dispatch entity creation, use it to complete Dispatch sync from SF to Avni
• Store DispatchStatusId as External ID instead of DispatchStatusName
• Map missing fields in Dispatch: "DispatchState": "Assam", "DispatchDistrict": "Bongaigaon", "DispatchDate": "08/05/2022",
• DispatchLineItems id field “DispatchLineItemId” and following non-mandatory fields
                  “DispatchLineItemId”: “adli1234”,
  "PurchaseItemCategory": null,
  "OtherKitDetails": null,
    "KitName": null,
  "MaterialName": null,
  "ItemName": null,
  "ItemCategory": null,
                  "ContributedItem": null,
  Remove incompatible fields :
  "Contributed (voided207321)": 12,
          "Purchase/High Value": 12,
  "Non-Kit (voided207320)": 12
• Implement support for  handling “DeletedObjects": { "DeletedDispatchStatusLineItems": [], "DeletedDispatchStatuses": []}
• Save error record for missing mapping for a field, its coded answers, or missing demand in Avni.
• The sync should resume from the previous point if an uncaught exception occurs.
• Make it ready for QA
• Deploy to Staging

Acceptance criteria

All the dispatches should be visible in the data entry app, under their demand, in Avni for any user who has the privilege. Note that only the fields mapped will be visible. All the field and their coded answers mapping will be handled via a separate implementation story.

GET Demand

Post as new subject

Prevent voided Goonj encounters from being upserted into Goonj SF

The purpose of this task is to, prevent voided Goonj encounters from being upserted into Goonj SF. We need to filter out encounters (DispatchReceipt, Activity and Distribution) that were deleted in Salesforce and had the Avni "/delete" external API invoked for them. We should not sync them back to Goonj, due to change in their "isVoided" and "lastModifiedDateTime" fields.

Test this change only in Goonj Sandbox (GoonjUAT on prod).

Steps to reproduce issue:

1. Login into Goonj salesforce sandbox
2. Selecte any demand which has anyone of the following: DispatchReceipt, Activity and Distribution
3. Delete the entity in Salesforce (This would internally invoke the Avni "/delete" external API for them)
4. Check that the encounter got deleted(isVoided=true) in Avni.
5. Ensure that it is not synced back to Goonj, due to change in their "isVoided" and "lastModifiedDateTime" fields.

Handle edit of DispatchReceipt

Edit of DispatchReceipt right now can be handled cleanly only when the existing dispatchReceipt is deleted and a new one added.

When getting an edit for DispatchReceipt, then delete the entry from SalesForce and then add a new one.

Notes

• Delete API details have been provided

POST Distribution

Minor changes to attributes.

Change source from encounter to subject of type distribution

Synchronize Distribution from Avni to SF

• The Distribution encounter should be Synced to SF using corresponding DispatchStatus with appropriate RecordType and DistributionLines - voiding status, last modified date time, and sourceId. For the remaining fields use the mapping metadata approach.
• Save error record for missing mapping for a field, its coded answers, or missing Distribution in Avni.
• The sync should resume from the previous point if an uncaught exception occurs.
• Make it ready for QA
• Deploy to Staging

Minor enhancements to Beneficiary processing flow

• check if locationHeirarchy has to be corrected and forms should be made to work

• Modify Sakhi to include Panchayat info:
  addressLevel type should be added //done
  data uploaded //done
  Panchayat added before village //Done
  externalID for all types //Pending

• Add mapping for following:
  literacyStatus //duplicate of educationName
  stateID
  districtID
  blockID
  districtBranchID
  vanID

• Create tasks for Amrit, And invoke Beneficiary and BeneficiaryScan as separate tasks..

• Work on Amrit Error Records in AmritErrorJob

• Process error records also as part of AmritMainJob

• Make use of GetAmritID API call to get Amrit Beneficiary ID and use the same in update of Beneficiary and Create/update of all other entities in field "beneficiaryRegID"

• If you face "Beneficiary registration not completed in AMRIT", during BeneficiaryScan, stop at that step.
  Move to next entity sync. Before every entityType sync, perform getAmritId operation of beneficiary.

Integration service and webapp deployment to staging

GET Dispatch

This will be posted as a subject, use AvniSubjectRepository instead of AvniEncounterRepository.

We also create Local Inventory items (subjects) if they do not exist and add them to the dispatch subject before creating it.

POST Activity

Minor changes to attributes

Change source from encounter to subject of type Activity.
Would be linked only to a Distribution, never to a Demand.

Use MetaDataMappings to transpose AvniResponse to GoonjRequests

Currently we make use of code to map Avni entity fields to Distribution entity fields for sync from Avni to Salesforce, instead make use of MetaDataMappings to transpose AvniResponse to GoonjRequests.

Instantiate new tenant in integration service

• Initialise Amrit module
• Fetch token to invoke API calls on Amit
• To communicate Avni configure Avni session
• Orchestrate sync from Avni to amrit (main and error job)

Exotel Integration

Objective

Push all missed calls created in Exotel as tasks in Avni

Notes

• APIs on both sides are available
• Auth is based on a static API key and token
• There is some mapping we will need based on the called number (each number is mapped to a state and program). This logic will stay within the integration software.
• Exotel API - https://developer.exotel.com/api-console/make-a-call-api#BulkCallDetails

Tasks

1. Setup a Indus Action gradle module - 0.25
2. DB Migration for the module to setup the metadata - 0.25
   2.5 Setup sync status
3. Improve our background scheduler to run integration of multiple projects - 0.5
4. Error handling - 2
5. Duplicate tasks should not be created if there is an open task for a call (Good to have) - (Task GET based on status and metadata observations. This might require standardization of numbers before creating tasks) - 1
6. Happy path (actually create the task) - 2
7. Metadata migration - 2 endpoints (1 to get metadata, another to store metadata. This will be async. Add uuid column) - 3
8. Training implementer to add metadata - 0.5

Analysis

1. Define meta structure for mapping data

Limitation
0801928303 and 91801928303 are 2 different numbers

Remove Purchased / high Value field from Dispatch and DispatchReceipt

Update CreatedBy and LastModifiedBy in Goonj SF entities

Handle Errors while Syncing between Salesforce and Avni

Handle Errors while Syncing between Salesforce and Avni:

• Create ErrorRecord on encountering error during Demand or Dispatch sync from SF to Avni
• Create ErrorRecord on encountering error during deletion of Demand or Dispatch in Avni
• Create ErrorRecord on encountering error during DispatchReceipt, Distribution, Activity from Avni to SF
• As part of mainJob iterate over the errorRecords and process them if they are ready for retry
• Terminate sync of anykind, if there are network issues

Goonj Salesforce integration part 2

In this iteration of the Goonj integration, we will be modifying the existing Goonj integration API to work with the new models.

Context

There has been a change in the implementation of Goonj. We need to modify our existing integration to start working with the new implementation. Details of the new model is documented here

Assumptions

• The new implementation with all its fields are ready. The rules need not be done to start work, but is required for completion.

New API

https://documenter.getpostman.com/view/21052731/2s8YRgpZHJ#eabe9124-1292-4dd3-8888-89f6549266fb

Amrit Integration

We have to push Beneficiary, Household, Birth, and CBAC data to Amrit.

The information on mandatory fields, conditional mandatory fields, and masters is in the following. For all APIs see the postman collection. Check the pre-request script and Tests also to understand the APIs better.
https://docs.google.com/document/d/17iYyxhYMJ3d4B5cZj7qRO_PH79YTM8mo/edit#

Beneficiary registration in Amrit is functionally async involving two APIs - sync and getAmritIdForAvniId. Sync API returns success. In order to get the Amrit ID for the beneficiary "Get AMRIT ID for Avni ID" API is called. If the beneficiary has been created then it would provide Amrit id. For unprocessed and unknown it would not provide any Amrit ID. For failures, it would provide an error.

You can read about integration architecture here first, as the approach discussed uses these concepts - https://avni.readme.io/docs/integration-architecture

Integration Processes

1. Subject: Before saving check whether the subject already exists using 'Get AMRIT ID for Avni ID'.
   If present or not present - then make the save call. If the subject is in error then log the error and proceed to the next record.
   Four states to handle - not saved, not attempted, saved, error.

2. Subject Save Error Check: This is to check for subjects for which create has been called and to make sure any errors in subject created are captured. Check for all subjects' "Get AMRIT ID for Avni ID". If there is an error then add it error records and process the next entity. If the entity is not found then end the integration process. If found then move to the next entity.

3. Subject Re-Save: Not sure whether we need this process. This process is required if a Subject remains in the not found status - this process can try to resave the subject. This will be required only if the async process on the Amrit end is faulty and is missing some records.

4. Child Entities: Call "Get AMRIT ID for Avni ID" for the subject ID. If the Subject is not found then end integration processing. If Subject ID is in error then log this Child Entity in the integration error_log in integration DB and move to the next record.

5. Error Process: Process all error record in order (Subject, Child Entities).

Master Data

There are certain master data used by Amrit in beneficiary registration (not for other entities). For these we have to send the id of master data corresponding to our our concept. Mapping for the master data can be done via the UI. All these masters except location related are available via the API "Get Master Data".

Location master data will be provided by Amrit team which will be imported in our Avni and to mapping metadata in integration database. Avni external api uses location title for identifying a location hence the mapping can be created by dump we receive from Amrit team.

Testing

There are three modes in which we need to do testing - developer, full run, and QA. But we have a single environment, hence the it may become difficult to verify the results clearly. We can assign locations for each type of testing. e.g. District 1-2 for dev, District 3-4 for qa etc.

Usually in integration testing we can do a full run on large volume of existing data to identify any issues, but in this case - we don't have much data. So this will remain a risk when we go live.

Details of access to UAT environment is present in KeeWeb. Using this login we can verify the successful and correct saving of avni records. Use advanced search to find beneficiaries.

Error Records Process

CBAC Integration Process

Handle Deletion of Avni Source entities in Salesforce

Currently we only create or update Avi source Goonj Enitities, DispatchReceipt, Distribution and Activity in Salesforce.
We would need to include "isVoided" field as part of Post Request for those EntityTypes or invoke a separate Deletion API, as per Salesforce design.

Acceptance Criteria:
.* We should be able to delete Goonj Enitities DispatchReceipt, Distribution and Activity in Salesforce if they have been voided in Avni.
.* Handle Deletion errors

Handle boundary conditions at the end of the feed while syncing to and from SF <=> Avni

Handle boundary conditions at the end of the feed while syncing from SF <=> Avni

• Store the lastUpdatedTime of the entity synced from SF to Avni in entitySyncStatus
• Make use of entitySyncStatus readUptoDateTime as is in API to fetch entities to sync

Handle boundary conditions at the end of the feed while syncing from Avni <=> SF

• Store the lastUpdatedTime of the entity synced from Avni to SF in entitySyncStatus
• Make use of entitySyncStatus readUptoDateTime+1second in AVNI get API to fetch entities to sync

Amrit Integration

Upload beneficiary data to Amrit (Blockers: Access to master data to be able to upload beneficiary data to Amrit)

[LAHI] Integrate form from Glific to Avni

Design document

Stories

Beta Give feedback

1. Set up new module in Integration sevice for LAHI #83
   +0
   
2. [Spike] Set up SpringBoot REST client for fetching data using BigQuery APIs with auth token #84
   +0
   
3. Configuration init on all environments #85
   +0
   
4. Set up Avni-client and session for LAHI org #86
   +0
   
5. Set up Mapping config from Glific Output to Avni Subject Input #87
   +0
   
6. Set up Glific Repository and Services #88
   +0
7. Error retries for student sync #89
   +0
   
8. [LAHI] Sync Students registration information from Glific to Avni LAHI#51
   LAHI +1
   

   
9. [LAHI] Add "Other" location values for all locations LAHI#34
   LAHI +1
10. [LAHI] Set-up new UAT org LAHI#61
    LAHI +1
11. Changes in Avni forms required due to Integration #91
    +0
    
12. Error handling for LAHI module #94
    +0
    
13. Create tests for different scenarios in the LAHI Glific flow #95
    +0
14. Make use of inputCategory while initializing Student Data for CodedConcepts #101
    +0
15. Only process those Glific FlowResults which has "registration_flow_complete" set to "Yes" #102
    +0
    

Sakhi Amrit Integration

Some context

Beneficiary post to Amrit takes data from multiple Avni entities - subject (Individual, Household), program enrolment (pregnancy, child), and encounter (CBAC) in Avni. The integration should be triggered on Individual, Pregnancy Enrolment, Child Enrolment, and CBAC encounter - in that order. This is based on assumption that Amrit can take partial data. If not, we would have to relook at this (see the spike below to validate this)

Amrit expects all the relationships of a beneficiary to be posted via a single API. In Avni, the relationship is an independent entity and can be added between two subjects at any time. Hence when posting a beneficiary to Amrit we will be posting only beneficiaries at that point in time. When a new relationship is added both the entities in Amrit should get updated.
One way to solve this is to call Amrit and update all affected beneficiaries on every relationship change.

1. Spike using postman to check whether we can create beneficiary without all the fields.
2. Create concept and concept answer mapping in production integration environment for Sakhi
3. Create following mapping groups and mapping types in integration database and integration code

• MG=(Beneficiary). MT=(CBAC, Household, Beneficiary, Birth)

3. If post of a beneficiary fails then an error record should be created with beneficiary id, enrolment id or encounter id - for each type of event respectively. Each event processing should check whether the parent id is already present in the error record. If it is already present in then it should also log itself to the error record.

4. Create external API in Avni, for individual relative feed, so that one can find the changes in relationship between individuals.

5. On individual relationships update, update Amrit - unless one of the relatives have error.

6. Process all errors for beneficiary and individual relationships feed.

Synchronise DispatchReceivedStatus from Avni to SF

• The DispatchReceivedStatus encounter should be Synced to SF using corresponding DispatchStatus with - voiding status, last modified date time, and sourceId. For the remaining fields use the mapping metadata approach.
• Need DispatchStatusLineItemId to be saved while creating DispatchStatus.DispatchLineItems in Avni
• Save error record for missing mapping for a field, its coded answers, or missing DispatchReceivedStatus in Avni.
• The sync should resume from the previous point if an uncaught exception occurs.
• Make it ready for QA
• Deploy to Staging

GET Local Inventory

New subject of type Local Inventory created/updated.

Create Subject Type and Conepts / AnswerConcepts and Form for Local Inventory. make use of Distribution concepts for this purpose.

Secure http endpoints and define base endpoint for the URLs

1. Currently, the endpoints are served from /. Since the service has to run in the common infrastructure along with other avni services, define the base path for all the endpoints.
2. Protect the URLs so that they can be accessed only by users who are authenticated. (WebSecurityConfig.java and using the role annotation). Some URLs are protected and some are not. Review needs to be done and then protected.

Born Birth Integration Process

Deploy Amrit integration to prod

Config clean up and deploy to

• Before every entityType sync, perform getAmritId operation of beneficiary.

• Config clean up in code base

• Setup config in prod

• Deploy to prod

• strore config in keeweb

Synchronise Demand from SF to Avni

• The demand records should be created in Avni with - address-level related fields, voiding status, last modified date time, and external id. For the remaining fields use the mapping metadata approach.
• Since there are very few fields in demand, the metadata mapping should also be done along with this story.
• Store AccountID as we may need it rather than AccountName while sending data back to SF
• Support handling “DeletedDemands"
• Save error records for missing mapping for a field and its coded answers.
• The sync should resume from the previous point if an uncaught exception occurs.
• Define two scheduled jobs. One to run demand sync and the second to process all demand errors.
• Make it ready for QA
• Deploy to Staging

Acceptance criteria

• All the demands should be visible in the data entry app in Avni for any user who has the privilege.

Deploy to prod

Acceptance criteria

• Prod deployment of integration-service and integration-webapp
• DNS setup
• Log rotation
• Scheduling of jobs for Goonj
• Increase size of ETL machine, deploy there //Not to be done, instead deployed on separate server

Handle duplicate task creation

Acceptance criteria

When there are tasks for a specific number that is not in a terminal state from the same number, then it should not create a new task.

PS: Lower in priority from a delivery order perspective

Household Integration Process

TBD: Download beneficiaries by village (these should contain asha id also for mapping).

POST dispatch receipt API

Context

Dispatch receipt is created in Avni as an Encounter

1. Authenticate with Amrit service

LAHI message integration

###Objective
By the end of this epic, Avni should be able to send HSM templated Whatsapp messages to Glific. These messages can be configured to run at the end of filling up a form, at scheduled times or based on a time that is derived from a form field.

Top level solution

There will be a new module - AvniMessagingService. This service will take in requests to send a message to a beneficiary at a specific point in time. It will also sync HSM messages from Glific, and provide mechanisms to store rules that map parameters in an HSM message to variables available in Avni.

• MessagingService will have its own domain and java module that will not be related to Avni.
• All messages to be sent are saved in a message_queue that will be evaluated and executed separately.
• Templates will be evaluated at the last responsible moment to prevent problems due to change in values
• Each rule will have 2 parts - scheduleRule and runRule. The scheduleRule helps figure out the time when a rule needs to be evaluated and sent. The runRule will evaluate the template.

Sample API for AvniMessagingService

• AvniMessagingService.processEvent(Event e) where Event contains a type (ProgramEnrolment, ProgramEncounter, Individual), state (Create, Delete) and an identifier (id of the object)
• AvniMessagingService will have APIs for CRUD of groups, group members etc that will be called by the webapp or integration service API
• AvniMessagingService will have a scheduled job that sends all relevant messages

API in AvniService

• AvniService.getSchedule(String rule, EntityType entityType, EntityIdentifier entityIdentifier). This will call rules-server and return results
• AvniService.runRules(String rule, EntityType entityType, EntityIdentifier entityIdentifier). This will call rules-server and return results

Data model available here - https://dbdiagram.io/d/6228ad1661d06e6eadcf53d1"

Bulk upload of metadata mapping

Happy path integration with Exotel

API Details

• Avni API - POST /api/task
• Exotel API - https://developer.exotel.com/api-console/make-a-call-api#BulkCallDetails

Tasks

• Setup a gradle module for power
• DB Migration for the module to setup the metadata
• Improve our background scheduler to run integration of multiple projects
• Migrate data from Exotel to @powertest organisation in staging
• Perform mapping
• Handle errors

Domain mapping details

Field Mapping

• Map Task type - "Call"
• Map Task name - "Call"
• Map Task status to "New"
• Map field "To" to "State" and "Program" for the metadata of the task as below
  • 011-41136600 -> State-Delhi, Program-BoCW
  • 011-41132680 -> State-Delhi, Program-RTE
  • 011-41132689 -> State-CG, Program-RTE
• Field "From" should be mapped to "Number"

Upload metadata mapping for locations and master data

Required mainly for production. In uat, we have configured only one village.