akvo / siwins

SIWINS

Development

1. Environment Setup

Instance

This app required a static config to seed and retreieve the data. So, first please provide the instance value like below environment setup (the instance name is "test"):

export SIWINS_INSTANCE="test"

Seed & Sync Auth

This app requires Akvo Flow API Authentication to provides correct credentials when seed or sync form and data points from Akvo FLOW.

Environment Setup:

export AUTH0_CLIENT="string"
export AUTH0_USER="string"
export AUTH0_PWD="string"

Storage Service

When running the test, any upload and download activities will not uploaded directly as Storage object (offered by Google Cloud). It stored inside ./backend/tmp/fake-storage

Environment Setup:

export GOOGLE_APPLICATION_CREDENTIALS=path_to_service_account.json

Email Service

To use the Mailjet Email API, you need to create a Mailjet account, then retrieve your API and Secret keys. They will be used for seed/sync notification purposes.

export MAILJET_SECRET="string"
export MAILJET_APIKEY="string"
export NOTIFICATION_RECIPIENTS="email1,email2"

2. Start the App

Now you have all the required environment ready, then run the App using:

docker volume create --name=siwins-docker-sync

./dc.sh up -d

To stop:

./dc.sh down -t 1

Reset the app:

./dc.sh down -v

The app should be running at: localhost:3000. Any endpoints with prefix

• /api is redirected to localhost:5000
• /config.js is a static config that redirected to localhost:5001/config.js

see: setupProxy.js

3. Database Seeder

Before you seed the baseline data, please make sure that you have all the required file in the following structure:

Folder Path: /backend/source/

backend/source/
├── administration
│   └── fake-geolocations.csv
├── forms
│   └── forms.json
└── solomon-island-topojson.json

Initial Forms & Data Points seeder

Assuming that you have forms.json inside ./backend/source/ folder and have correct Environment setup you will be able to run.

• Run form and fake data seeder

docker-compose exec backend ./fake_seed.sh

• Run form seeder with flow data (require Environment setup)

docker-compose exec backend ./seed.sh

Sync Data Points

To get updated data points from akvo-flow instance, you need to run (require Environment setup):

docker-compose exec backend ./sync.sh

Run Fake Data Points & History seeder

To seed fake data points, run command below:

docker-compose exec backend python -m seeder.fake_datapoint <number_of_data_points>

To seed fake history for data points, run command below:

docker-compose exec backend python -m seeder.fake_history

Running Test

docker-compose exec backend ./test.sh

---

Production

export CI_COMMIT='local'
./ci/build.sh

This will generate two docker images with prefix eu.gcr.io/akvo-lumen/siwins for backend and frontend

docker-compose -f docker-compose.yml -f docker-compose.ci.yml up -d

Then visit: localhost:8080. Any endpoints with prefix

• /api is redirected to backend API: localhost:5000
• /config.js is a static config that redirected to localhost:5001/config.js inside the network container

see:

• nginx config
• mainnetwork container setup

Contact

For further information about the file formats please contact [email protected]

siwins's Issues

Add more param to questions

• Add attributes param to questions inside form json

Additional info for the datapoint

What are you trying to do?

We're trying to add some additional information such as :

• Head Teacher
• Province

in the Pop-up chart details

Migration

• Create database migrations & models
  • Implement to support monitoring survey

---

• Form
• Question Group
• Question
• Option
• Data
• Answer
• History
• Flow Sync

JMP Visualization

What are you trying to do?

Creating JMP visualization in a particular marker or specific data point.

Describe the solution you'd like

• Create JMP questions in Akvo Flow (Questionnaire updated)
• Create JMP Score View
• Sync questions and options with JMP criteria
• #75
• #82

Create endpoint for generic bar charts

Setup reusable chart component

• Remove antd chart
• Install echarts
• Create reusable chart component, to support bar & donut
  • echart docs
  • example: https://github.com/akvo/wai-sdg-portal/tree/main/frontend/src/chart
• Modify chart on datapoint detail, replace antd bar chart with echarts

Implement filtering only for latest data

• Details : https://app.asana.com/0/1204193430573075/1204388170647402
• Before merge this, seed test with only 1 datapoint

Update seeder to support administration/cascade

• Update survey in flow, add cascade question for administration cascade
• Update flow data seeder & sync
• Update fake data seeder

Add clusters on map

Add rose pie chart

Update maps endpoint

• update current maps endpoint to support filter by indicator / indicator options
• update datapoint detail endpoint

Seeder

• Form seeder
  • Registration form
  • Monitoring form
• Datapoint seeder
  • Registration datapoint
  • Monitoring datapoint with history
    • error 500 when fetching from Flow API
• Flow Sync API

Show JMP Visualization in Front-end

Data Detail Endpoint

Endpoint: /data/{datapoint_id}

Response:

{
   id: 1,
   name: "Testing",
   geo: {
      "lat": 2.3454,
      "long": 3.4543,
   },
   answers: [{
       question_id: 134265,
       value: "Any",
       history: false
   }]
}

Add School Information cascade

• School type
• School name
• School code

---

Todo

• Create new migration for cascade (parent children)
• Create a seeder (fetch data from flow cascadeResource)
• Create cascade dump from local db to csv, and use that as resource to seed on test/prod

Frontend - maps & chart

• Show maps with dots (location of the data point)
• Show chart when a dot is clicked (show the monitoring data)
• Add button "show history" to show chart history

Data API

• GET method
• Data endpoint with pagination
• Modify current data detail endpoint (current data detail endpoint is a copy from data/chart/{data_id} and some of value not being used in any UI)

New endpoint for cascade and school type filter

• Endpoint to show cascade value (province, school type)
• Update current maps endpoint to filter by province / school type

School detail popup endpoint

• Create a materialized view for Province level data
  • Group materialized view by year conducted
• Update data detail endpoint
• Create answer history endpoint (to show history on the history switch button)
• Refine duplicated code
• Create test case

Add description to repo

Would be helpful to add a description to the repository to help differentiate it from other repositories in your repository list.

Add province and school type dropdown

Restructure FE filter placement

Add release

Would be nice to have a V1.0.0 (or a V0.1.0) release in GitHub to make it clearer to see when there are major & minor releases.

Instructions for reference on how to do this can be seen here

Refactor map component

Create api for home page national data

Each of the JMP categories with JMP levels and school count

[
  {
    "name": "Water Service Level",
    "option": [
      {
        "name": "Basic",
        "count": 200
      },
      {
        "name": "Limited",
        "count": 50
      },
      ...
    ]
  },
  ...
]

Update BE sources

• Update topojson
• Update administration seeder
• Update fake location for fake datapoint seeder

Incorporating new questionnaire

• Questionnaire on sig.akvoflow under Training > Wash in School copy
• Check current status
• Then prioritize todo/task to unblock Navin

Fake Seeder

Create a fake seeder for the datapoint (registration & monitoring) answer, and the answer history

command:

python -m seeder.fake_datapoint registration <total_datapoint>
python -m seeder.fake_datapoint monitoring <total_datapoint>

Deployment Setup

Endpoint

• GET /api/data/maps
  • get registration form datapoint with GPS location
  • expected return value

  [
    {
      "id": 1,
      "name": "Registration Datapoint name",
      "geo": { "lat": 0.89, "long": -0.98 }, 
    }  
  ]

• GET /api/data/chart/{data_id}
  • get chart data from registration & monitoring datapoint (including history)
  • expected return value

  [
    {
      "id": 1,
      "name": "Registration Datapoint name",
      "monitoring": [
        // monitoring answers include their history
        {
          "question_id": 1,
          "question": "string",
          "value": "answer value",
          "date": "date created",
          "history": "bool"
        }
      ]
    }
  ]

Add advance filter

What are you trying to do?

Add advance filter to filter the markers shown on the map.

---

Add filter params to the endpoint:

• Create a materialized view to summarize all of options answer for each data. Follow this query
• Create a model for the view. See example
• Create endpoint to call all question. See example
• Add the query filter to the endpoint. See example
• Implement to UI

Update maps endpoint

• Datapoint with geopoint and answer by indicator & question id

---

[
  {
    "id": 0,
    "name": "string",
    "geo": [0],
    "answers": { "question": 0, "value": 0 }
  }
]

Wrap up seeder config to general file

Add new homepage

Add school information & year conducted to maps API param

Update question API

• Add question type to question by attributes endpoint
• Add number range to question (number type question) by attributes endpoint

Update fake datapoint seeder

• support registration monitoring condition
• add single dependency checker
• Add MONITORING_FORM const to inform that monitoring form available/not

---

• Need to update datapoint seeder & sync to provide value for year_conducted and school_information columns on data table

Handle updated datapoint on Sync API

• Updated Registration datapoint coming from flow, should be only updated the datapoint and answers, no datapoint going to the history table
• Updated Monitoring datapoint coming from flow, should only update the datapoint and answers, no datapoint going to the history table
• New Monitoring datapoint coming from flow, the new monitoring data should be current answers and the previous answers will be moved to the history table

---

REFACTOR

• use formsInstanceId as table data-id instead of using auto-increment (datapoint seeder file)
  • need to save all datapoint for monitoring data
• use this updated id to detect updated/new datapoint coming from flow sync (data_sync file)
  • check the answer and history table for updated data

Add JMP category to indicator to dropdown

Task references: https://app.asana.com/0/0/1204388170647400/f

Question API

• GET method
• get all questions
• get question filtered by question attribute

---

[
  {
    "id": 0,
    "name": "string",
    "attributes": [
      "string"
    ],
    "option": [
      {
        "id": 0,
        "name": "string",
        "order": 0,
        "code": "string"
      }
    ]
  }
]

Update README

Implement question display name, option description / info text & colour code

Initial Setup

• Docker setup

Add new dashboard page

Add indicator hint in the FE

Add new column to question & option table

• Related to the below asana task:
  • https://app.asana.com/0/1204193430573075/1204388170647398
  • https://app.asana.com/0/1204193430573075/1204386590989280

---

Todo:

• Add attributes & display name column to question table
  • Modify models
  • Modify form seeder (include crud and test)
• Add color & info text column to option table
  • Modify models
  • Modify form/option seeder (include crud and test)
• Modify question indicator endpoint to show question attributes, display_name and option color, description

Add piwik

Create JMP Visualization endpoint

• JMP Chart endpoint
• JMP Chart with filter (same as data/maps filter)
• Add labels to category.json

Create administration seeder

• Create administration table migration, model, crud
• Create administration seeder based on topojson file

Support history on JMP and generic bar chart

• Support history on JMP chart endpoint
• Support history on generic bar chart endpoint