davidhuser / dhis2.py

Generic and lightweight Python wrapper for the DHIS2 API using requests

dhis2.py

A Python library for DHIS2 wrapping requests for general-purpose API interaction with DHIS2. It attempts to be useful for any data/metadata import and export tasks including various utilities like file loading, UID generation and logging. A strong focus is on JSON.

Supported and tested on Linux/macOS, Windows and DHIS2 versions >= 2.25. Python 3.6+ is required.

Installation

Python 3.6+ is required.

pip install dhis2.py

For instructions on installing Python / pip for your operating system see realpython.com/installing-python.

Note: this project is not related with the module dhis2 which is installed with pip install dhis2. However, the import statement is for example from dhis2 import Api which is similar to the other dhis2 module.

Quickstart

Create an Api object:

from dhis2 import Api

api = Api('play.dhis2.org/demo', 'admin', 'district')

Then run requests on it:

r = api.get('organisationUnits/Rp268JB6Ne4', params={'fields': 'id,name'})

print(r.json())
# { "name": "Adonkia CHP", "id": "Rp268JB6Ne4" }

r = api.post('metadata', json={'dataElements': [ ... ] })
print(r.status_code) # 200

• api.get()
• api.post()
• api.put()
• api.patch()
• api.delete()

see below for more methods.

They all return a Response object from requests except noted otherwise. This means methods and attributes are equally available (e.g. Response.url, Response.text, Response.status_code etc.).

Usage

Api instance creation

Authentication in code

Create an API object

from dhis2 import Api

api = Api('play.dhis2.org/demo', 'admin', 'district')

optional arguments:

• api_version: DHIS2 API version
• user_agent: submit your own User-Agent header. This is useful if you need to parse e.g. Nginx logs later.

Authentication from file

Load from a auth JSON file in order to not store credentials in scripts. Must have the following structure:

{
  "dhis": {
    "baseurl": "http://localhost:8080",
    "username": "admin",
    "password": "district"
  }
}

from dhis2 import Api

api = Api.from_auth_file('path/to/auth.json', api_version=29, user_agent='myApp/1.0')

If no file path is specified, it tries to find a file called dish.json in:

1. the DHIS_HOME environment variable
2. your Home folder

Get info about the DHIS2 instance

API version as a string:

print(api.version)
# '2.30'

API version as an integer:

print(api.version_int)
# 30

API revision / build:

print(api.revision)
# '17f7f0b'

API URL:

print(api.api_url)
# 'https://play.dhis2.org/demo/api/30'

Base URL:

print(api.base_url)
# 'https://play.dhis2.org/demo'

system info (this is persisted across the session):

print(api.info)
# {
#   "lastAnalyticsTableRuntime": "11 m, 51 s",
#   "systemId": "eed3d451-4ff5-4193-b951-ffcc68954299",
#   "contextPath": "https://play.dhis2.org/2.30",
#   ...

Getting things

Normal method: api.get(), e.g.

r = api.get('organisationUnits/Rp268JB6Ne4', params={'fields': 'id,name'})
data = r.json()

Parameters:

• `timeout`: to override the timeout value (default: 5 seconds) in order to prevent the client to wait indefinitely on a server response.

Paging

Paging for larger GET requests via api.get_paged()

Two possible ways:

1. Process every page as they come in:

for page in api.get_paged('organisationUnits', page_size=100):
    print(page)
    # { "organisationUnits": [ {...}, {...} ] } (100 organisationUnits)

2. Load all pages before proceeding (this may take a long time) - to do this, do not use for and add merge=True:

all_pages = api.get_paged('organisationUnits', page_size=100, merge=True):
print(all_pages)
# { "organisationUnits": [ {...}, {...} ] } (all organisationUnits)

Note: Returns directly a JSON object, not a requests.Response object unlike normal GETs.

SQL Views

Get SQL View data as if you'd open a CSV file, optimized for larger payloads, via api.get_sqlview()

# poll a sqlView of type VIEW or MATERIALIZED_VIEW:
for row in api.get_sqlview('YOaOY605rzh', execute=True, criteria={'name': '0-11m'}):
    print(row)
    # {'code': 'COC_358963', 'name': '0-11m'}

# similarly, poll a sqlView of type QUERY:
for row in api.get_sqlview('qMYMT0iUGkG', var={'valueType': 'INTEGER'}):
    print(row)

# if you want a list directly, cast it to a ``list`` or add ``merge=True``:
data = list(api.get_sqlview('qMYMT0iUGkG', var={'valueType': 'INTEGER'}))
# OR
# data = api.get_sqlview('qMYMT0iUGkG', var={'valueType': 'INTEGER'}, merge=True)

Note: Returns directly a JSON object, not a requests.response object unlike normal GETs.

Beginning of 2.26 you can also use normal filtering on sqlViews. In that case, it's recommended to use the stream=True parameter of the Dhis.get() method.

GET other content types

Usually defaults to JSON but you can get other file types:

r = api.get('organisationUnits/Rp268JB6Ne4', file_type='xml')
print(r.text)
# <?xml version='1.0' encoding='UTF-8'?><organisationUnit ...

r = api.get('organisationUnits/Rp268JB6Ne4', file_type='pdf')
with open('/path/to/file.pdf', 'wb') as f:
    f.write(r.content)

Updating / deleting things

Normal methods:

• api.post()
• api.put()
• api.patch()
• api.delete()

Post partitioned payloads

If you have such a large payload (e.g. metadata imports) that you frequently get a HTTP Error: 413 Request Entity Too Large response e.g. from Nginx you might benefit from using the following method that splits your payload in partitions / chunks and posts them one-by-one. You define the amount of elements in each POST by specifying a number in thresh (default: 1000).

Note that it is only possible to submit one key per payload (e.g. dataElements only, not additionally organisationUnits in the same payload).

api.post_partitioned()

import json

data = {
    "organisationUnits": [
        {...},
        {...} # very large number of org units
    ]
{
for response in api.post_partitioned('metadata', json=data, thresh=5000):
    text = json.loads(response.text)
    print('[{}] - {}'.format(text['status'], json.dumps(text['stats'])))

Multiple params with same key

If you need to pass multiple parameters to your request with the same key, you may submit as a list of tuples instead when e.g.:

r = api.get('dataValueSets', params=[
        ('dataSet', 'pBOMPrpg1QX'), ('dataSet', 'BfMAe6Itzgt'),
        ('orgUnit', 'YuQRtpLP10I'), ('orgUnit', 'vWbkYPRmKyS'),
        ('startDate', '2013-01-01'), ('endDate', '2013-01-31')
    ]
)

alternatively:

r = api.get('dataValueSets', params={
    'dataSet': ['pBOMPrpg1QX', 'BfMAe6Itzgt'],
    'orgUnit': ['YuQRtpLP10I', 'vWbkYPRmKyS'],
    'startDate': '2013-01-01',
    'endDate': '2013-01-31'
})

Utilities

Load JSON file

from dhis2 import load_json

json_data = load_json('/path/to/file.json')
print(json_data)
# { "id": ... }

Load CSV file

Via a Python generator:

from dhis2 import load_csv

for row in load_csv('/path/to/file.csv'):
    print(row)
    # { "id": ... }

Via a normal list, loaded fully into memory:

data = list(load_csv('/path/to/file.csv'))

Generate UID

Create a DHIS2 UID:

uid = generate_uid()
print(uid)
# 'Rp268JB6Ne4'

To create a list of 1000 UIDs:

uids = [generate_uid() for _ in range(1000)]

Validate UID

Check if something is a valid DHIS2 UID:

uid = 'MmwcGkxy876'
print(is_valid_uid(uid))
# True

uid = 25329
print(is_valid_uid(uid))
# False

uid = 'MmwcGkxy876 '
print(is_valid_uid(uid))
# False

Clean an object

Useful for deep-removing certain keys in an object, e.g. remove all sharing by recursively removing all user and userGroupAccesses fields.

from dhis2 import clean_obj

metadata = {
    "dataElements": [
        {
            "name": "ANC 1st visit",
            "id": "fbfJHSPpUQD",
            "publicAccess": "rw------",
            "userGroupAccesses": [
                {
                    "access": "r-r-----",
                    "userGroupUid": "Rg8wusV7QYi",
                    "displayName": "HIV Program Coordinators",
                    "id": "Rg8wusV7QYi"
                },
                {
                    "access": "rwr-----",
                    "userGroupUid": "qMjBflJMOfB",
                    "displayName": "Family Planning Program",
                    "id": "qMjBflJMOfB"
                }
            ]
        }
    ],
    "dataSets": [
        {
            "name": "ART monthly summary",
            "id": "lyLU2wR22tC",
            "publicAccess": "rwr-----",
            "userGroupAccesses": [
                {
                    "access": "r-rw----",
                    "userGroupUid": "GogLpGmkL0g",
                    "displayName": "_DATASET_Child Health Program Manager",
                    "id": "GogLpGmkL0g"
                }
            ]
        }
    ]
}


cleaned = clean_obj(metadata, ['userGroupAccesses', 'publicAccess'])
pretty_json(cleaned)

Which would eventually recursively remove all keys matching to userGroupAccesses or publicAccess:

{
  "dataElements": [
    {
      "name": "ANC 1st visit",
      "id": "fbfJHSPpUQD"
    }
  ],
  "dataSets": [
    {
      "name": "ART monthly summary",
      "id": "lyLU2wR22tC"
    }
  ]
}

Print pretty JSON

Print easy-readable JSON objects with colors, utilizes Pygments.

from dhis2 import pretty_json

obj = {"dataElements": [{"name": "Accute Flaccid Paralysis (Deaths < 5 yrs)", "id": "FTRrcoaog83", "aggregationType": "SUM"}]}
pretty_json(obj)

... prints (in a terminal it will have colors):

{
  "dataElements": [
    {
      "aggregationType": "SUM",
      "id": "FTRrcoaog83",
      "name": "Accute Flaccid Paralysis (Deaths < 5 yrs)"
    }
  ]
}

Check import response

Check the importSummary response from e.g. /api/dataValues or /api/metadata import. Returns true if import went well, false if there are ignored values or the status reports not a OK or SUCCESS. This can be useful if the response returns a 200 OK but the summary reports ignored data.

from dhis2 import import_response_ok

# response as e.g. from response = api.post('metadata', data=payload).json()
response = {
    "description": "The import process failed: java.lang.String cannot be cast to java.lang.Boolean",
    "importCount": {
        "deleted": 0,
        "ignored": 1,
        "imported": 0,
        "updated": 0
    },
    "responseType": "ImportSummary",
    "status": "WARNING"
}

import_successful = import_response_ok(response)
print(import_successful)
# False

Logging

Logging utilizes logzero.

• Color output depending on log level
• DHIS2 log format including the line of the caller
• optional logfile= specifies a rotating log file path (20 x 10MB files)

from dhis2 import setup_logger, logger

setup_logger(logfile='/var/log/app.log')

logger.info('my log message')
logger.warning('missing something')
logger.error('something went wrong')
logger.exception('with stacktrace')

* INFO  2018-06-01 18:19:40,001  my log message [script:86]
* ERROR  2018-06-01 18:19:40,007  something went wrong [script:87]

Use setup_logger(include_caller=False) if you want to remove [script:86] from logs.

Exceptions

There are two exceptions:

• RequestException: DHIS2 didn't like what you requested. See the exception's code, url and description.
• ClientException: Something didn't work with the client not involving DHIS2.

They both inherit from Dhis2PyException.

Examples

• Real-world script examples can be found in the examples folder.
• dhis2.py is used in dhis2-pk (dhis2-pocket-knife)

Changelog

Versions changelog

Contribute

Feedback welcome!

• Add issue
• Install the dev environment (see below)
• Fork, add changes to master branch, ensure tests pass with full coverage and add a Pull Request

pip install pipenv
git clone https://github.com/davidhuser/dhis2.py
cd dhis2.py
pipenv install --dev
pipenv run tests

# install pre-commit hooks
pipenv run pre-commit install

# run type annotation check
pipenv run mypy dhis2

# run flake8 style guide enforcement
pipenv run flake8

License

dhis2.py's source is provided under MIT license. See LICENCE for details.

• Copyright (c), 2020, David Huser

dhis2.py's Issues

Org Unit Level and Paging issue

When querying a dhis2 instance with ~4200 organisation units, I wanted to find the root org unit, namely the single one with level = 1. Using the get_paged function (even though not necessary), the get_paged function returns the single root orgunit info multiple times, base on the number of pages needed for the full org unit tree - in other words, if I set the page_size to 50, I get 85 copies of the root (~4200/50) and if I set it 500, I get 9 copies, (~4200/500)

api.delete does not allow payload

Description

Cannot remove dataElements from dataElementGroups as in DHIS2 dev guide, section:
1.9.4.2. Adding or removing multiple objects
Here you are required to DELETE with a payload of the dataElements in the group you wish to remove.
Attempting a DELETE with payload parameter of either json= or data= fails as this is not supported currently in dhis2.py.
Could you please add this feature in the next release?

Many thanks,
Pete

generate uid with option to do this locally

Drop Python 2.7 support

include a pandas extension

pipenv install dhis2.py[pandas]

Retry on failure

Retry on failure for HTTP requests. e.g. with a function decorator with arguments for #retries / backoff factor

• https://findwork.dev/blog/advanced-usage-python-requests-timeouts-retries-hooks/

request param validation should accept lists of tuples

Thanks for the lovely library!

I found a limitation that should be simple to fix.
Many DHIS2 API endpoints allow repeated parameters, but this library does not.

For example, table 1.29 in the 1.17.6. Reading data values section of DHIS2 API docs describes several parameters that "Can be repeated any number of times." (such as dataSet, orgUnit, period, etc)

Luckily, the requests library supports repeated parameters if the caller provides as a list of tuples: http://docs.python-requests.org/en/latest/user/quickstart/#more-complicated-post-requests

However, the _validate_request method raises if params is not a dict

Timeouts

implement optional parameter for https://requests.readthedocs.io/en/master/user/quickstart/#timeouts

You can tell Requests to stop waiting for a response after a given number of seconds with the timeout parameter. Nearly all production code should use this parameter in nearly all requests. Failure to do so can cause your program to hang indefinitely.

at https://github.com/davidhuser/dhis2.py/blob/master/dhis2/api.py#L202

Initial Update

The bot created this issue to inform you that pyup.io has been set up on this repo.
Once you have closed it, the bot will open pull requests for updates as soon as they are available.

Suppress chain of exceptions when custom exception is raised

https://stackoverflow.com/questions/52725278/during-handling-of-the-above-exception-another-exception-occurred

Remove assertion in loading of dish file

minor bug: Remove assertion statement:

... used assert to enforce interface constraints. However, assert is removed with compiling to optimised byte code (python -o producing *.pyo files). This caused various protections to be removed.

Test results:
>> Issue: [B101:assert_used] Use of assert detected. The enclosed code will be removed when compiling to optimised byte code.
   Severity: Low   Confidence: High
   Location: dhis2.py/dhis2/api.py:176
   More Info: https://bandit.readthedocs.io/en/latest/plugins/b101_assert_used.html
175	            password = section["password"]
176	            assert all([baseurl, username, password])
177	        except (KeyError, AssertionError):

import_response_ok is fooled by new (?) return format from the api

I am seeing this on DHIS2 2.40.0:
import_response_ok assumes that the number of imports can be found through response['importCount']
Unfortunately, this has apparently been changed to response['response']['importCount'].
(There two responses, the parameter-name of import_response_ok and the element returned by the api ....)

json pretty print

With Pygments so that a JSON string appears pretty.
http://pygments.org

paging for /api/analytics/events/query fails with a KeyError

dhis2.py 2.0.2 -- 2.1.2

from dhis2 import Api

PROGRAM_UID = 'IpHINAT79UW'

def get_data_no_paging(api, params):
  result = api.get(f'analytics/events/query/{PROGRAM_UID}', params=params).json()
  print(result['rows'][0])


def get_data_paging(api, params):
  for page in api.get_paged(f'analytics/events/query/{PROGRAM_UID}', params=params, page_size=100):
    print(page['rows'][0])


def main():
  api = Api('play.dhis2.org/demo', 'admin', 'district')
  
  # doesn't work with this type of params
  #params = [
  #  ('dimension', 'pe:LAST_12_MONTHS'),
  #  ('dimension', 'ou:ImspTQPwCqd'),
  #  ('dimension', 'A03MvHHogjR.a3kGcGDCuk6'),
  #  ('stage', 'A03MvHHogjR'),
  #  ('outputType', 'EVENT')
  #]

  params = {
    'dimension': [
      'pe:LAST_12_MONTHS', 
      'ou:ImspTQPwCqd', 
      'A03MvHHogjR.a3kGcGDCuk6'
      ],
    'stage': 'A03MvHHogjR',
    'outputType': 'EVENT'
  }

  get_data_no_paging(api, params)
  get_data_paging(api, params)

if __name__ == '__main__':
  main()

results in:

['q0Hcikut16c', 'A03MvHHogjR', '2019-11-12 00:00:00.0', '2020-11-12 01:00:00.0', '2020-11-12 01:00:00.0', '', '0.0', '0.0', 'Ngelehun CHC', 'OU_559', 'DiszpKrYNg8', '5.0']

Traceback (most recent call last):
  File "main.py", line 41, in <module>
    main()
  File "main.py", line 38, in main
    get_data_paging(api, params)
  File "main.py", line 11, in get_data_paging
    for page in api.get_paged(f'analytics/events/query/{PROGRAM_UID}', params=params, page_size=100):
  File "/opt/virtualenvs/python3/lib/python3.8/site-packages/dhis2/api.py", line 409, in page_generator
    page_count = page["pager"]["pageCount"]
KeyError: 'pager'