paging for /api/analytics/events/query fails with a KeyError about dhis2.py

[davidhuser]

Hi @pvanliefland , thanks for reporting. The paging control properties are located in another place, if you inspect an example JSON response they are in the data['metaData']['pager'] and not in data['pager'] as usual.

I'd be happy to review a PR! The idea would be to check whether the endpoint is analytics/events/query and switch the way it controls the paging inside the page_generator() func , e.g.

# sketch:
if endpoint = 'analytics/events/query':
    page_count = page['metaData']['pager']['pageCount']

Accessing the rows with the paged data would then be delegated to the calling function (of get_paged) I think.

• https://jira.dhis2.org/browse/DHIS2-10742
• https://docs.dhis2.org/master/en/developer/html/webapi_event_analytics.html

[pvanliefland]

Hi @davidhuser , I'm encountering the same issue. I could try to fix it and make a PR, any pointer?

Thanks!

[pvanliefland]

Hey @davidhuser , I'm working on a fix for this. However, paging is not the only issue - the response format is also quite different. This could be an issue when using merge=True. I've attached a sample response below.

What would be your thoughts on this? I'm leaning towards having collections as a list rather than a string, and iterate over the collections as well when appending data.

{
  "headers": [
    {
      "name": "pi",
      "column": "Enrollment",
      "valueType": "TEXT",
      "type": "java.lang.String",
      "hidden": false,
      "meta": true
    },
    {
      "name": "tei",
      "column": "Tracked entity instance",
      "valueType": "TEXT",
      "type": "java.lang.String",
      "hidden": false,
      "meta": true
    },
    {
      "name": "enrollmentdate",
      "column": "Enrollment date",
      "valueType": "DATE",
      "type": "java.util.Date",
      "hidden": false,
      "meta": true
    },
    {
      "name": "incidentdate",
      "column": "Incident date",
      "valueType": "DATE",
      "type": "java.util.Date",
      "hidden": false,
      "meta": true
    },
    {
      "name": "geometry",
      "column": "Geometry",
      "valueType": "TEXT",
      "type": "java.lang.String",
      "hidden": false,
      "meta": true
    },
    {
      "name": "longitude",
      "column": "Longitude",
      "valueType": "NUMBER",
      "type": "java.lang.Double",
      "hidden": false,
      "meta": true
    },
    {
      "name": "latitude",
      "column": "Latitude",
      "valueType": "NUMBER",
      "type": "java.lang.Double",
      "hidden": false,
      "meta": true
    },
    {
      "name": "ouname",
      "column": "Organisation unit name",
      "valueType": "TEXT",
      "type": "java.lang.String",
      "hidden": false,
      "meta": true
    },
    {
      "name": "oucode",
      "column": "Organisation unit code",
      "valueType": "TEXT",
      "type": "java.lang.String",
      "hidden": false,
      "meta": true
    },
    {
      "name": "ou",
      "column": "Organisation unit",
      "valueType": "TEXT",
      "type": "java.lang.String",
      "hidden": false,
      "meta": true
    },
    {
      "name": "de0FEHSIoxh",
      "column": "WHOMCH Chronic conditions",
      "valueType": "BOOLEAN",
      "type": "java.lang.Boolean",
      "hidden": false,
      "meta": true
    },
    {
      "name": "sWoqcoByYmD",
      "column": "WHOMCH Smoking",
      "valueType": "BOOLEAN",
      "type": "java.lang.Boolean",
      "hidden": false,
      "meta": true
    }
  ],
  "metaData": {
    "pager": {
      "page": 2,
      "total": 163,
      "pageSize": 4,
      "pageCount": 41
    },
    "items": {
      "ImspTQPwCqd": {
        "name": "Sierra Leone"
      },
      "PFDfvmGpsR3": {
        "name": "Care at birth"
      },
      "bbKtnxRZKEP": {
        "name": "Postpartum care visit"
      },
      "ou": {
        "name": "Organisation unit"
      },
      "PUZaKR0Jh2k": {
        "name": "Previous deliveries"
      },
      "edqlbukwRfQ": {
        "name": "Antenatal care visit"
      },
      "WZbXY0S00lP": {
        "name": "First antenatal care visit"
      },
      "sWoqcoByYmD": {
        "name": "WHOMCH Smoking"
      },
      "WSGAb5XwJ3Y": {
        "name": "WHO RMNCH Tracker"
      },
      "de0FEHSIoxh": {
        "name": "WHOMCH Chronic conditions"
      }
    },
    "dimensions": {
      "pe": [],
      "ou": [
        "ImspTQPwCqd"
      ],
      "sWoqcoByYmD": [],
      "de0FEHSIoxh": []
    }
  },
  "width": 12,
  "rows": [
    [
      "A0cP533hIQv",
      "to8G9jAprnx",
      "2019-02-02 12:05:00.0",
      "2019-02-02 12:05:00.0",
      "",
      "0.0",
      "0.0",
      "Tonkomba MCHP",
      "OU_193264",
      "xIMxph4NMP1",
      "0",
      "1"
    ],
    [
      "ZqiUn2uXmBi",
      "SJtv0WzoYki",
      "2019-02-02 12:05:00.0",
      "2019-02-02 12:05:00.0",
      "",
      "0.0",
      "0.0",
      "Mawoma MCHP",
      "OU_254973",
      "Srnpwq8jKbp",
      "0",
      "0"
    ],
    [
      "lE747mUAtbz",
      "PGzTv2A1xzn",
      "2019-02-02 12:05:00.0",
      "2019-02-02 12:05:00.0",
      "",
      "0.0",
      "0.0",
      "Kunsho CHP",
      "OU_193254",
      "tdhB1JXYBx2",
      "",
      "0"
    ],
    [
      "nmcqu9QF8ow",
      "pav3tGLjYuq",
      "2019-02-03 12:05:00.0",
      "2019-02-03 12:05:00.0",
      "",
      "0.0",
      "0.0",
      "Korbu MCHP",
      "OU_678893",
      "m73lWmo5BDG",
      "",
      "1"
    ]
  ],
  "height": 4
}

[davidhuser]

I see @pvanliefland, yes, since the response here is a list of lists (in rows) the merging needs to be handled while still preserving the rest of the response (metaData and headers). I don't understand your suggestion yet, do you refer to use collection as in this line) ?

Given the existing rather high complexity of the get_paged function it might prove useful for maintainability to refactor the different types of responses into new "private" methods, instead of trying to handle everything inside this function. This could later be extended to even handle other types of responses in the future. get_paged is then just used as an entry point for the types of responses that require different handling.

Important would be to maintain the public API of that method, which should detect the kinds of responses (as mentioned, like if endpoint = 'analytics/event/query': do_this().

Sorry that was not fully practical advice but general thoughts, but hoping that it might be a bit easier to implement.

[pvanliefland]

@davidhuser your suggestions make sense, I'm giving it a try right now. Forget about my remark regarding collection - it was unclear. I just need to take into account that for analytics, the key in the response data will always be rows - and not dataElements or indicators.

[pvanliefland]

Hey @davidhuser! I've started a draft PR for this. I intend to test it in a real use case in the coming days.

What do you think about the proposed implementation, with paging classes?

In any case, I think that there are still a few things to clarify - I'm not sure that merging the rows is the only thing to do (thinking about metadata among other things).