CommCare Integration with DHIS2 via OpenFn

chaiwa · December 25, 2017, 3:42pm

Dear OpenFn Users,

In our project, we will need to be able to pull data from DHIS2 into CommCare and vice versa. The Government workers will be submitting data via their DHIS2 Mobile App to their DHIS2 server and we would like to be able to automatically push that data to CommCare for integrated data management and reporting. Similarly, we would like to be able to automatically push data collected via CommCare App to DHIS2.

I am new to OpenFn and tried to look at how that can be achieved via OpenFn but again it is not very clear to me yet.

Anyone with experience setting up such an integration?

I will greatly appreciate.

Thanks and kind regards,

Chaiwa.

taylordowns2000 · February 16, 2018, 11:20am

Adding this to the forum. Thanks, Chaiwa.

@Santiago, could you take a look at the message below? Chaiwa is using the following job (with the inbox UUID removed):

fetchData({
fields: {
dataSet: ‘SaIdNyi73XA’,
orgUnit: ‘J7PQPWAeRUk’,
period: ‘201702’
},
postUrl: “https://www.openfn.org/inbox/XXXXXXXXXXXXXXXXXXXXXXXXXXXXX”

});

sgutierrezakel · February 16, 2018, 2:51pm

Hi Chaiwa,

I’ve tested the fetchData function using this expression and this state:

fetchData(
  { fields: {
      "dataSet": 'pBOMPrpg1QX',
      "orgUnit": 'DiszpKrYNg8',
      "period": '201711'
  }}
)

{
  "data": {},
  "configuration": {
    "username": "admin",
    "password": "district",
    "apiUrl": "[https://play.dhis2.org/demo](https://play.dhis2.org/demo)"
  }
}

As you can see, I’m using the test account for DHIS2. I get this result when I run it:

Getting Data Value Sets:
Get Result: { dataSet: 'pBOMPrpg1QX',
  period: '201711',
  orgUnit: 'DiszpKrYNg8',
  dataValues: 
   [ { dataElement: 'Ix2HsbDMLea',
       period: '201711',
       orgUnit: 'DiszpKrYNg8',
       categoryOptionCombo: 'bRowv6yZOF2',
       attributeOptionCombo: 'bRowv6yZOF2',
       value: '2',
       storedBy: 'system',
       created: '2014-02-27T06:43:18.659+0000',
       lastUpdated: '2014-02-27T06:43:18.659+0000',
       followUp: false },
     { dataElement: 'MSZuQ1mTsia',
       period: '201711',
       orgUnit: 'DiszpKrYNg8',
       categoryOptionCombo: 'bRowv6yZOF2',
       attributeOptionCombo: 'bRowv6yZOF2',
       value: '5',
       storedBy: 'system',
       created: '2014-01-28T00:16:26.887+0000',
       lastUpdated: '2014-01-28T00:16:26.887+0000',
       followUp: false },
     { dataElement: 'r6nrJANOqMw',
       period: '201711',
       orgUnit: 'DiszpKrYNg8',
       categoryOptionCombo: 'bRowv6yZOF2',
       attributeOptionCombo: 'bRowv6yZOF2',
       value: '7',
       storedBy: 'system',
       created: '2014-01-28T00:16:25.438+0000',
       lastUpdated: '2014-01-28T00:16:25.438+0000',
       followUp: false },
     { dataElement: 'eY5ehpbEsB7',
       period: '201711',
       orgUnit: 'DiszpKrYNg8',
       categoryOptionCombo: 'bRowv6yZOF2',
       attributeOptionCombo: 'bRowv6yZOF2',
       value: '2',
       storedBy: 'system',
       created: '2014-02-27T06:43:18.184+0000',
       lastUpdated: '2014-02-27T06:43:18.184+0000',
       followUp: false },
     { dataElement: 'NpJtsQkMTm3',
       period: '201711',
       orgUnit: 'DiszpKrYNg8',
       categoryOptionCombo: 'bRowv6yZOF2',
       attributeOptionCombo: 'bRowv6yZOF2',
       value: '6',
       storedBy: 'system',
       created: '2014-01-28T00:16:25.158+0000',
       lastUpdated: '2014-01-28T00:16:25.158+0000',
       followUp: false },
     { dataElement: 'f7n9E0hX8qk',
       period: '201711',
       orgUnit: 'DiszpKrYNg8',
       categoryOptionCombo: 'bRowv6yZOF2',
       attributeOptionCombo: 'bRowv6yZOF2',
       value: '6',
       storedBy: 'system',
       created: '2014-01-28T00:16:25.983+0000',
       lastUpdated: '2014-01-28T00:16:25.983+0000',
       followUp: false },
     { dataElement: 'lXolhoWewYH',
       period: '201711',
       orgUnit: 'DiszpKrYNg8',
       categoryOptionCombo: 'bRowv6yZOF2',
       attributeOptionCombo: 'bRowv6yZOF2',
       value: '5',
       storedBy: 'system',
       created: '2014-01-28T00:16:26.415+0000',
       lastUpdated: '2014-01-28T00:16:26.415+0000',
       followUp: false } ] }

I get a full set of dataValues. However, when I tried a value of 201601 for period, I got your same results. So maybe you have to check that you fetch dataValues for other periods that has data attached to it.

Hope this info will be helpful for you. Let us know how this goes for you.

Bests,

Santiago

chaiwa · February 25, 2018, 10:44am

Hi @Santiago,
I replicated your steps, using the test account and I got the data as expected! But I am still wondering why I am not able to get the data when I authenticate using the “live user” on our server. I am able to confirm the data is there for the selected period and orgUnit, when I use DHIS2 data visualizer, but getting it via Browser API call or via OpenFn is not giving me the expected results.Are there special permissions(Authorities) the user needs to have, on DHIS2, for them to be able to pull the data via API calls? I know this might be better posted in the DHIS2 forum but just in-case you aware of such a requirement would be great to share!

Thanks and kind regards,
Chaiwa.

chaiwa · June 13, 2018, 7:39am

Dear Taylor and Santiago,

I have tested pulling data from DHIS2 Demo Instance, and all looks good. We are now getting ready for a live integration, would you kindly advise on what specific things we need to put in place for a successful live integration? Particularly, would you kindly highlight key things to put in place like the ones below(both on the DHIS2 side and CommCare side)?:

Access rights

On the side of DHIS2, what level of access should the authenticating credentials have?1. Organisation Units

Datasets
Period

Is there a function that can be used to return current month and then pass the value?
What happens when a job is run more than once in a given period? Data gets duplicated?1. Jobs
Can we pull more than one dataset in a single job run? How can that be configured?
What are the Pros and Cons of using a free tier?
Can we use the same job(same as used to pull data from DHIS2) to push data to CommCare?1. Setup Costs
If we were to consult you to help us setup-up the configurations, what is the expected total cost?
There could be many more questions that I may not have asked that can be critical to a successful integration and I would appreciate if you can highlight them for us as well.

Thanks and kind regards,
Chaiwa.

taylordowns2000 · June 13, 2018, 9:23am

Hey Chaiwa, these are all great questions but since we’re starting to get into the specifics of your project it might be best to take this to email. I’ll do my best to respond here, but I’ll then follow up on email to set a time for us to speak.

If you’ve been managing this DHIS2 implementation for a while now, you’ve likely got a better understanding of what access rights are required to read or write the information you want to use for this project. Once you’ve configured a user with the proper rights, you can use that user’s login information for your OpenFn credential.
Here’s a link to DHIS2 Org Units documentation: https://docs.dhis2.org/2.23/en/user/html/ch04.html . Do you have a specific question?
Also not sure of the ask here: You’ll notice that when writing to DHIS2 using the data values API, you’ll can specify the data set: https://docs.dhis2.org/2.22/en/developer/html/ch01s13.html
Period
Absolutely—use the Javascript Date module: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Date/getMonth . You could write something like return new Date(Date.now()).getMonth()
Let’s discuss—not sure which way you’re sending data but you could configure OpenFn to always create new records, update existing records, or do some logic-based combination of the two.1. Jobs
Yes. Job expressions consist of operations—so long as you use a single adaptor (HTTP, DHIS2, CommCare, etc.) you can chain together as many operations as you’d like and keep the relevant data accessible to you in that job’s state. The only limit is that a single run must be completed in less than 100 seconds, unless you’re using an enterprise plan.
The most obvious pro is that it’s free! The drawback is that you can only have one job active at any given time, and execute no more than 100 runs each month.
You might be able to achieve this using the generic HTTP adapter, but then you’ll loose the pre-configured functionality that handles authentication and wraps DHIS2 and CommCare’s respective APIs.1. Projects can range from less than an hour to several weeks of in-person work! I’m not sure what the scope of your integration project is, but we can certainly work to define it and estimate costs on a call.
Best,
Taylor

chaiwa · June 13, 2018, 9:31am

Thanks Taylor,
This is appreciated. I am sending you a separate email to schedule a call, shortly.

Thanks and kind regards,
Chaiwa.