Example job to extract CommCare data via API

Job Writing
1

A community user recently asked how we might use OpenFn to fetch data from CommCare. Per their API documentation, we can see that there are multiple API endpoints to choose from to work with case or form data.

For example, if we want to get case data, we can leverage the List Cases API endpoint and use the OpenFn http adaptor to send a GET request to extract data.

fn(state => {
  //Let's add an fn(...) block to build our query URL

  //see CommCare API docs for request parameter options
  const caseType = 'Household'; //define case type(s) you have in CommCare
  const limit = 1000; //define how many to extract; max limit is 1k, so you may need to use paging to get more
  const startDate = '2023-05-31T14:00:00'; //define extract start date

  const url = `${state.configuration.baseURl}?type=${caseType}&indexed_on_start=${startDate}&limit=${limit}`;

  return { ...state, url };
});

get(url, {
  headers: { 'content-type': 'application/json' },
});

Now if you are expecting to query more than 1000 records (the CommCare API limit), then the CommCare API will use paging to return only 1k records per page. he trick is that then you will need to send multiple GET requests to page through the API and extract all data (e.g., if there are 1200 cases, the API will separate the data across 2 pages, and you would need to send 2 GET requests to get all of the data).

See the code below for an example job that will dynamically send GET requests depending on the # of total pages returned by the CommCare API.

fn(state => {
    const { baseUrl } = state.configuration;
  
    const caseTypes = ['Household', 'Patient']; //define case type(s) you have in CommCare
    const limit = 1000; //define how many to extract; max limit is 1k, so you may need to use paging to get more
    const startDate = '2023-05-31T14:00:00'; //define extract start date
  
    const queries = caseTypes.map(
      t => `?type=${t}&indexed_on_start=${startDate}&limit=${limit}`
    );
  
    return { ...state, queries, baseUrl, payloads: [] };
  });
  
  // create a "recursiveGet" which will call itself again if CommCare tells us there's
  // more data to fetch; so if there are 1,200 records --> then 2 GET requests will be sent
  fn(state => {
    const recursiveGet = url =>
      get(
        url,
        {
          headers: { 'content-type': 'application/json' },
        },
        nextState => {
          const now = new Date();
          const { baseUrl, data, payloads } = nextState;
  
          const { meta, objects } = data;
          console.log('Metadata from CommCare response:', meta);
  
          const finalState = {
            ...nextState,
            payloads: [...payloads, ...objects],
          };
  
          if (meta.next) {
            console.log('Another query detected, recursing...');
            return recursiveGet(`${baseUrl}${meta.next}`)(finalState);
          }
          finalState.lastRunAt = now.toISOString().slice(0, 19);
          return finalState;
        }
      );
  
    return { ...state, recursiveGet };
  });
  
  // for each initial query, fetch data recursively
  each(
    '$.queries[*]',
    fn(state => {
      return state.recursiveGet(`${state.baseUrl}${state.data}`)(state);
    })
  );
  // log the total number of payloads returned by CommCare
  fn(state => {
    console.log('Count of payloads', state.payloads.length);
  
    return { ...state, references: [], data: {} };
  });

OpenFn’s CommCare adaptor (see OpenFn Docs) doesn’t yet have a function for extracting data from CommCare, but there are available functions for submitting form submissions and/or bulk uploading case data via CommCare’s Bulk Upload feature - see submitXls docs.