Managing Data Clip size

Platform Help
1

Hi all,

I have a workflow that pulls data from Salesforce and updates Google Sheets. I sometimes receive the error ‘Datclip too large. The result will not be sent back to lightning’. I have tried cleaning the state after the final call to batchUpdateValues in the workflow by using

fn(state => {

return {
    data: state.data,
  };
});

, but I still run into the same issue which causes jobs to fail with ‘Lost’ or ‘Killed’ status. Are there any recommended workarounds for the issue? Is there a way to increase the Dataclip size limit on lightning? The workflow runs without errors locally.

Thank you

2

Hi @djohnson ,

It’s Ayodele here.

There are standard OpenFn limits that are applied out of the box for the smooth performance of the platform for all users (see these standard limits here). For projects on cloud-hosted OpenFn (app.openfn.org), we apply these limits based on the project plans and it is detailed here on the pricing page. These limits include the max run duration, maximum memory usage per run, number of workflows, and data clip size (10 MB).

When a run exceeds the max run duration (5 minutes for a project on the Core plan), it gets killed resulting in a Killed finalState.

The lost runs, on the other hand, we’d like to investigate. The large dataclips generated by that first step may be causing an issue, but we are looking into it right now and will get back in touch soon. Sorry for the trouble here!

Re, those standard limits, there are two basic options here:

  1. You could upgrade to the Growth plan to get up to 20 minutes per run and allow you to store data clips up to 10 MB.
  2. You could also refactor your workflow slightly… if you increase the frequency of the time trigger and implement a cursor (see docs here), so that you’re only getting new or updated records during each run, that would significantly reduce the payload size and duration of each run.

Hope this is helpful!

Ayodele

3

Hi Ayodele,

Thanks for your message. From looking at the logs, it looks like the dataclip was about 95 MB (Final memory usage: [step 95mb]). Even with the upgrade, I will likely need to refactor to fetch fewer records from Salesforce and use appendValues instead of batchUpdate for the google sheets steps.

Thanks again for your help.

4

Hrmmm. Do you need to store the dataclips on Lightning at all, @djohnson ?

Saving 100mb to your account for each step in each run seems like a lot. If you don’t need that data (but also don’t want that warning message) you could disable I/O retention from the “Data Storage” section of your project settings.