Receive response to a downstream step when triggering a workflow via HTTP request

1

Problem: I am sending data from an external application to an OpenFN webhook. In OpenFN, I am creating a job that sends the data to an external API and returns the result from that API. However, OpenFN always returns a response containing worker_order_id instead of the data I want (which is the actual response from the external API).

2

Hi @yemane! Thanks for reaching out.

I’m not quite sure I follow the problem :thinking:

Let me see if I can breakdown what’s happening:

  1. An external application calls the openfn webhook
  2. The workflow is triggered with the correct incoming data
  3. Your job code takes this data and calls out to another external service
  4. The job code takes the response
  5. Maybe something else happens?

Is that a fair summary? At which point in that flow do you get the incorrect data?

1 Like
3

hey @yemane also check out these docs for help understanding:

  • Once you (or an external app) send a POST request to an OpenFn workflow with a webhook trigger → each request will create a Work Order in your OpenFn project and is accessible via the History view . See docs for more on navigating this view: History & Search in OpenFn | OpenFn/docs
  • The body of your POST request (the “data”) will be visible as an Input in the app → Watch this video on how to “inspect” these Inputs and start building your workflow based on these incoming data: Edit Steps via the Inspector | OpenFn/docs

Hoping that helps! If not, please share a summary of what you’re trying to achieve and we’ll try to assist from there

1 Like
4

Hey @yemane , is what you’re looking for synchronous workflow execution?

From how you’re phrasing your problem, it sounds like you’re hoping that when you make a POST to OpenFn, we’ll execute the entire workflow before responding with a 200 (or some other code) and return to you the response from whatever that external API sends back.)

This is really cool. We’ve actually got an old feature request (or github issue) which describes it in greater detail, but the idea is that in future versions of OpenFn you might be able to configure when and how the request that initiates a workflow gets responded to. Here are the options we envision:

  1. Async (how it currently works): we send a 200 when we create the “work order” and your workflow runs asynchronously, whenever an available worker picks it up off the queue.
  2. Sync: we hold open your request and don’t respond until the entire workflow finishes, sending the final state of some particular branch and mapping run exit_codes to HTTP status codes, somehow.
  3. Custom: we hold open your initial HTTP request and only reply when a “respond to webhook request here” step is reached in the workflow, sending whatever reply you want (you pick the code and the response body)

#2 doesn’t really do it for me, but I love the idea of #3. The downside would be that HTTP requests would need to wait a long time to get responses if your Lightning instance had a few million items in the queue.

What do you think?

1 Like
5

Thanks, Taylor! You understood my question exactly. I have solved my problem by sending it to my external API after successfully finishing the job (I added a new API to handle this challenge). However, in my opinion, the best solution is option #3 (custom). Thanks for your help!

1 Like
6

Fantastic! If you’ve got a moment, it would be awesome to hear your thoughts on how you’d like to see #3 work. Are up for requesting it formally in the Feature Requests - OpenFn Community category?