Hey all,
First off, I just learned about OpenFn but it looks incredibly cool! I’m just trying to wrap my head around integrating it and I was curious about message duplicates. In particular, what happens if a message is received multiple times by the service. Will it execute any relevant jobs a second time? If so, is there a way to prevent that from happening?
Thanks,
Eric
Hi Eric, sorry for the delay. The short answer is that handling duplicates is a choice in how you implement your integration. “Duplicates” could mean lots of different things, and it’s up to you to decide what’s a duplicate and how you’d like to handle it.
Imagine you get a different record with the same UUID. If your job is idempotent (i.e., if you’re using an “upsert” function rather than a “create” function) you’d let it run and your “desitination system” would handle the duplicate—in this case, performing a “update” on the existing record.
Alternatively, you might not allow for “edits” like that, and actually design OpenFn to fail when it received a message with the same UUID. It would run a “create” in that destination system and then when the resulting “error, duplicate UUID found” message came back from that system, you’d configure another job that notified whoever submitted the record in the first place (maybe over SMS, maybe on Slack, maybe on email?) that their submission was blocked.
Finally, you might have some sort of stateless destination endpoint, like a “make payment” API endpoint on a mobile payment gateway. I wouldn’t want to make duplicate payments over MPESA each time we got a duplicate message in OpenFn, so I might define my own criteria for duplication and write a slightly more complex job in OpenFn. That job would receive the message and first check another MPESA endpoint (maybe it GETs a list of the last 1000 payments we’ve made). If there is a payment matching the one I’m trying to make, based on my own custom criteria, I’d block the new payment and notify some sort of administrator. If there wasn’t, I’d make that POST request to the make_payment API and pay the CHW.
There are plenty of ways to think about this, and each pattern can be configured in OpenFn.
For a more technical answer: the only internal “state” that is kept across jobs in OpenFn is the final state of a timer job. That final state is inherited as the initial state of the next run of that job. (Much in the same way the final state of job A becomes the initial state of job B if job B is triggered by the success of Job A.) Holler if you’ve got more questions, and again, apologies for the delay.
Taylor