This is philmon from MedcoAnalytics Technology Solutions. We are experiencing processing time issues when tracking payments and handling data in our database, which is impacting efficiency, especially with high transaction volumes. I would appreciate any insights on optimizing database performance, improving workflow execution speed, and handling large-scale payment tracking efficiently. Additionally, any best practices for monitoring and troubleshooting performance bottlenecks would be very helpful. Looking forward to your suggestions—thank you!
Hey @FilmonK, thanks for reaching out! I presume you’re self-hosting?
What sort of volumes are we talking about here? How many runs per hour, at peak?
Do you know where the bottlenecks are? Are runs slow to get started or just slow to execute? If they’re slow to execute, can you tell from runtime logs where things are slow? Are you crunching a lot of data or calling out to a lot of external services?
Thanks for your response! Yes, we’re self-hosting. At peak, we process 50 to 500 payment transactions per hour, mostly for healthcare providers in Addis Ababa. The bottleneck seems to be during execution—runs start on time but take longer to complete, mainly due to database queries and external API calls. Any insights on optimizing performance would be greatly appreciated!
Ahhh ok. That’s a bit trickier. I was thinking maybe you were having OpenFn DB issues, but it sounds like its external APIs and external DBs that are slow. (Am I understanding this right?)
Sorry if this is obvious/you’ve already tried, but can you batch up requests to reduce the total number of calls? Do these external systems have bulk APIs?
Yes, you’re understanding it correctly.The main delay occurs when external APIs call the OpenFn webhook.
Since our system is handling real-time payment transactions, we need to ensure accuracy and up-to-date tracking. We’ve considered batching, but real-time processing is a priority.Would you have any recommendations on optimizing webhook performance or handling high-frequency external API calls more efficiently?
Hrmmmmm. OK that’s actually a little different. So the issue is that an exrternal system is sending lots of requests to OpenFn.
Two things you could consider:
Measure the time response. A response indicates that the workorder (and the payload, called a “dataclip”) has been stored in the OpenFn DB and the workers have been told to process it. If you detect that the response time on that external-system-to-openfn-inbox-api call is too big, consider adjusting the repo settings in your config, or consider adding more cores/RAM to your DB.
Measure the average queue wait time. (We have a promex endpoint for this.) If the problem is not that getting data INTO OpenFn is too slow, but rather having the workers process that data is too slow, consider adding more ws-worker replicas.
When we manage deployments for our customers, we’re using kubernetes and autoscaling policies that allow the system to rapidly deploy more copies of the ws-worker automatically respond to spikes in traffic.
Is this something we could help you with? (I think you’re already in touch with @aleksa-krolls and @AishaH .)