When I run a very simple job code, I often stare at the output window for several seconds before I see my results. This may not sound much but it feels like an eternity in the moment, especially when I’m testing stuff and running a lot.
Hitting ctrl+enter is immediate and visceral and exciting. Waiting for “Nothing yet…” to vanish is boring and frustrating.
Look I wrote half this code so I understand that the actual runtime is super quick.
The biggest bottleneck is the time it takes for the worker to pick up the work from Lightning. Depending on the state of the backoff this might be a couple of seconds.
There may be a bottleneck in the Worker while we download the Workflow and credential and state and stuff from Lightning.
There may be a bottleneck in the Worker when we auto-install - it might take most of a second, even when adaptors are installed, to verify the installation. This is often the slowest bit in the CLI
There may be a delay or latency in the processing and sending of log messages from the Worker into Lightning.
At the very least, we discussed telling the user why it’s taking so long.
So:
hubtel adaptor.”N other jobs in the queue.” (If they’re on the official OpenFn.org hosted service, we could even explain how they get their own queue if they upgrade to a paid plan.)If we show users why they’re waiting, it might actually not be so bad.
Yeah there are two ways of cutting this. One is to make the Run feel faster. Keeping the user up to date helps here.
The other is to make it actually faster in the code. Making the lightning push to workers, rather than waiting for a pull, would help - but it’s a big architectural change (which should be discussed elsewhere)
oh but before we’d consider that i’d say we reduce the max of the backoff… it’s probably at 2s now. would it be so bad to check every 500ms?