In Microservice, the job using "Flow" trigger always use the "configuration" from the previous Job even though it has its own credential

Let’s say:

  • Job A, assigned with Credential A, the trigger is cron every 5 minutes
  • Job B, assigned with Credential B, the trigger is “Flow” trigger of Job A

I’ve tried this in the platform, it works correctly, the Job B (which got triggered by the success of Job A) has Credential B in state.configuration (because Job A got assigned Credential A, and Job B got assigned Credential B)

But, if using the Microservice (I exported the project.yaml config and applied it to the microservice), the Job B always got the Credential A as its initial state.configuration because of the “Flow” trigger.

Hi @Levi - Thanks for reporting this. If it is working on platform, it sounds like a bug. I have created a Github issue (Flow jobs bug - job with a "flow" trigger is using the credential of the parent job · Issue #35 · OpenFn/microservice · GitHub) and shared with our core team’s engineers to take a closer look.

How did you discover this issue? Was there an error thrown when the Job B started to run? Or did you seen in the Job B run logs that it was trying to connect with the wrong Credential A system?

1 Like

@Levi , thanks for pointing this out—possible that there’s a bug there. Also, we’ve stopped development on Microservice and Lightning, even though it’s in alpha, has actually already surpassed it. Are you up for connecting with @amber so she can help you get set up on Lightning?

Taylor

1 Like

Hi @aleksa-krolls, okay, thank you for the support on this.

Yes, there is an error thrown when the Job B runs which when I read through, it said it was trying to connect using credential A.

I also saw the Job B run logs, then get inside the OpenFn/Microservice container and see the initial state file (state-${EPOCH}-${RANDOM_STRING}.json) used inline with the expression for Job B, and it has configuration object the same value as configuration object in the initial state file for Job A.

@taylordowns2000 Oh I see, I think that’s why I found some rough edges hehe including the “exclusion trigger” not available in the Engine, and also I think the expression key in the project.yaml should start with | instead of >, because I found issues when there is a // comment above a function call (the function call become commented, because in the folded scalars >, newlines become spaces)

Thank you for the invitation, but currently there are quite many stuff on my plate, I will let you know later on when we’re ready.

Hey @Levi!

I have verified a bug in engine (the backend library that we use in microservice and lightning) that would result in the previous jobs credential being added to the state instead of the job that is about to be triggered.

I have fixed it in engine here: Merge pull request #43 from OpenFn/next_state_credential_fix · OpenFn/engine@2c77afc · GitHub, and have released a new version of microservice on DockerHub here: openfn/microservice:v0.4.0.

As @taylordowns2000 pointed out, Lightning is shortly going to completely replace microservice - but we hope this fix sorts out your issue right now.

1 Like

Oh, I wanted to mention - since microservice development has pretty much stopped (your issue here did warrant an exception to that status :sweat_smile: ), with that the changes to engine since the last release of microservice are fairly significant.

So while we do have plenty of tests, and CI gave us the go ahead, this was a very quick release so let us know if you encounter any show stoppers - I can’t guarantee how long we will support microservice.

I did have to upgrade the version of elixir and nodejs in microservice, 1.11 to 1.13 and 14.18 to 16.15 respectively. Node modules are now always installed in priv/openfn, and I would advise against changing that path.

Internally we’ve changed how we install adaptors, namely we only support npm modules now; this means two things:

  • Adaptor names must be full npm module names: e.g. @openfn/language-salesforce@2.10.0
  • Make sure your a required adaptor/version is available npm. (You can check with npm view @openfn/language-salesforce for example). Or head to https://www.npmjs.com/package/@openfn/language-postgresql (replacing the adaptor name with the one you are checking).
1 Like

@stu When using the new version of microservice:v.0.4.0, the container exited with code 0 when running docker-compose up :frowning:

I was using a hacky workaround for microservice:v0.3.3, that is to use the same credential for the 2 jobs but modified the key-value pairs, with the configuration related to Job B added the prefix something like “job_b_” (i.e. “job_b_access_token”)

Hey @Levi , exiting with code 0 sounds like something related to the docker-compose file or even the dockerfile itself, rather than the application. I think Stu mentioned this but microservice isn’t under active development—the only FOSS funding we have right now is earmarked for Lightning.

Did you have an active microservice deployment that was impacted by this upgrade, or were you just testing it out?