Another point:
I’d like to build a docker image with all the required adaptors. So I try to split the build in 2 phases and do some npm -i to download the adaptors source but it’s not the correct solution as I add adaptors as dependencies…
What is it that you’re reaching to local adaptors for? Are you trying to run in an offline environment? Or prebuild an image with adaptors cached? Or something else?
When you start Lightning (and actually the Worker, which is the runtime execution service), you can set the adaptor installation folder with WORKER_REPO_DIR
If the adaptor has already been installed to the repo, then the Worker won’t go off and try to install it.
Thanks for you answer.
To be sure I understand well: you propose to not use the property LOCAL_ADAPTORS=true and to just warm the cache with WORKER_REPO_DIR. Is it correct ? I yes it means that we wont be fully offline as the list of adaptors will be downloaded.
I tried several time with these docker images to be sure we will be fully offline but it fails:
ARG REPO_DIR=/app/local_adaptors
## Web
FROM openfn/lightning:v2.14.13 as web
ARG REPO_DIR
ENV OPENFN_ADAPTORS_REPO=${REPO_DIR}
# just to have the common option in the list
# I dont install package here as worker is the runtime
RUN mkdir -p $OPENFN_ADAPTORS_REPO/packages/common
# for openshift....
RUN chmod -R a+rwX /app/lib/tzdata-1.1.3/priv/
## Worker
FROM openfn/ws-worker:v1.19.3 as worker
ARG REPO_DIR
ENV WORKER_REPO_DIR=${REPO_DIR}
ENV OPENFN_REPO_DIR=${REPO_DIR}
RUN npm install -g @openfn/cli \
&& openfn repo install -a common@3.1.2
it fails with:
Failed to import module “@openfn/language-common”
I try to compare files with LOCAL_ADAPTORS=true and LOCAL_ADAPTORS=false and it seems I have the correct resources in right place.
For instance the file /app/local_adaptors/node_modules/@openfn/language-common_3.1.2/dist/index.cjs is present but I have the error
you propose to not use the property LOCAL_ADAPTORS=true and to just warm the cache with WORKER_REPO_DIR.
Yep, exactly. Don’t set LOCAL_ADAPTORS at all. You should be able to just pre-install the adaptors into the correct folder and it should all just work
I suppose alternatively you can bundle the adaptors repo in the image and only load adaptors from there. We usually just do this at dev time when testing adaptors out, but it might work for you.
I’m a bit puzzled that it doesn’t seem to be working in either approach. Can you post the full run logs for me? You might have to switch the filter to debug. There should be more details in there about where it’s trying to load files from.
By the way there are a few things that the app will want the internet for - like if you’re trying to edit a workflow, the editor will call out for docs. Officially we don’t really support offline mode yet. But you should able to run workflows perfectly well once cached and configured.
Thanks for feeding back @icrc-fdeniger . Sounds like you’ve got it working?
But I’m a bit confused looking at your gist!
In my mind local adaptors and a warm repo are incompatible solutions. Adaptors will either be loaded from the monorepo (ie, using LOCAL_ADAPTORS), or they’ll be loaded from the worker’s repo (which the CLI can install straight into). If you’ve got both configured I expect LOCAL_ADAPTORS will win. But I’ve not tested and I’ve not looked at this stuff for a while (probably since January!)
Looks like your gist implements both approaches? In the app I expect it lists adaptors as having the “local” version?
There are pros and cons to both. If you want to edit workflows in the app while offline, local adaptors might be a better solution. If you just want to execute workflows offline, warming the cache is probably the way to go.
In fact I was (totally) confused and was mixing 2 approaches indeed. It was working because by using “warm repo” I was installing required dependencies in the folderOPENFN_ADAPTORS_REPO/node_modules to let LOCAL_ADAPTORS work.
So I kept the LOCAL_ADAPTORS approach, remove the “warm repo” and I had to use this command to install missing dependencies in the ${OPENFN_ADAPTORS_REPO} : openfn repo install -a common@<the_version> --repoDir=${OPENFN_ADAPTORS_REPO}
