Openfn/ws-worker:latest docker restarts

Self-Hosting
1

To whom this may concern.
I have cloned the git repo and followed all steps to install via docker however the openfn/ws-worker:latest image (lightning-worker-1 docker name) keeps restarting. I have no errors when deploying docker compose up.
Anyone here found this before?

2

hello, @evertb! Tagging @midigofrank from our team who might be able to assist. Frank, any ideas?

3

Interesting. Thanks @evertb . Can you add some information on your operating system version, docker version, and any logs/details on what’s restarting the service?

4

Awesome thanks Aleksa and Taylor. Sure I will provide you this info.

OS: ubuntu 20.04 LTS
Docker version: 27.1.2 community edition
Error log:

[SRV] ✘ WORKER_SECRET is not set
ELIFECYCLE Command failed with exit code 1.
@openfn/ws-worker@1.5.0 start:prod /app/packages/ws-worker
node dist/start.js “-l” “ws://web:4000/worker”

5

Ah! Environment variables! (Specifically, WORKER_SECRET.) We need to update the docker compose stuff to make it clear (it was not clear—sorry for that) that you’ve still got to set up your environment variables.

I’ll make some changes to the docs, but in the meantime…

cp .env.example .env ## and make changes as needed

then add

env_file: ".env"

to both your web and worker services.

Fuller, better docs coming soon. Sorry for the trouble.

6

Thanks for the fast response Taylor. I have done what you requested however I am still facing the same issue.

7

Weird. I can’t recreate. Would you mind pasting your new docker compose file in here along with the command you ran and the error?

If you’re getting the same error “[SRV] ✘ WORKER_SECRET is not set” it probably means that you haven’t quite set up the WORKER_SECRET environment variable or passed it into those web and worker services quite right.

Drop that file in here and maybe I can help!

8

Docker compose:

x-lightning: &default-app
build:
dockerfile: Dockerfile-dev
context: ‘.’
target: ‘${MIX_ENV:-dev}’
args:
- ‘MIX_ENV=${MIX_ENV:-dev}’
- ‘NODE_ENV=${NODE_ENV:-development}’
depends_on:
- ‘postgres’
restart: ‘${DOCKER_RESTART_POLICY:-unless-stopped}’
stop_grace_period: ‘3s’
tty: ${TTY:-false}
volumes:
- ‘${LIGHTNING_VOLUME:-./priv/static:/app/priv/static}’

services:
postgres:
deploy:
resources:
limits:
cpus: ‘${DOCKER_POSTGRES_CPUS:-0}’
memory: ‘${DOCKER_POSTGRES_MEMORY:-0}’
environment:
- POSTGRES_USER=${POSTGRES_USER:-postgres}
- POSTGRES_PASSWORD=${POSTGRES_PASSWORD:-postgres}
- POSTGRES_DB=${POSTGRES_DB:-lightning_dev}
image: ‘postgres:14.2-alpine’
restart: ‘${DOCKER_RESTART_POLICY:-unless-stopped}’
stop_grace_period: ‘3s’
volumes:
- ‘postgres:/var/lib/postgresql/data’

web:
<<: *default-app
env_file: “.env”
deploy:
resources:
limits:
cpus: ‘${DOCKER_WEB_CPUS:-0}’
memory: ‘${DOCKER_WEB_MEMORY:-0}’
environment:
- DATABASE_URL=postgresql://postgres:postgres@postgres:5432/lightning_dev
depends_on:
- postgres
healthcheck:
test: ‘${DOCKER_WEB_HEALTHCHECK_TEST:-curl localhost:4000/health_check}’
interval: ‘10s’
timeout: ‘3s’
start_period: ‘5s’
retries: 3
ports:
- ‘${LIGHTNING_EXTERNAL_PORT:-127.0.0.1:${PORT-4000}}:${URL_PORT-4000}’

worker:
image: ‘openfn/ws-worker:latest’
env_file: “.env”
deploy:
resources:
limits:
cpus: ‘${DOCKER_WORKER_CPUS:-0}’
memory: ‘${DOCKER_WEB_MEMORY:-0}’
depends_on:
web:
condition: service_healthy
restart: true
command:
[‘pnpm’, ‘start:prod’, ‘-l’, ‘ws://web:${URL_PORT-4000}/worker’]
restart: ‘${DOCKER_RESTART_POLICY:-unless-stopped}’
stop_grace_period: ‘3s’
expose:
- ‘2222’

volumes:
postgres: {}

Docker commands I use:
Stop all containers
docker stop $(docker ps -q)

Start docker compose
docker compose up -d

I monitor the ps by using
watch docker ps

9

Hi Taylor any feedback on this?

10

Okay i managed to solve this. I had to hash out the WORKER_SECRECT form the .env.

1 Like
11

Thanks Taylor for your support on this. I can confirm that after hashing out the WORKER_RUNS_PRIVATE_KEY and WORKER_SECRET entries the Openfn instance is working and I am able to login. I used the exact same docker-compose.yml file that I pasted on this chat.

You may close this as resolved, however you might need to look into this to add these step on your deployment guide.

2 Likes
12

That’s great @evertb ! Would you mind raising a PR that adds your approach to the “Run via Docker” section?

While running via Mix doesn’t require any ENVs, it looks like that the docker path you’ve follow does require ENVs. Maybe borrow from this bit: lightning/README.md at main · OpenFn/lightning · GitHub?

Either way, would be great to share what you’ve done! (I’m also not entirely sure what you meant by “hashing out”. Keen to see that in action!)