(DBConnection.ConnectionError) running a pre-built image ie openfn/lightning:v2.0.0-rc3

Self-Hosting
1

Hello Community!
I am running Docker version 24.0.7, build afdd53b, and Docker Compose version v2.23.3-desktop.2
but i am getting an error (DBConnection.ConnectionError) connection not available and request was dropped from queue after 298. Also see my .env file

see my docker compose file

...
  postgres:
    deploy:
      resources:
        limits:
          cpus: '${DOCKER_POSTGRES_CPUS:-0}'
          memory: '${DOCKER_POSTGRES_MEMORY:-0}'
    env_file:
      - '.env'
    image: 'postgres:14.2-alpine'
    restart: '${DOCKER_RESTART_POLICY:-unless-stopped}'
    stop_grace_period: '3s'
    volumes:
      - 'postgres:/var/lib/postgresql/data'

  web:
    image: 'openfn/lightning:v2.0.0-rc3'
    deploy:
      resources:
        limits:
          cpus: '${DOCKER_WEB_CPUS:-0}'
          memory: '${DOCKER_WEB_MEMORY:-0}'
    env_file:
      - '.env'
    depends_on:
      - postgres
    healthcheck:
      test: '${DOCKER_WEB_HEALTHCHECK_TEST:-curl localhost:4000/health_check}'
      interval: '10s'
      timeout: '3s'
      start_period: '5s'
      retries: 3
    ports:
      - '${LIGHTNING_EXTERNAL_PORT:-127.0.0.1:${PORT}}:${URL_PORT}'

  worker:
    image: 'openfn/ws-worker:latest'
    deploy:
      resources:
        limits:
          cpus: '${DOCKER_WORKER_CPUS:-0}'
          memory: '${DOCKER_WEB_MEMORY:-0}'
    depends_on:
      web:
        condition: service_healthy
        restart: true
    env_file:
      - '.env'
    command:
      ['pnpm', 'start:prod', '-l', 'ws://web:${URL_PORT}/worker']
    restart: '${DOCKER_RESTART_POLICY:-unless-stopped}'
    stop_grace_period: '3s'
    expose:
      - '2222'

volumes:
  postgres: {}

FYI, using the main Lightning repo, following run via Docker, the web container connects to the database but the application is not starting at localhost:4000

cc @taylordowns2000

2

Hi @jnsereko!

I think the errors above the connection errors are interesting:

 Postgrex.Protocol (#PID<0.3938.0>) failed to connect: ** (Postgrex.Error) ssl not available

Looks like SSL is enabled for the DB connection (it’s on by default), so I’d suggest changing your env to disable SSL for the compose environment; assuming the DB isn’t accessible outside of your localhost.

# Disable SSL connections for Postgres
# In production mode, SSL connections are enforced by default - uncomment to
# disable this behaviour.
DISABLE_DB_SSL=true    # <== uncommented and true

Let me know if that works for you?

I’d prefer the application instead bailed after several connection failures, because the error you’re referring to is a side effect of not connecting. But thinking out loud if we changed that behaviour the application would exit during DB outages or network issues.

3

Thank you @stu.
I am not getting the failed to connect: ** (Postgrex... but, however the web container exits with the following error that i pretty much do not understand

2024-01-19 13:58:46 10:58:46.664 [info] Loading 140 CA(s) from :otp store
2024-01-19 13:58:46 10:58:46.820 [notice] Application mnesia exited: :stopped
2024-01-19 13:58:46 10:58:46.981 [warning] The `oban_peers` table is undefined and leadership is disabled.
2024-01-19 13:58:46 
2024-01-19 13:58:46 Run migrations up to v11 to restore peer leadership. In the meantime, distributed plugins
2024-01-19 13:58:46 (e.g. Cron, Pruner, Stager) will not run on any nodes.
2024-01-19 13:58:46 
2024-01-19 13:58:46 10:58:46.994 [notice] Application lightning exited: Lightning.Application.start(:normal, [:timex]) returned an error: shutdown: failed to start child: Cachex
2024-01-19 13:58:46     ** (EXIT) an exception was raised:
2024-01-19 13:58:46         ** (MatchError) no match of right hand side value: {:error, {:shutdown, {:failed_to_start_child, Cachex.Services.Incubator, {:shutdown, {:failed_to_start_child, Lightning.AuthProviders.CacheWarmer, {%Postgrex.Error{message: nil, postgres: %{code: :undefined_table, line: "1381", message: "relation \"auth_providers\" does not exist", position: "142", file: "parse_relation.c", unknown: "ERROR", severity: "ERROR", pg_code: "42P01", routine: "parserOpenTable"}, connection_id: 63, query: "SELECT a0.\"id\", a0.\"name\", a0.\"client_id\", a0.\"client_secret\", a0.\"discovery_url\", a0.\"redirect_uri\", a0.\"inserted_at\", a0.\"updated_at\" FROM \"auth_providers\" AS a0"}, [{Ecto.Adapters.SQL, :raise_sql_call_error, 1, [file: ~c"lib/ecto/adapters/sql.ex", line: 1047, error_info: %{module: Exception}]}, {Ecto.Adapters.SQL, :execute, 6, [file: ~c"lib/ecto/adapters/sql.ex", line: 945]}, {Ecto.Repo.Queryable, :execute, 4, [file: ~c"lib/ecto/repo/queryable.ex", line: 229]}, {Ecto.Repo.Queryable, :all, 3, [file: ~c"lib/ecto/repo/queryable.ex", line: 19]}, {Ecto.Repo.Queryable, :one, 3, [file: ~c"lib/ecto/repo/queryable.ex", line: 151]}, {Lightning.AuthProviders.CacheWarmer, :execute, 1, [file: ~c"lib/lightning/auth_providers/cache_warmer.ex", line: 24]}, {Lightning.AuthProviders.CacheWarmer, :handle_info, 2, [file: ~c"deps/cachex/lib/cachex/warmer.ex", line: 95]}, {Lightning.AuthProviders.CacheWarmer, :init, 1, [file: ~c"deps/cachex/lib/cachex/warmer.ex", line: 79]}]}}}}}}
2024-01-19 13:58:46             (cachex 3.6.0) lib/cachex.ex:301: Cachex.start_link/1
2024-01-19 13:58:46             (stdlib 5.0.2) supervisor.erl:420: :supervisor.do_start_child_i/3
2024-01-19 13:58:46             (stdlib 5.0.2) supervisor.erl:406: :supervisor.do_start_child/2
2024-01-19 13:58:46             (stdlib 5.0.2) supervisor.erl:390: anonymous fn/3 in :supervisor.start_children/2
2024-01-19 13:58:46             (stdlib 5.0.2) supervisor.erl:1256: :supervisor.children_map/4
2024-01-19 13:58:46             (stdlib 5.0.2) supervisor.erl:350: :supervisor.init_children/2
2024-01-19 13:58:46             (stdlib 5.0.2) gen_server.erl:962: :gen_server.init_it/2
2024-01-19 13:58:46             (stdlib 5.0.2) gen_server.erl:917: :gen_server.init_it/6
2024-01-19 13:58:48 Kernel pid terminated (application_controller) ({application_start_failure,lightning,{{shutdown,{failed_to_start_child,'Elixir.Cachex',{'EXIT',{{badmatch,{error,{shutdown,{failed_to_start_child,'Elixir.Cachex.Services.Incubator',{shutdown,{failed_to_start_child,'Elixir.Lightning.AuthProviders.CacheWarmer',{#{connection_id => 63,message => nil,query => <<"SELECT a0.\"id\", a0.\"name\", a0.\"client_id\", a0.\"client_secret\", a0.\"discovery_url\", a0.\"redirect_uri\", a0.\"inserted_at\", a0.\"updated_at\" FROM \"auth_providers\" AS a0">>,'__struct__' => 'Elixir.Postgrex.Error','__exception__' => true,postgres => #{code => undefined_table,line => <<"1381">>,message => <<"relation \"auth_providers\" does not exist">>,position => <<"142">>,file => <<"parse_relation.c">>,unknown => <<"ERROR">>,severity => <<"ERROR">>,pg_code => <<"42P01">>,routine => <<"parserOpenTable">>}},[{'Elixir.Ecto.Adapters.SQL',raise_sql_call_error,1,[{file,"lib/ecto/adapters/sql.ex"},{line,1047},{error_info,#{module => 'Elixir.Exception'}
2024-01-19 13:58:48 
2024-01-19 13:58:48 Crash dump is being written to: erl_crash.dump...done
4

It looks like the migrations haven’t been run, have you run the mix task outlined in the readme over here?

5

Thank you @stu. I got the application running with docker compose run --rm web mix ecto.migrate && docker compose up

I am now going to do the same using a prebuilt image

1 Like