Note: Lightning (sometimes called “the v2”) is the latest, fully open source version of OpenFn.
OpenFn provides hosting-as-a-service and deployment-as-a-service for Lightning and lots of folks ask us how we do it. I’m starting this thread so that DevOps teams who are responsible for hosting Lightning can share their approaches.
At OpenFn.org, we use Kubernetes. When that’s not possible or if you’re not doing a “high availability” deployment, we typically recommend two VMs (see “a” below) with the following specs, but it’s also possible to put everything—including a reverse-proxy/load-balancer—on one machine (see “b” in the image).
-
Application VM: 2 vCPU (roughly a single core of a 2.6 GHz Intel Xeon E5) with 3.75 GB memory and 15 gb of storage for the application
- Any linux-based operating system that can run Docker (Ubuntu 20.04+ or Debian 9+).
- Docker (18 or greater).
-
Database VM: 2 vCPU (roughly a single core of a 2.6 GHz Intel Xeon E5) with 3.75 GB memory. Storage required for the DB varies by how many days (if any) of message data you’d like to store on the app itself and cannot be determined without estimates for message/run throughput. If scaling physical storage is not difficult for your particular deployment, start at 40gb.
- A Postgres (at least v14.2) instance (as we run this on a separate server) from the application for greater stability.
If both the application and database are hosted on the same machine (which is not recommended) that machine should have roughly the sum of the requirements above.
It’s also important to note that the application itself provides an HTTP endpoint (no TLS/SSL). A reverse-proxy/load-balancer is expected to provide both HTTPS (HTTP2 compliant) and load balancing between instances.