Support compression on Lightning webhook

Feature Requests
1

Hello OpenFn team. I’d like to suggest that Lightning webhooks support compressed request bodies. I view this working the “standard” way:

  • Sender compresses a JSON payload body (e.g. with gzip)
  • Sender triggers the webhook via HTTP POST with headers: Content-Type: application/json (or Content-Type: application/json; charset=utf-8) and Content-Encoding: gzip, plus the compressed body.
  • Webhook receives the POST, decompresses the body according to the Content-Encoding header, does whatever it needs behind-the-scenes (e.g., generate and persist state to the inbox), and replies to the caller.

The response code should only be a 20x if content is successfully decompressed AND the decompressed content matches the Content-Type header (i.e., JSON).

I’d suggest starting with support for gzip and strongly consider support for brotli.

2

It looks like support for decompress stream handler has been merged into Cowboy and is targeted for the upcoming Cowboy 2.11 release. Link to the PR is below. The latest version of plug_cowboy seems like it’s currently pinned at Cowboy 2.7, so it may be a while before things catch up. This is all wild speculation on my part, but it may reflect an on-the-horizon opportunity to support this feature request without building custom request handling using zlib.gunzip or zlib.safeInflate.