The SFTP adaptor fails to connect to the server defined in my Credentials object.
Here’s the log from a test run that fails.
RTE Memory limit: 128mb
RTE Timeout: 60s
RTE Payload limit: 10mb
VER Versions:
▸ node.js ***.15.0
▸ worker 1.13.4
▸ @openfn/language-sftp 2.0.12
R/T Executing 0447da97-cad8-4185-9bc8-f6239290***f5
R/T Starting step Fetch Purchase Orders
R/T [linker] loading module @openfn/language-sftp
R/T [linker] Loading module @openfn/language-sftp from /tmp/openfn/worker/repo/node_modules/@openfn/language-sftp_2.0.12/dist/index.cjs
R/T Resolved adaptor @openfn/language-sftp to version 2.0.12
R/T Executing expression (1 operations)
ADA Disconnected
R/T Fetch Purchase Orders aborted with error (1.505s)
R/T Cleaning up state. Removing keys: configuration
R/T connect: Address lookup failed for host
R/T Check state.errors.077f1b81-8f7c-49a5-8a50-23239f3df924 for details
R/T Run complete with status: fail
JobError: connect: Address lookup failed for host
I suspect the issue is due to the fact that the underlying client used by the SFTP adaptor is expecting the host to be provided as an IP address or domain name (without a protocol e.g. http:// sftp://), but the Credentials config requires a protocol (unless I’m doing something wrong there).
Some possible workarounds (would need to know whether any of these are possible)
Use a pre-execution hook to override the config (remove the sftp:// protocol) before the SFTP clients attempts to connect to the server.
Use the common adaptor (and pull in the SFTP client library). OpenFn needs to allow the job to include an external library (I haven’t seen any examples that demonstrate that capability yet).
Build my own SFTP adaptor (or override the existing one) and remove the protocol before the call to connect.
Any other thoughts?
To be clear, I’m not 100% sure the sftp:// protocol (in the host config property) is the problem here but I’m limited to the info in the logs and what I’m seeing in the docs for these javascript libraries to make an educated case.
Hey @jmiranda, Thank you for reporting the bug and sincerely apologize for the inconvenience. I will work on a fix for the sftp adaptor host validation issue you reported. Meanwhile can you try the following approach.
@jmiranda thank you for the feedback—I’d love to understand more about the specific problem this approach aims to solve.
At the moment, we export a few adaptor functions from the common adaptor that we believe are broadly useful, and include them in each adaptor. These functions support basic transformations where needed.
However, for more specific or complex data transformation needs, we recommend adding a dedicated transformation step in the workflow that uses the common adaptor directly. This allows you to leverage any function from common to meet your business logic requirements.
By decoupling transformation from the extraction or loading steps, it becomes easier to troubleshoot issues, test transformations independently, and maintain a clear separation of concerns in your workflow pipeline.
I don’t. The idea was that if I couldn’t get the SFTP adaptor working on its own, I could build similar SFTP logic using the underlying libraries to sidestep the credentials bug. But the Raw JSON credentials workaround is working so I’m good with that.
Good news @jmiranda, We have fixed the validation error on our recent sftp adaptor release. You should be able to add credentials without hostUrl validation errors
for creating Raw JSON credential
, I will keep you posted when the credential validation is fixed for sftp adaptor