Exporting and Importing data

Self-Hosting
1

How do I export data from openfn lightning instance and import it to another instance.

2

Hi Mahao, I am working on this feature set at the moment so I believe I may be in a position to help.

You can achieve this by using the openfn cli it provides 2 commands that will do this for you.

1.openfn pull - this command will at pull the current project spec and state from the instance and will provide you with project.yaml and state.json

2.openfn deploy - this command deploys a project.yaml spec to an instance

Let me know how I can assist you further I am actively working on this feature at the moment and I am on standby to see how I can help

1 Like
3

Hi Zacck,
Thanks for this, I will try it out and get back to you if I need more assistance

5

this is What I get when I run pull command

openfn pull
[CLI] :diamonds: Downloading project yaml and state from instance

[CLI] ✘ Command failed!
[CLI] ✘ Error: Failed to fetch project undefined: Bad Request
at getProject (file:///home/mahao/.nvm/versions/node/v19.4.0/lib/node_modules/@openfn/cli/node_modules/@openfn/deploy/dist/index.js:386:13)
at process.processTicksAndRejections (node:internal/process/task_queues:95:5)
at async pullHandler (file:///home/mahao/.nvm/versions/node/v19.4.0/lib/node_modules/@openfn/cli/dist/process/runner.js:1049:33)
at async parse (file:///home/mahao/.nvm/versions/node/v19.4.0/lib/node_modules/@openfn/cli/dist/process/runner.js:1297:20)

6

I have apiKey and endpoint in my .config file

7

Hey @mahaomolise92 would you like to schedule some time to meet and run you through this, I am looking at your issue I apologize I’ve been swamped and held up

8

Hello Zacck, kindly share the time when you will be available to take me through this.

kind regards
Mahao

9

Hello Zacck when will you be available

10

Hey Mahao! Let’s move to email—shoot me and justin@openfn.org and we’ll get something on the books!

11

@mahaomolise92 , openfn pull <uuid> is working well for me on cli version 0.4.0 and above – want to try updating with npm install -g @openfn/cli and seeing how it goes?

You can also bump the log level to debug: openfn pull <some-project-uuid> -l debug

Thanks! Let me know how it goes.

12

I am on version
0.4.0
and this is what I get
[CLI] ❯ reading spec from /home/mahao/projects/lightning_export/Lightning/project.yaml
[CLI] ❯ validated spec: {
“errors”: [
{
“context”: {
“error”: “Not Found”
},
“message”: “Project must have a name”
},
{
“message”: “workflows: must be a map”,
“path”: [
“workflows”
]
}
],
“doc”: {
“error”: “Not Found”
}
}
[CLI] ✘ ERROR: invalid spec
[CLI] ✘ [
{
“context”: {
“error”: “Not Found”
},
“message”: “Project must have a name”
},
{
“message”: “workflows: must be a map”,
“path”: [
“workflows”
]
}
]

13

Hrmmm, I know that the endpoint paths have changed… can you try 0.4.1 with another npm install -g @openfn/cli and test out the following?

Configuration:

{
  "apiKey": "<your-secret-here>",
  "endpoint": "https://demo.openfn.org",
  "statePath": ".state.json",
  "specPath": ".project.yaml"
}

Command: openfn pull <your-project-uuid> -l debug -c ./tmp/.config.json ?

Note that the endpoint is no longer the api path, but the host (should probably be renamed at some point) and you’re pointing to your config file in the command itself.

For me, running on the demo site with the credentials from the demo superuser, I get this:

openfn pull 4adf2644-ed4e-4f97-a24c-ab35b3cb1efa -l debug -c ./tmp/.config.json
[CLI] ♦ Downloading existing project state (as JSON) from the server.
Checking https://demo.openfn.org/api/provision/4adf2644-ed4e-4f97-a24c-ab35b3cb1efa for existing project.
Project found.
[CLI] ♦ Downloading the project spec (as YAML) from the server.
[CLI] ❯ Fetching project spec from "https://demo.openfn.org/api/provision/yaml?id=4adf2644-ed4e-4f97-a24c-ab35b3cb1efa"
[CLI] ❯ reading spec from /Users/taylor/openfn/kit/packages/cli/.project.yaml
[CLI] ✔ Project pulled successfully

And it creates a new project.yaml and state.json file in my working directory.