I’m intending to use the CLI to export collection data from one server and import the data into a different server.
I’m using this command to export:
$ openfn collections get "cce-data-rtmd" \* --lightning http://<source_server>
It’s working, ALMOST as expected. For deeply nested objects I’m getting a javascript representation of the object instead of the actual object data itself.
Here’s a snippet of what I’m getting:
'abc123': {
AID: 'xxx',
DLST: { TVC: [Object], TAMB: [Object] },
},
…versus a snippet of what I’m expecting:
'abc123': {
AID: 'xxx',
DLST: {
TVC: {
SID: 'aaa',
SMFR: 'Berlinger & Co. AG',
SMOD: 'SmartSensor TEMP'
},
TAMB: {
SID: 'bbb',
SMFR: 'Berlinger & Co. AG',
SMOD: 'SmartMonitor SITE L'
}
},
},
Is there a flag I should be using or is this perhaps a limitation of the implementation?
Sharing just for fun:
I thought outside the box and I’ve been able to unblock myself by using the fancy Gmail adaptor! Here’s a sample of the job code I used:
collections.get("cce-data-rtmd", { key: "*" });
fn(state => {
// Convert the collection data to a string.
const data = JSON.stringify(state.data);
// Embed the collection string into a file object.
const file = {
filename: "data.json",
content: data,
};
// Embed the file object into an archive object.
const archive = {
filename: "data.zip",
archive: [file],
};
// Add the archive object as an attachment.
state.message = {
to: "ccdxprocessor@gmail.com",
subject: "collection dump",
body: "collection dump",
attachments: [archive],
};
return state;
});
sendMessage($.message);
Once I received the email, I extracted data.json from the archive and referenced it to the job state in my workflow:
{
"workflow": {
"steps": [
{
"id": "hydrate",
"adaptors": [
"common",
"collections"
],
"expression": "jobHydrate.js",
"state": "data.json"
}
]
}
}
Finally, here’s the “hydrate” job code I used to add the data into my new, collection:
// Add the "key" property for each item so we can use it as the item key iterator.
function flattenArray(arr) {
return arr.map((item) => ({
key: item.key,
...item.value,
}));
}
fn((state) => {
state.collection = flattenArray(state.collection);
return state;
});
collections.set("cce-data-rtmd", (item) => item.key, $.collection);
