Salesforce Upsert Statement

We are struggling to get our upsert statement for the Salesforce adapter to work. Does anything look wrong? How will we know if the records have successfully upserted? I understand in the openfn.org UI, SF will send back record Ids to the inbox, how can we verify this in the CLI? When we run this we are getting an empty reference.

Our http adapter job is running successfully, but I am including this as well in case.

http adapter Job 1: 
// GET PATIENT DATA FROM EMR
get('/users');

// Map patient data
fn(state => {
    const patients = state.data; 
    const mappedPatientsForSf = patients.map(c => {
        return {
            patient: {
            email: c.email,
            name: c.name,
            username: c.username,
            street: c.address.street,
            suite: c.address.suite, 
            //vera__Province__c: c.address.city,
            zipcode: c.address.zipcode, 
            emergency_phone: c.phone ,
            parent_acount: "rtu_hospital"
            }
        }
    });

    return { ...state, mappedPatientsForSf };
})
Salesforce adapter Job 2: 
fn( state => {
each( 
  dataPath('mappedPatientsForSf.patient[*]'),
  upsert(
    'vera__Beneficiary__c',
    'vera__External_ID__c',
    fields(
      field('Name', dataValue('mappedPatientsForSf.name')),
      field('vera__External_ID__c', dataValue('mappedPatientsForSf.email')),
      field('vera__Username__c', dataValue('mappedPatientsForSf.username')),
      field('vera__Emergency_Contact_Phone__c', dataValue('mappedPatientsForSf.emergency_phone')),
      //relationship('vera__Hospital__r', 'vera__External_ID__c', dataValue('parent_account'))
    ) 
  )
)
return state;
});

hey @bryceVeraSolutions ! welcome :heart:

yeah, looks like two things:

  1. you’ve wrapped your stuff up inside one of these “custom advanced javascript escape hatch” fn(...) blocks. That’s generally a no-no unless you are specifically doing some javascript object manipulation and not using one of the built-in helper functions.
  2. the path to the array looks like it might be off. dataPath(...) takes you to a point inside state.data, whereas your mappedPatientsForSf array is up at the root of state.

try this:

each(
  "$.mappedPatientsForSf[*]",
  upsert(
    "vera__Beneficiary__c",
    "vera__External_ID__c",
    fields(
      field("Name", dataValue("mappedPatientsForSf.name")),
      field("vera__External_ID__c", dataValue("mappedPatientsForSf.email")),
      field("vera__Username__c", dataValue("mappedPatientsForSf.username")),
      field(
        "vera__Emergency_Contact_Phone__c",
        dataValue("mappedPatientsForSf.emergency_phone")
      )
      //relationship('vera__Hospital__r', 'vera__External_ID__c', dataValue('parent_account'))
    )
  )
);

Hi Tyler, thanks for the reply. We’ve had the upsert job run successfully a few times, but we are unable to validate what was added, and nothing is getting added to the state after this. Here is the code I am running to try to query recently added records so we can get some feedback as we can’t login to the org to validate. This is the code I have so far:

const setDays = (date, x) => {
  let updatedDate = new Date(new Date(date).setDate(date.getDate() + x));
  return updatedDate.toISOString();
};

each(
  '$.mappedPatientsForSf[*]',
  upsert(
    'vera__Beneficiary__c',
    'vera__External_ID__c',
    fields(
      field('Name', dataValue('mappedPatientsForSf.name')),
      field('vera__External_ID__c', dataValue('mappedPatientsForSf.email')),
      field('vera__Username__c', dataValue('mappedPatientsForSf.username')),
      field(
        'vera__Emergency_Contact_Phone__c',
        dataValue('mappedPatientsForSf.emergency_phone')
      )
      //relationship('vera__Hospital__r', 'vera__External_ID__c', dataValue('parent_account'))
    )
  )
);

query(
  `SELECT Name,  Id, vera__External_ID__c 
    From vera__Beneficiary__c Where CreatedDate >= ${setDays(new Date(), -1)}`
)(state).then(state => {
  if (!state) {
    console.error('Query result is undefined or null');
    return;
  }
  console.log('Data from Salesforce:', state.references[0]);
});

hey @aseidas-vera After you run your job with the upsert operation, what’s saved in the output.json file? It should have the final output you sent to the Salesforce API.

When using the CLI, your logs might look like this, and tell you where the output is being saved (e.g., ./output.json).

[CLI] ✔ Compiled job from hello.js
[JOB] ℹ My name is { YourName }
[R/T] ✔ Operation 1 complete in 0ms
[CLI] ✔ Writing output to ./output.json
[CLI] ✔ Done in 366ms! ✨

You can also specify a custom output destination when you run your job by including -o {filename} in your command…
openfn path/to/job.js -ia {adaptor-name} -o ./tmp/aleksaOutput.json

Another thing I think you can do is add an fn(..) blog after your upsert operation to log the final state of the job…

upsert(...); 

fn(state => {
  console.log('Final output ::', state.data); 
  return state; 
}); 

All that said, your approach of querying Salesforce should also work as extra QA. Did that query return anything in your last console.log?

@taylordowns2000 any other best practices to suggest?

Spot on. Also @aseidas-vera , it looks like that particular salesforce upsert will push the response from each call into state.references.

If you added an:

fn(state => {
  console.log(state.references)
  return state;
});

at the end you should be able to see what they’re sending back.

Hi All, I’ve started testing with our own sandbox for some better visibility. I’m still struggling to get any useful debugging errors. The job seems to be failing while logging into the instance but it still sends back a 200 response code. Here are the jobs and config:

getUsers.js


get('users');

fn(state => {
  const patients = state.data;
  const mappedPatientsForSf = patients.map(c => {
    return {
      email: c.email,
      name: c.name,
      username: c.username,
      street: c.address.street,
      suite: c.address.suite,
      city: c.address.city,
      zipcode: c.address.zipcode,
      emergency_phone: c.phone,
    };
  });

  return { ...state, mappedPatientsForSf };
});

alterState(state => {
  state.data = state.mappedPatientsForSf;
  return state;
});

syncToSalesforce.js

const account = '001DQ000006BEcjYAG';

each(
  '$.data[*]',
  console.log('Input First Name ::', state.data[0].name.split(' ')[0]),
  console.log('Input Last Name ::', state.data[0].name.split(' ')[1]),
  console.log('Input Email ::', state.data[0].email),
  console.log('Input Phone ::', state.data[0].phone),
  console.log('Input Account ::', account),

  upsert(
    'Contact',
    'External_ID__c',
    fields(
      field('FirstName', dataValue('name').toString().split(' ')[0]),
      field('LastName', dataValue('name').toString().split(' ')[1]),
      field('External_ID__c', dataValue('email')),
      field('Email', dataValue('email')),
      //field('vera__Username__c', dataValue('username')),
      //field('vera__Address__c', dataValue('street')),
      field('OtherPhone', dataValue('emergency_phone')),
      field('AccountId', account)    )
  )
);

fn(state => {
  console.log(state.references);
  return state;
});

The records are getting into data.state but the upsert is failing. I am not seeing anything logged in Salesforce other than a successful login to the org.


I’m sure I’m missing something simple… :melting_face:

Hi Gang,

Happy Monday, I think we are almost there, I’m still getting weird results with the upsert. I’m adding a key prefix to the external ID to ensure we aren’t stepping on external IDs that have been previously uploaded. Here is what I have:

const setAddress = (street, suite, city, zipcode) => {
  if (street === undefined) {
    console.log('street is undefined');
  } else {
    let concatAddress = street + ',' + suite + ',' + city + ',' + zipcode;
    console.log(concatAddress);
    return concatAddress;
    console.log(concatAddress);
  }
};

const randomString = 'xx_';
const account = 'rtu_hospital';

const provinceMap = {
  'Cumbria': 'Gwenborough',
  'Wisokyburgh': 'Cumbria',
  'McKenziehaven': 'Cornwall',
  'Cornwall': 'South Elvis',
  'Roscoeview': 'Cornwall',
  'South Christy': 'Cornwall',
  'Howemouth': 'Essex',
  'Aliyaview': 'Essex',
  'Bartholomebury': 'Essex',
  'Lebsackbury': 'Essex',
};




each(
  dataPath('data[*]'),
  upsertIf(
    dataValue('data.email') != undefined,
    'vera__Beneficiary__c',
    'vera__External_ID__c',
    fields(
      field('Name', dataValue('name')),
      field('vera__External_ID__c', randomString + dataValue('email')),
      field('vera__Username__c', dataValue('username')),
      field('vera__Emergency_Contact_Phone__c', dataValue('phone')),
      field('vera__Address__c', state =>
        setAddress(
          dataValue('address.street'),
          dataValue('address.suite'),
          dataValue('address.city'),
          dataValue('address.zipcode')
        )(state),
        console.log('address: '       + setAddress(
          dataValue('address.street'),
          dataValue('address.suite'),
          dataValue('address.city'),
          dataValue('address.zipcode')))
      ),
      field(
        'vera__Province__c',
        state => provinceMap[dataValue('address.city')(state)]
      )
      //relationship('vera__Hospital__r', 'vera__External_ID__c', account)
    ),

    console.log('upserted: ' + dataValue('data.name'))
  )
);

Here is the output, I am expecting to see the string values, but seeing this:

"(state) => {
    return (0, import_jsonpath_plus.JSONPath)({ path, json: state })[0];
  },

Here is the output:

Hello @aseidas-vera,

I took a close look into the upsertIf() operation and i have noticed couple of issues, For example
randomString + dataValue('email') needs to be state => randomString + dataValue('email')(state)

Another one is one

state => setAddress(
    dataValue('address.street'),
    dataValue('address.suite'),
    dataValue('address.city'),
    dataValue('address.zipcode')
 )(state)

since setAddress() is not an adaptor function then it doesn’t take state, but dataValue() is adaptor function and it take state. The right way for that will be

state => setAddress(
  dataValue('address.street')(state),
  dataValue('address.suite')(state),
  dataValue('address.city')(state),
  dataValue('address.zipcode')(state)
)

Here is the full correction for the upsertIf() operation

each(
  'data[*]', // Use this json path to get data from state.data
  upsertIf(
    dataValue('email') != undefined,
    'vera__Beneficiary__c',
    'vera__External_ID__c',
    fields(
      field('Name', dataValue('name')),
      field(
        'vera__External_ID__c',
        state => randomString + dataValue('email')(state)
      ),
      field('vera__Username__c', dataValue('username')),
      field('vera__Emergency_Contact_Phone__c', dataValue('phone')),
      field('vera__Address__c', state =>
        setAddress(
          dataValue('address.street')(state),
          dataValue('address.suite')(state),
          dataValue('address.city')(state),
          dataValue('address.zipcode')(state)
        )
      ),
      field(
        'vera__Province__c',
        state => provinceMap[dataValue('address.city')(state)]
      )
      //relationship('vera__Hospital__r', 'vera__External_ID__c', account)
    )
  )
);

hey @aseidas-vera , to do some custom string manipulation like this, throw it into an anonymous function.

In my refactor below, I’ve done 3 things:

  1. put that string concatenation inside a function and used interpolation
  2. when operating inside an anonymous function, like for that “setAddress” section you had, i feel like using the dataValue helper is a bit ugly because you have to call it explicitly. (you’ve moved away from the magic of the pipeline, you’re in custom land, just point to the thing you want with state.data.blah rather than saying dataValue('blah')(state)
  3. same for the provinceMap stuff at the end!
each(
  'data[*]',
  upsertIf(
    dataValue('email') != undefined,
    'vera__Beneficiary__c',
    'vera__External_ID__c',
    fields(
      field('Name', dataValue('name')),
      field(
        'vera__External_ID__c',
        state => `${randomString}${state.data.email}`
      ),
      field('vera__Username__c', dataValue('username')),
      field('vera__Emergency_Contact_Phone__c', dataValue('phone')),
      field('vera__Address__c', state => {
        const { street, suite, city, zipcode } = state.data.address;
        return setAddress(street, suite, city, zipcode);
      }),
      field('vera__Province__c', state => provinceMap[state.data.address.city])
      // relationship('vera__Hospital__r', 'vera__External_ID__c', account)
    )
  )
);

whatcha think? is this useful?