Paginate through Salesforce Records

Workflow

Sales

Beginner

Use Template

This is a 'Workflow' template which means that it is a single standalone workflow.

Some workflow templates can be modified to work with other workflow templates - e.g. to convert a data sync between two services from uni-directional to bi-directional

Overview

Copy

This template demonstrates how you can pull records from Salesforce and process them in batches using a process called pagination.

This is a standard method which you will need to use any time you wish to pull large numbers of records (accounts, contacts etc.) from Salesforce, using the Salesforce 'Find records' operation.

The screenshot below shows a set of results that exceeds 20,000 records.

When this happens, Salesforce returns a 'next_page_offset' token.

In this case, you can then run a pagination loop which pulls SFDC results in batches and runs until the 'next_page_offset' token is equal to Null.

If, for example, you have 6000 records, you can pull these in three batches of 2000 (Salesforce allows you to pull a maximum of 2000 records in one call). Rather than making 6000 individual calls which would overwhelm your Salesforce API limits.

Connectors Used

Copy

The following connectors are used in this template. This is provided for reference only - there is no need to read through the linked connector pages, as all you need to know for this template is explained on this page:

End Result

Copy

The data fetched from SFDC will be available in batches for whatever processing you need to carry out. In this case, we are using Data Storage as a generic placeholder to store the fetched Salesforce data.

Prerequisites

Copy

This workflow assumes the following:

• You have a Salesforce instance and can authenticate as an API user.

Getting Live

Copy

In order to configure this workflow for use simply:

1. Authenticate to Salesforce and any other needed systems. Ensure that your authentication will have access to the records required by your automation goals. This authentication must be added to both Salesforce connector steps within the workflow.

2. Identify a relevant trigger based on your requirements:

   • If this is a standalone process to be triggered in the Tray platform, you can use a Manual trigger.

   • If it is to initiate daily or on specific intervals then use a Scheduled Trigger.

   • If you wish to fire the workflow when something happens in another system, then you can use a relevant Service Trigger or a Webhook Trigger if there is no pre-built Tray Service Trigger.

3. Determine which type of record you wish to fetch from Salesforce, and what fields from each record. In the Find Records (salesforce-1) step you can specify the type of record, and also the fields you want to be returned for each record:

   

4. Determine how you wish to process the SFDC records. If the process needs to run quickly, you may wish to leverage the Call/Callable Workflow concept. For detailed instructions, refer to the Implementation notes below.

5. The workflow uses a Manual trigger; click Run workflow in the bottom left of the builder anytime you wish to run the workflow.

Workflow Logic

Copy

The overall logic of the workflow is:

1. A forever loop runs until there are no more batches of records.

2. Within the loop, the 'next_page_offset' token from the last batch of records is fetched using the Data Storage 'Get Value' operation (on the first run, this will be null).

3. The Salesforce 'Find records' operation then pulls a batch of records, using the 'next_page_offset' token to start from where the last batch finished. Each batch is then available for further processing.

4. A boolean condition checks if the 'next_page_offset' token ($.steps.salesforce-1.next_page_offset) received from salesforce is equal to null.

   • If the 'next_page_offset' token = null, the loop will break using the Break Loop step.

   • If the 'next_page_offset' token is not equal to null, it means there are more batches of records to process, and the loop will continue.

5. If the loop continues the final step is to use the Data Storage 'Set Value' operation to set the 'next_page_offset' token for the next batch of records.

Implementation notes

Copy

The Process SFDC Records step within the workflow is a generic placeholder for whatever you may wish to do with each batch of SFDC records.

For simplicity's sake, we are just using data storge to store the records.

There are various ways in which you can use this fetched Salesforce data, such as:

• Looping through each batch of records and using our CSV Editor as an intermediate storage medium as shown in our

  Paginate through Salesforce Records and Email Resulting CSV template

  .

• Looping through each batch of records and sending them to another service

• Making use of callable workflows to send data for sub-processing. This will allow you to do parallel processing of batches. Please also see our documentation on Workflow Threads for further guidance

Step-by-step Explanation

Copy

For educational purposes, the following will take you through how the workflow paginates through the Salesforce records and email resulting CSV.

This will help deepen your understanding of Tray.io and will give you the power to maintain and edit your workflow as necessary.