Handling rate limits when fetching data (pagination)

When managing the API limits of 3rd party services one of the first things to consider is Pagination.

This comes into play when you are using connector operations (typically 'list' operations such as 'list customers', 'list contracts' etc.) which might return long lists of results (which could be in the dozens, hundreds or even thousands) which you then want to process in some way.

Operations work in this fashion so that e.g. you don't have to make 1000 single API calls to retrieve 1000 records. Doing so would be very inefficient and would also quickly exceed the API limits of the service.

In this case, it is necessary to create a 'pagination' system which breaks the results down into manageable batches (of e.g. 100 records each) and knows when to stop.

The options for pagination will depend on the service connector you are using.

The following example is taken from the Pagination Architecture snippet which shows how to paginate results from a table in Airtable:

It illustrates the key steps involved in basic pagination:

1. At the start of the loop you retrieve the offset token using Data Storage 'Get value' (Scope - Workflow) (Default empty string for first run)

2. You retrieve the records using a 'List records'-type operation. Passing the offset token where appropriate and the batch size / page size (stored in your Project Config as best practice)

3. You then process the records as required (in this example each record is being added to a Google Sheet)

4. At the end of each loop you check if the service (Airtable) has returned an offset token, indicating there are more records to be pulled

5. If so then you store the offset token using Data storage 'Set value' (Scope - workflow) to be retrieved at the start of the next run

6. If not then you break the forever loop as there are no more records to process

Investigating pagination requirements

Copy

When building and testing in Tray you can inspect the input and output logs of connector operations to look for key fields such as 'has_more' or 'offset_token'.

To save time when scoping out your project requirements, you can also make use of our Ops Explorer (beta) dev tool to explore sample input and output payloads for different operations to get a quick idea of any pagination requirements:

Aside from the Airtable example above, some other connector operations which require pagination are:

• The Stripe connector has a 'List customers' operation which returns a has_more property so you know whether to make another request. It also lets you pass in a value called Starting After so that you can enter the ID of the last customer in the previous list of 100; thus Stripe will know to start the next batch of 100 from the customer which comes after this ID.

• The Salesforce connector has a 'Find Records' operation which has a Limit parameter, as well as a Page offset parameter which allows you to set the record to start from (i.e. after the last record from the previous batch)