Using Microsoft Flow to Process a Lot of CDS Records

How well is Microsoft Flow suited for processing a lot of records in the Common Data Service? Probably it depends on what you consider to be a lot. It used to be that the most number of records that flow could retrieve using the CDS Trigger was 512. That has changed with the introduction of the Settings feature – Pagination. If you want to dig deeper into the details of Pagination then check out fellow Business Applications MVP and Regional Director David Yack’s post – Using Flow CDS List Records action with > 512 records

I decided to put it to the test and see what the run rate for a Flow that was updating CDS records would deliver.

Here is the experiment:

As he identified in that blog post you can return up to 100,000 records by manually setting the Pagination in the Settings area. You can see below that I have set it to 15,000 records.

To test this out I created a simple flow that I could kick off using a Button trigger (instant flow). It would record the time it started, then list all the Account records (there are 15,000) in my default environment. Record the time when that action was completed and them move to the next action step which is to loop through the records and update the name of the Account record and insert the current time into the Account Number field. Then record the time the flow was totally complete.

NOTE: I did not make any changes to Concurrency setting for the Apply to each action, that will be a separate test.

Here is a look at the performance results along with a picture of the actual Flow.

• Flow Start – 2019-03-14T01:47:11.5349534Z
• All Account Records Retrieved – 2019-03-14T01:48:57.2503579Z   —   elapsed time 1.76 minutes
• All Account records updated and flow is complete – 2019-03-14T04:35:51.4067870Z – elapsed time 168.66 minutes

Flow was able to retrieve 15,000 records and update them at a rate of 88.93 records/minute. Not exactly blazing speed but still a very credible performance rate that would work for a lot of business scenarios. And there were no failed actions.

Another NOTE: This is just a simple update action, update two text fields. There isn’t any parenting going on or any new record creation. It is also running in one of our Sandbox environments. It was run after ‘regular business hours’ in the USA Central Time Zone. Your results may be totally different.