Looping using subset inside a big batch in Adobe Campaign

 I had many scenarios where I ran a costly query to select some good no of records and then process them further. But when it comes to processing I had to restrict the number of records to be processed in one go. And once the subset is processed, then select from the table again and then process the subset. Something like below where the end activity calls the signal when the complement from Split has more than 0 records to start the whole process again.

But the problem is every time you have to run the costly select process to process a fraction of the records. As a result the overall throughput is decreased heavily.

As an alternate I used a looping through the selected records avoiding selection every time I need to process the subset.

Here the JS Activity after select is dummy one but required. I added a log in it with count. Test activity is also needed to ensure we have the exit path from the loop. In the above example the select query fetched 8680 records and then split restricted to 1000 records. Thus the loop ran 9 times as below

Note the and activity needs to have the complement from subset as primary set to ensure we look through the remaining records

Drawback:

In scenario where more priority records are injected all the time, there this process is not helpful. For example, I am running a big batch of records and the batch contains those that need NRT emails and rests are once in a day. So when the workflow selected the records, it will continue processing all the records before it selects the next set of records. As a result the NRT records will have to wait until the whole running batch is complete before it is picked up.