API Pagination

[Phil007]

Hi community, I am new here.

I am trying to integrate the Cortex XDR API for incidents into Azure Sentinel using the new Codeless Connector Platform (CCP).  The challenge I have is that the Cortex API doesn't appear to have any indicator as to where you are up to in the response that is coming back.  It gives the total number of records and the number in the batch but not the number of batches or where you are up to.  With the CCP there are limited options when it comes to pagination and tracking the number of responses does not appear to be something it can do.

I was wondering if anyone has been able to implement pagination in such a way that you don't have to track where you are up to.  I had looked at using the last items date as the start of the next, but because the filter is a "gte" operator that won't work as I will always get stuck on the last item as it will always return something and never return an empty result set.

Any help appreciated.

Phil

[bbarmanroy]

Hi @Phil007 The way I'd try to handle pagination is by iterating through calls based on the total_count parameter. Note that the list of results, 

The search size must fulfill the requirement: 0 < search_size <= 100. So you'll need to iterate through the total_count of incidents, incrementing the starting and ending values of the search_size by 100 with each iteration.

An example of that would be a tenant with 256 incidents (incident values 0-255).

1. The first call to the Get Incidents API will retrieve the following data: total_count=256 and result_count=100, and the first 100 results (0-99) based on the filter. Since the total_count>100, we need to make floor(256/100)=2 additional calls.
2. Start an iteration with the same request_data field with the following values: iterator=1 to num_of_pages, search_from=100*iterator (=100) and search_to=min(100*(iterator+1), total_count)) (=200). Using those values, you'll be able to retrieve the next 100 results (100-199). Increment iterator by 1.
3.  In the next step, iterator=2, search_from=100*iterator (=200) and search_to=min(100*(iterator+1), total_count)) (=256). You'll get the next 56 results (200-255).

I have also provided another example in this post here for you to review and see if it meets your needs.

[Phil007]

Thanks @bbarmanroy , I had seen your post, but unfortunately I don't currently have the ability to track the number of times I have called the API as the CCP is closed code ATM.

[bbarmanroy]

You're welcome @Phil007 . I'm assuming you've already played around with the pageSize and nextPageUrlQueryParameters to see if those can solve the issue.