Data Recording From REST API Get Recording is not consistent

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements

Welcome Idira to LIVEcommunity!

Our community just got stronger! Explore new identity security resources, connect with experts, and discover more ways to learn and collaborate.

Meet Idira

Data Recording From REST API Get Recording is not consistent

L2 Linker

Dear All,

 

I retrieve data of recording via REST API (Get Recording).

First tried compare to Second tried are different.

The result of First tried:

Field_Content_versions_oct20__068Vy00000Wnu0FIAR.png 

The result of Second tried :

Field_Content_versions_oct20__068Vy00000Wn7KnIAJ.pngIf we check on the result of second tried, recording with start 03/01/2025 06:08:00 AM and 03/01/2025 06:11:23 AM (Yellow Background) are Not Exist in the result of First tried.

I have checked the parameter of MaxRecords on PSM (via PVWA) is 8000 and i use data retrieve limitation 1000 in my source code, but actually the amount of data that be retrieved only under 200 data.

 

Additional information : Source code to retrieve data from Cyberark REST API using Pyhton. Cyberark version is 12.6.

 

Please any advice. Thanks

 

6 REPLIES 6

L4 Transporter

Odd. I notice that the two missing recordings are very long sessions - started on 1 March 6am and ended on 2 March 3am - around 21 hours each.

 

My first question; is it possible that you ran the first query before those two sessions ended? I guess not since that was two months ago, but perhaps we can just confirm it.

 

Secondly perhaps there was a delay in uploading the recording from PSM into the Vault due to the unusual size of the recordings? so that the recording may have finished on 2 March but the PSM did not successfully upload to the Vault until you were running the query.

 

What parameters were passed to the API call?

L2 Linker

Dear @Jon Musker​ 

 

Yes. the remote session is in march, 1 2025. we have confirm to user that they did monitoring. So that why the remote session is 75310 second (20 Hour 10 Minute). And the report is generate on April 2025.

 

Actually there are another several different data between First tried and Second tried.

Result of First Tried :

Field_Content_versions_oct20__068Vy00000X6bMrIAJ.png 

Result of Second Tried :

Field_Content_versions_oct20__068Vy00000X6LEwIAN.png 

We can see above. The data that not show in first tried (yellow background) have remote session duration only under 9000 second.

 

This is my code to retrieve recording data via REST API (using python):

def recordings(url, token, timeStampStartDate, timeStampEndDate, limit, offset, sort):

   requests.packages.urllib3.disable_warnings(InsecureRequestWarning)

   headers = {'Authorization': token, 'Content-Type': 'application/json'}

   params = {"FromTime": timeStampStartDate, "ToTime": timeStampEndDate,

             "limit": limit, "offset": offset, "sort": sort}

   response = requests.get(url, params=params, headers=headers, verify=False)

   statusCode = response.status_code

   if statusCode == 200:

       responseApi = response.json()

       return ([statusCode, responseApi])

   else:

       return ([statusCode, response])

 

=====================================================

 

This how I call the function to retrieve:

responseApi = recordings(url,token,int(start_dt.timestamp()),int(until_dt.timestamp()),limit,offset,sort)  

 

well that is definitely weird, the same call should produce the same result set

 

Not sure what that python library is doing with datetime conversions - is it possible that the timestamps are being translated into different localisation? In US locale 3/1/2025 is March 1st, but in most European local it is 3rd January..? Probably not relevant if you are testing from the same machine each time.

 

Or perhaps a pagination issue - your code doesn't seem to have any loop to handle retrieving the next page of data so it must be the calling function that is responsible for handling this. Perhaps the function that calls recordings() is not looping correctly to handle pagination - limit and offset are the important values here. We start with offset=0 and a set page size such as limit=100. If the first call returns 100 records (ie the value of limit) then set offset=100 and call the API again. Keep repeating adding 100 (limit) to offset each time, until the number of records returned is less than 100 (limit). Then you know you have retrieved all the records. If you aren't properly setting limit and offset in the function that calls recordings() then the results could be a bit random.

Dear @Jon Musker​ 

 

first of all thanks for response.

 

The information that I can share are limit = 1000 , if we set time range 1 month (30 days) it will loop 30 time send API request.

So looping is based on 1 day. And the data in 1 day less than 1000 (around 200 to 500 data).

Why I loop per day, because to avoid MaxRecords parameter if the data that will be retrieve is more that MaxRecords.

I have tried the source code to Dev Environment many time. And the source code run smoothly. The data in dev environment only less than 200.

But when I run to production environment, the problem happen. The data in production environment in 1 month around 10.000 data.

 

Does Cyberark take data from the database or from the log file when generating JSON?

Looks like there is a certain process that lock the data that make it is locked and can not be retrieved.

 

 

try removing the sort. If you are just retrieving all records then the order you retrieve them will be largely irrelevant; I guess it is possible that it is confusing the issue in between the calls, since each api call is stateless - it's not as if the system has a frozen dataset that you are iterating over. After you have retrieved all the records to the app, sorting them in memory will be trivial.

 

I don't think you should assume that each day will have less than 1000 records; limit/offset should be setup to work reliably and then just request the actual date range (ie one month). Breaking it down arbitrarily into one day at a time seems like you don't trust the API, but you must trust your code otherwise it isn't ready to release! Once you have it correct it should work properly whether the limit is 5 or 5000, the only reason to vary limit would be for performance. Also when searching by date range in any complex database there is always a risk of edge cases near the start and end of the range (what time zone, what counts as start or end time etc). I particularly note that you say you are reading data in one day at a time, but at least two of the 'weird' records are recordings that span two different days.

 

Make sure you have enough error handling; maybe add some trace logging to your app to see what is happening on each loop (how many records returned, what value for limit/offset is used on each pass)

 

Data should be coming from the database rather than logs afaik, it involves joining multiple tables to get all the session data.

L4 Transporter

Hello @Krida Kusuma​ 

 

Have you received an answer to your question? If so, we kindly encourage you to share it with us.

Your insights could be valuable to other community members in the future.

 

Thank you, Isar.

  • 160 Views
  • 6 replies
  • 0 Likes
Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!

The LIVEcommunity thanks you for your participation!