Elasticsearch integration events return limit

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements

Elasticsearch integration events return limit

L1 Bithead

Hello everyone!

 

I am currently using the Elasticsearch integrations to retrieve events related to an incident or events for a specific report and generally have no issues with that. However, sometimes some "reports" have queries that retrieve +10k events.

 

Looking at the Elasticsearch integration, I can see that the maximum event count limit is 10k and I'm wondering if there is an "easy" solution to this.

 

I know that the Elasticserach API, when you need to search for more than 10k events, gives you a scroll_id that you can do a second query to and retrieve the rest of the events, but I haven't seen anything about this parameter or situation in the integration options by Elasticsearch

 

Is there a solution for this without developing a custom script based on the same Elasticsearch integration?

 

Thanks in advance.

1 accepted solution

Accepted Solutions

L5 Sessionator

Hi @SergioPalacios,

 

Did you try the "page" option.Screenshot 2023-07-17 at 10.40.36 am.png

 

You can run the search command multiple times till the returned size is less than 10k. Please note that we not recommend storing large amounts of data inside the context. If not converted automatically, please save this as a file. 

View solution in original post

2 REPLIES 2

L5 Sessionator

Hi @SergioPalacios,

 

Did you try the "page" option.Screenshot 2023-07-17 at 10.40.36 am.png

 

You can run the search command multiple times till the returned size is less than 10k. Please note that we not recommend storing large amounts of data inside the context. If not converted automatically, please save this as a file. 

Hi @jfernandes1!

Thanks for your reply. It was very helpful for the purpose of the post.

 

However, I would like to know if there is a way to create a loop to fetch all events to the end. I mean, I just created a built-in loop in a subplaybook that increments the page from 0 to 2. This method allows me to double search and retrieve 20k events, but what if there were 50k waiting to be retrieved?

 

My method allows to search, introducing a static parameter, those events. However, in the worst case, if instead of 50k, there were 20k, you would be making 3 queries that will not be efficient since there will be no results.
In the same way that if there were 100k events (instead of 50k), we would be "losing" events in the report.

  • 1 accepted solution
  • 1193 Views
  • 2 replies
  • 0 Likes
Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!

The LIVEcommunity thanks you for your participation!