Elasticsearch integration events return limit

[SergioPalacios]

Hello everyone!

I am currently using the Elasticsearch integrations to retrieve events related to an incident or events for a specific report and generally have no issues with that. However, sometimes some "reports" have queries that retrieve +10k events.

Looking at the Elasticsearch integration, I can see that the maximum event count limit is 10k and I'm wondering if there is an "easy" solution to this.

I know that the Elasticserach API, when you need to search for more than 10k events, gives you a scroll_id that you can do a second query to and retrieve the rest of the events, but I haven't seen anything about this parameter or situation in the integration options by Elasticsearch

Is there a solution for this without developing a custom script based on the same Elasticsearch integration?

Thanks in advance.

[jfernandes1 · Accepted solution]

Hi @SergioPalacios,

Did you try the "page" option.

You can run the search command multiple times till the returned size is less than 10k. Please note that we not recommend storing large amounts of data inside the context. If not converted automatically, please save this as a file. 

[SergioPalacios]

Hi @jfernandes1!

Thanks for your reply. It was very helpful for the purpose of the post.

However, I would like to know if there is a way to create a loop to fetch all events to the end. I mean, I just created a built-in loop in a subplaybook that increments the page from 0 to 2. This method allows me to double search and retrieve 20k events, but what if there were 50k waiting to be retrieved?

My method allows to search, introducing a static parameter, those events. However, in the worst case, if instead of 50k, there were 20k, you would be making 3 queries that will not be efficient since there will be no results.
In the same way that if there were 100k events (instead of 50k), we would be "losing" events in the report.