XSIAM API pagination

[jon_thurston]

Hello,

I'm trying to use the API to pull a lookup dataset that is larger than 10,000 rows. I don't see any options for pagination and the filters listed in the documentation seem too rigid to easily pull the data. 

https://docs-cortex.paloaltonetworks.com/r/Cortex-XSIAM-REST-API/Get-data-from-a-lookup-dataset

Is there any way to paginate the data? Other REST APIs do this without a problem. Is there another way to pull the information other than that API endpoint?

[susekar · Accepted solution]

Hello @jon_thurston ,

Greetings for the day.

Based on the internal information and technical support cases, the specific API endpoint you are using for lookup datasets does have a hard-coded limitation, and standard pagination (like offset/limit) is not natively supported for that specific call.

1. Hard-Coded Retrieval Limit:

The endpoint /public_api/v1/xql/lookups/get_data/ (often used via the command !xdr-xql-lookup-get-data) has a built-in restriction that limits data retrieval to a maximum of 10,000 rows. This is a design constraint, and attempting to pull more data through this specific method will typically result in truncated results or errors.

2. Alternative: Use XQL Query and Stream APIs:

To retrieve more than 10,000 rows or to implement pagination, you should use the standard XQL Query API instead of the specialized lookup retrieval endpoint. Lookup tables can be queried directly using Cortex Query Language (XQL).

Steps to Pull Large Datasets:

1. Start an XQL Query:

Use the /public_api/v1/xql/start_xql_query/ endpoint. In your query string, target your lookup dataset directly:

dataset = <yourlookupdataset_name>

Note: Ensure the dataset name is in lowercase, as the API can be case-sensitive for table names.

2. Retrieve Results with Stream ID:

The initial response from the XQL API is limited to 1,000 results. However, if the query generates more than 1,000 results, the API response will include a unique stream_id.

3. Paginate via Stream API

Use the /public_api/v1/xql/get_query_results_stream/ endpoint with the provided stream_id to fetch the remaining data in chunks. This method supports up to 1,000,000 results per query.

3. Summary of Limitations:

• Lookup-specific API: Limited to 10,000 rows; no pagination.

• XQL Query API: Limited to 1,000 rows per call but supports pagination via stream_id for up to 1,000,000 rows.

• Rate Limits: 10 requests per second per tenant.

If your workflow requires increasing the hard-coded 10,000-row limit for the /lookups/get_data/ endpoint itself, you must contact your account team or open a support case to request an engineering override for your tenant, as this cannot be changed via standard configuration.

If you feel this has answered your query, please let us know by clicking like and on "mark this as a Solution".

Thanks & Regards,
S. Subashkar Sekar