What is the relationship between XDR and Datalake

[Grady]

I saw that there is datalake in the official admin guide structure, but I don't know what datalake does. I bought 200 XDR pro licence and found that my account has more datalake, which has 1TB of storage space. I am very confused. What data is it used to store? I didn't buy a per TB license. Actually my XDR has some data, but I don't see it in datalake

[bbarmanroy]

Hi @Grady , XDR agents send data to XDR's own datalake. The sizing recommendation is 1 TB for every 200 endpoints (see sizing calculator here).
Here are a few points for your understanding:

1. The data in XDR's datalake stores logs, alerts, incidents etc. for XDR only. This data is isolated and does not show in the Cortex Data Lake which you see in Palo Alto Networks App Hub. 

2. The Cortex Datalake (CDL) you see in App Hub is used for storing logs for PANW firewalls and Prisma Access. As you don't have Pro per TB, I assume that you do not have PANW NGFW's in your portfolio. If you had it, you would be able to scroll in CDL and observe you have firewall/Prisma Access logs only. See screenshot below which lists the type of data present in CDL.

3. However, if your CDL is setup to receive firewall and Prisma logs, they will show up in XDR as well. 

4. If you don't have Pro per TB, you should not see a Cortex Data Lake app enabled in your Palo Alto Networks App Hub.

Hope this clarifies.

[etugriceri · Accepted solution]

Hi @Grady 

Cortex Data Lake is a PANW service for keeping logs. Cortex XDR is also using benefit of Cortex Data Lake and keeping telemetry data on CDL for further analyses. Your XDR license is also including CDL license but this usage on CDL is limited to XDR telemetry datas. 

if you want to keep and ingest other type of datas with XDR datas,(Firewall logs, windows event logs, Cloud provider logs, etc), You need to purchase XDR Pro per TB license. 

You cannot see XDR data directly from CDL. but you can search XDR data by using XQL queries from Investigation > Query Builders.

[Grady]

Hi Bbarmanroy

Thank you for your explanation, but there are still some doubts. I understand that the license of 200 pro can store data for 30 days. Does the 30-day data belong to the range of 1TB? What happens if you exceed 1TB

[Grady]

Hi Etugriceri

Thanks for the explanation, then if I have XDR Pro per TB, I can see firewall logs in datalake, right? For example, can forinet logs be quota in datalake?

[etugriceri]

Yes. totally correct. Panw firewall can able to send logs directly to CDL. for any other type of logs, you are able use BrokerVM , XDR collector, XDR API and FileBeat. 

if log type does not known by XDR, You need to write your parser rules on XDR management console and last things, logs will be searchable by using XQL and will be ingested 

XDR log rotation date is 30 days. You don't need to concern if you exceed 1TB.  This is SAAS service. Please check you average daily log size from XDR Management console > Configurations > Dataset management. Filter out "dataset name = xdr_data"

if your license is not enough for keeping 30days xdr data, you may choose to increase your license. But sizing calculator is pretty accurate.