What is the relationship between XDR and Datalake

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements
Please sign in to see details of an important advisory in our Customer Advisories area.

What is the relationship between XDR and Datalake

L2 Linker

I saw that there is datalake in the official admin guide structure, but I don't know what datalake does. I bought 200 XDR pro licence and found that my account has more datalake, which has 1TB of storage space. I am very confused. What data is it used to store? I didn't buy a per TB license. Actually my XDR has some data, but I don't see it in datalake

1 accepted solution

Accepted Solutions

L3 Networker

Hi @Grady 

Cortex Data Lake is a PANW service for keeping logs. Cortex XDR is also using benefit of Cortex Data Lake and keeping telemetry data on CDL for further analyses. Your XDR license is also including CDL license but this usage on CDL is limited to XDR telemetry datas. 

if you want to keep and ingest other type of datas with XDR datas,(Firewall logs, windows event logs, Cloud provider logs, etc), You need to purchase XDR Pro per TB license. 

You cannot see XDR data directly from CDL. but you can search XDR data by using XQL queries from Investigation > Query Builders.

View solution in original post

5 REPLIES 5

L5 Sessionator

Hi @Grady , XDR agents send data to XDR's own datalake. The sizing recommendation is 1 TB for every 200 endpoints (see sizing calculator here).
Here are a few points for your understanding:

1. The data in XDR's datalake stores logs, alerts, incidents etc. for XDR only. This data is isolated and does not show in the Cortex Data Lake which you see in Palo Alto Networks App Hub

2. The Cortex Datalake (CDL) you see in App Hub is used for storing logs for PANW firewalls and Prisma Access. As you don't have Pro per TB, I assume that you do not have PANW NGFW's in your portfolio. If you had it, you would be able to scroll in CDL and observe you have firewall/Prisma Access logs only. See screenshot below which lists the type of data present in CDL.

bbarmanroy_0-1647419338012.png

 

3. However, if your CDL is setup to receive firewall and Prisma logs, they will show up in XDR as well. 

4. If you don't have Pro per TB, you should not see a Cortex Data Lake app enabled in your Palo Alto Networks App Hub.

Hope this clarifies.

L3 Networker

Hi @Grady 

Cortex Data Lake is a PANW service for keeping logs. Cortex XDR is also using benefit of Cortex Data Lake and keeping telemetry data on CDL for further analyses. Your XDR license is also including CDL license but this usage on CDL is limited to XDR telemetry datas. 

if you want to keep and ingest other type of datas with XDR datas,(Firewall logs, windows event logs, Cloud provider logs, etc), You need to purchase XDR Pro per TB license. 

You cannot see XDR data directly from CDL. but you can search XDR data by using XQL queries from Investigation > Query Builders.

Hi Bbarmanroy

 

Thank you for your explanation, but there are still some doubts. I understand that the license of 200 pro can store data for 30 days. Does the 30-day data belong to the range of 1TB? What happens if you exceed 1TB

Hi Etugriceri

 

Thanks for the explanation, then if I have XDR Pro per TB, I can see firewall logs in datalake, right? For example, can forinet logs be quota in datalake?

Yes. totally correct. Panw firewall can able to send logs directly to CDL. for any other type of logs, you are able use BrokerVM , XDR collector, XDR API and FileBeat. 

if log type does not known by XDR, You need to write your parser rules on XDR management console and last things, logs will be searchable by using XQL and will be ingested 

 

XDR log rotation date is 30 days. You don't need to concern if you exceed 1TB.  This is SAAS service. Please check you average daily log size from XDR Management console > Configurations > Dataset management. Filter out "dataset name = xdr_data"

if your license is not enough for keeping 30days xdr data, you may choose to increase your license. But sizing calculator is pretty accurate. 

  • 1 accepted solution
  • 3466 Views
  • 5 replies
  • 0 Likes
Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!

The LIVEcommunity thanks you for your participation!