Requesting Clarity on XDR XQL API Logging

cancel
Showing results for 
Show  only  | Search instead for 
Did you mean: 
Announcements
Please sign in to see details of an important advisory in our Customer Advisories area.

Requesting Clarity on XDR XQL API Logging

L1 Bithead

Hello Everyone, 

 

For one of the client, we need to fetch logs from XDR API using XQL. Currently, the ask is for windows event logs only, but later they want IIS logs as well. 

Any help in below queries would be appreciated:

 

1. There are two queries by which I fetched logs successfully. One is using:

dataset= xdr_data | filter event_type = EVENT_LOG

Second query I tried is using preset:

preset = xdr_event_log
By both of these queries I'm getting event logs. I would like to know if both the queries are correct and I'm not missing on anything. 
If both the queries are correct, which would be efficient one?
 
2. Second query is regarding schema. I have gone through XDR_DATA schema. I just want to clarify if agent_hostname field's value changes each endpoint or is it same? This may sound stupid but basically we want to know which endpoint logged this event and if we can be sure that each agent on each endpoint has separate hostname. 
 
3. In future we need to fetch IIS logs as well. I could not find any document or help regarding this. Which dataset or preset is used for IIS logs? Does XQL query even work for IIS or it logs to CDL? 
 
Any help would be great. 
Thanks in advance. 
1 accepted solution

Accepted Solutions

L2 Linker

Hi @sushant1601 ,

Thank you for posting these questions, please find below my response to each of them:
1. You can query against either datasets or Presets in a dataset query ; however, presets offer groupings of xdr_data fields that are useful for analyzing specific areas of network and endpoint activity. All of the fields available for a preset are also available on the larger xdr_data dataset, but by using the preset your query can run more efficiently
For that, I would also recommend going over our newly published detailed training for XQL, the full course can be found here, and for the data sources part that is part of it , it can be also found under the title "2. XQL Building block: Data Source & Structure ". 

2. The field "agent_hostname" will operate the same in both the "preset = xdr_event_log" and "dataset = xdr_data", and yes, the hostname should remain the same in the query regardless when/ how you query the result, the value of the field should not changed unless the hostname itself changes. 

3. Yes, you can query IIS data ingested to XDR using XQL. Cortex XDR provides YAML templates for DHCP, DNS, IIS, and NGINX. This can be achieved by configuring an XDR Collector Profile for Windows. Then, to query the IIS logs, you can use XQL to query "dataset = iis_iis_raw" for the raw logs ingested to XDR, as well as the parsed logs if you happen to create a parsing rule for the raw logs, an example for doing so is below 
[INGEST:vendor = "iis", product = "iis", target_dataset ="microsoft_iis_parsed", no_hit = keep]
alter date = arrayindex(regextract(_raw_log, "[^\s]+"),0),
time = arrayindex(regextract(_raw_log, "[^\s]+"),1),
s_ip = arrayindex(regextract(_raw_log, "[^\s]+"),2),
cs_method = arrayindex(regextract(_raw_log, "[^\s]+"),3),
cs_uri_stem = arrayindex(regextract(_raw_log, "[^\s]+"),4),
cs_uri_query = arrayindex(regextract(_raw_log, "[^\s]+"),5),
s_port = arrayindex(regextract(_raw_log, "[^\s]+"),6),
cs_username = arrayindex(regextract(_raw_log, "[^\s]+"),7),
c_ip = arrayindex(regextract(_raw_log, "[^\s]+"),8),
cs_user_agent = arrayindex(regextract(_raw_log, "[^\s]+"),9),
cs_referer = arrayindex(regextract(_raw_log, "[^\s]+"),10),
sc_status = arrayindex(regextract(_raw_log, "[^\s]+"),11),
sc_substatus = arrayindex(regextract(_raw_log, "[^\s]+"),12),
sc_win32_status = arrayindex(regextract(_raw_log, "[^\s]+"),13),
time_taken = arrayindex(regextract(_raw_log, "[^\s]+"),14);

If you have this configured, then you cna query the dataset "dataset = microsoft_iis_parsed", and the name of the dataset is coming from the parsing rule first line seen above "[INGEST:vendor = "iis", product = "iis", target_dataset ="microsoft_iis_parsed", no_hit = keep]"

In addition, and as partially mentioned above, we have published a detailed training for XQL , with 2 parts, i will link theme below here as well, please feel free to take advantage of those and let me know if you happen to have any questions.

Cortex XDR Basic XQL Crash Course
Cortex XDR XQL Use Cases and Applications Crash Course

I hope the above was helpful and answered your questions, please let me know if any! 
Thanks
Z


Z

View solution in original post

4 REPLIES 4

L2 Linker

Hi @sushant1601 ,

Thank you for posting these questions, please find below my response to each of them:
1. You can query against either datasets or Presets in a dataset query ; however, presets offer groupings of xdr_data fields that are useful for analyzing specific areas of network and endpoint activity. All of the fields available for a preset are also available on the larger xdr_data dataset, but by using the preset your query can run more efficiently
For that, I would also recommend going over our newly published detailed training for XQL, the full course can be found here, and for the data sources part that is part of it , it can be also found under the title "2. XQL Building block: Data Source & Structure ". 

2. The field "agent_hostname" will operate the same in both the "preset = xdr_event_log" and "dataset = xdr_data", and yes, the hostname should remain the same in the query regardless when/ how you query the result, the value of the field should not changed unless the hostname itself changes. 

3. Yes, you can query IIS data ingested to XDR using XQL. Cortex XDR provides YAML templates for DHCP, DNS, IIS, and NGINX. This can be achieved by configuring an XDR Collector Profile for Windows. Then, to query the IIS logs, you can use XQL to query "dataset = iis_iis_raw" for the raw logs ingested to XDR, as well as the parsed logs if you happen to create a parsing rule for the raw logs, an example for doing so is below 
[INGEST:vendor = "iis", product = "iis", target_dataset ="microsoft_iis_parsed", no_hit = keep]
alter date = arrayindex(regextract(_raw_log, "[^\s]+"),0),
time = arrayindex(regextract(_raw_log, "[^\s]+"),1),
s_ip = arrayindex(regextract(_raw_log, "[^\s]+"),2),
cs_method = arrayindex(regextract(_raw_log, "[^\s]+"),3),
cs_uri_stem = arrayindex(regextract(_raw_log, "[^\s]+"),4),
cs_uri_query = arrayindex(regextract(_raw_log, "[^\s]+"),5),
s_port = arrayindex(regextract(_raw_log, "[^\s]+"),6),
cs_username = arrayindex(regextract(_raw_log, "[^\s]+"),7),
c_ip = arrayindex(regextract(_raw_log, "[^\s]+"),8),
cs_user_agent = arrayindex(regextract(_raw_log, "[^\s]+"),9),
cs_referer = arrayindex(regextract(_raw_log, "[^\s]+"),10),
sc_status = arrayindex(regextract(_raw_log, "[^\s]+"),11),
sc_substatus = arrayindex(regextract(_raw_log, "[^\s]+"),12),
sc_win32_status = arrayindex(regextract(_raw_log, "[^\s]+"),13),
time_taken = arrayindex(regextract(_raw_log, "[^\s]+"),14);

If you have this configured, then you cna query the dataset "dataset = microsoft_iis_parsed", and the name of the dataset is coming from the parsing rule first line seen above "[INGEST:vendor = "iis", product = "iis", target_dataset ="microsoft_iis_parsed", no_hit = keep]"

In addition, and as partially mentioned above, we have published a detailed training for XQL , with 2 parts, i will link theme below here as well, please feel free to take advantage of those and let me know if you happen to have any questions.

Cortex XDR Basic XQL Crash Course
Cortex XDR XQL Use Cases and Applications Crash Course

I hope the above was helpful and answered your questions, please let me know if any! 
Thanks
Z


Z

L1 Bithead

Hi @zarnous 

Thank you so much for explaining in details. Really appreciate your response.

 

Just last small query. In the second point you mentioned the value of the field should not be changed unless the hostname itself changes. So, is agent hostname is same as of hostname of the endpoint?

Clearing this because we are working to ingest the logs and we do not have the device with us. So, I can not verify this on the device itself.

Again, thank you for the response!

L2 Linker

Hi @sushant1601 ,

Sure i will be happy to clarify this more, Yes that is correct the field  "agent_hostname" in both the "preset = xdr_event_log" and "dataset = xdr_data", is the hostname of the endpoint .

Please feel free to let me know if you have any follow up question , happy to help anytime!
Thanks
Z


Z

L1 Bithead

Thank you so much @zarnous 

  • 1 accepted solution
  • 1395 Views
  • 4 replies
  • 0 Likes
Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the LIVEcommunity as a whole!

The LIVEcommunity thanks you for your participation!