Requesting Clarity on XDR XQL API Logging

[sushant1601]

Hello Everyone, 

For one of the client, we need to fetch logs from XDR API using XQL. Currently, the ask is for windows event logs only, but later they want IIS logs as well. 

Any help in below queries would be appreciated:

1. There are two queries by which I fetched logs successfully. One is using:

dataset= xdr_data | filter event_type = EVENT_LOG

Second query I tried is using preset:

preset = xdr_event_log

By both of these queries I'm getting event logs. I would like to know if both the queries are correct and I'm not missing on anything. 

If both the queries are correct, which would be efficient one?

2. Second query is regarding schema. I have gone through XDR_DATA schema. I just want to clarify if agent_hostname field's value changes each endpoint or is it same? This may sound stupid but basically we want to know which endpoint logged this event and if we can be sure that each agent on each endpoint has separate hostname. 

3. In future we need to fetch IIS logs as well. I could not find any document or help regarding this. Which dataset or preset is used for IIS logs? Does XQL query even work for IIS or it logs to CDL? 

Any help would be great. 

Thanks in advance. 

Cortex XDR 

[zarnous · Accepted solution]

Hi @sushant1601 ,

Thank you for posting these questions, please find below my response to each of them:
1. You can query against either datasets or Presets in a dataset query ; however, presets offer groupings of xdr_data fields that are useful for analyzing specific areas of network and endpoint activity. All of the fields available for a preset are also available on the larger xdr_data dataset, but by using the preset your query can run more efficiently
For that, I would also recommend going over our newly published detailed training for XQL, the full course can be found here, and for the data sources part that is part of it , it can be also found under the title "2. XQL Building block: Data Source & Structure ". 

2. The field "agent_hostname" will operate the same in both the "preset = xdr_event_log" and "dataset = xdr_data", and yes, the hostname should remain the same in the query regardless when/ how you query the result, the value of the field should not changed unless the hostname itself changes. 

3. Yes, you can query IIS data ingested to XDR using XQL. Cortex XDR provides YAML templates for DHCP, DNS, IIS, and NGINX. This can be achieved by configuring an XDR Collector Profile for Windows. Then, to query the IIS logs, you can use XQL to query "dataset = iis_iis_raw" for the raw logs ingested to XDR, as well as the parsed logs if you happen to create a parsing rule for the raw logs, an example for doing so is below 
[INGEST:vendor = "iis", product = "iis", target_dataset ="microsoft_iis_parsed", no_hit = keep]
alter date = arrayindex(regextract(_raw_log, "[^\s]+"),0),
time = arrayindex(regextract(_raw_log, "[^\s]+"),1),
s_ip = arrayindex(regextract(_raw_log, "[^\s]+"),2),
cs_method = arrayindex(regextract(_raw_log, "[^\s]+"),3),
cs_uri_stem = arrayindex(regextract(_raw_log, "[^\s]+"),4),
cs_uri_query = arrayindex(regextract(_raw_log, "[^\s]+"),5),
s_port = arrayindex(regextract(_raw_log, "[^\s]+"),6),
cs_username = arrayindex(regextract(_raw_log, "[^\s]+"),7),
c_ip = arrayindex(regextract(_raw_log, "[^\s]+"),8),
cs_user_agent = arrayindex(regextract(_raw_log, "[^\s]+"),9),
cs_referer = arrayindex(regextract(_raw_log, "[^\s]+"),10),
sc_status = arrayindex(regextract(_raw_log, "[^\s]+"),11),
sc_substatus = arrayindex(regextract(_raw_log, "[^\s]+"),12),
sc_win32_status = arrayindex(regextract(_raw_log, "[^\s]+"),13),
time_taken = arrayindex(regextract(_raw_log, "[^\s]+"),14);

If you have this configured, then you cna query the dataset "dataset = microsoft_iis_parsed", and the name of the dataset is coming from the parsing rule first line seen above "[INGEST:vendor = "iis", product = "iis", target_dataset ="microsoft_iis_parsed", no_hit = keep]"

In addition, and as partially mentioned above, we have published a detailed training for XQL , with 2 parts, i will link theme below here as well, please feel free to take advantage of those and let me know if you happen to have any questions.

Cortex XDR Basic XQL Crash Course
Cortex XDR XQL Use Cases and Applications Crash Course

I hope the above was helpful and answered your questions, please let me know if any! 
Thanks
Z

[sushant1601]

Hi @zarnous 

Thank you so much for explaining in details. Really appreciate your response.

Just last small query. In the second point you mentioned the value of the field should not be changed unless the hostname itself changes. So, is agent hostname is same as of hostname of the endpoint?

Clearing this because we are working to ingest the logs and we do not have the device with us. So, I can not verify this on the device itself.

Again, thank you for the response!

[zarnous]

Hi @sushant1601 ,

Sure i will be happy to clarify this more, Yes that is correct the field  "agent_hostname" in both the "preset = xdr_event_log" and "dataset = xdr_data", is the hostname of the endpoint .

Please feel free to let me know if you have any follow up question , happy to help anytime!
Thanks
Z

[sushant1601]

Thank you so much @zarnous