XSIAM Parsing Success Rate Metrics

[W.Kishore594287]

Hi Team, 

Is it possible to calculate Parsing Success Rate Metrics  in XSIAM i.e., % of events successfully parsed into SIEM schema.

Regards,

Wincy

[susekar]

Hello @W.Kishore594287 ,

Greetings for the day.

Yes, it is possible to calculate Parsing Success Rate metrics in Cortex XSIAM using XQL queries. While XSIAM does not provide a single out-of-the-box "Parsing Success Rate" metric, you can derive this percentage by querying internal datasets that track ingestion errors and comparing them against total ingestion volumes.

 

Recommended Methods for Calculation:

1. Utilizing the parsing_rules_errors Dataset

Cortex XSIAM tracks explicit parsing failures in the parsing_rules_errors dataset. This table records "Data Format" errors and specific failures encountered when a parsing rule fails to process a log entry.

To calculate the rate, you can compare the count of errors in this table against the total event count from the metrics_source dataset or the relevant target log dataset.

 

2. Analyzing Null Values in Mapped Fields (Log Drift Detection)

For many integrations, if a log fails to parse correctly, the XSIAM ingestion engine may still ingest the record but place the entire content into the _raw_log field while leaving schema-defined fields (such as action, src_ip, etc.) as NULL.

A common best practice for monitoring parsing success and detecting log drift is to generate a report on the percentage of NULL values in key mandatory fields over a specific period. If the log format changes at the source and the parser is no longer compatible, the percentage of NULL values will typically increase significantly.

 

Example XQL Logic:

To calculate the success rate for a specific dataset, you can use logic similar to the following (based on the null-check methodology commonly used for monitoring log drift):

dataset = <your_dataset_name_raw>
| comp 
    count(_id) as total_events, 
    count(xdm.event.type) as parsed_events   // Replace with a key field that should always be parsed
| alter success_rate = (parsed_events * 100.0 / total_events)
| fields total_events, parsed_events, success_rate

In this logic:

• total_events represents all ingested logs.

• parsed_events represents logs where a mandatory parsed field is populated.

• success_rate gives the parsing success percentage.

 

Monitoring Tools in XSIAM

Command Center Dashboard: Provides high-level interactive overviews of system activity and overall ingestion rates, though it may not display granular parsing success percentages by default.

metrics_view Preset: Can be used to monitor daily data ingestion rates and identify periods of unusually low ingestion, which may indicate parsing or collection issues.

Health Alerts: XSIAM generates health alerts for "Data Format" errors or when logs are not collected for an abnormally long period.

For complex environments where logs are ingested through multiple custom parsers, you can aggregate errors from the parsing_rules_errors dataset by vendor and product to identify specific integrations with lower parsing success rates.

If you feel this has answered your query, please let us know by clicking like and on "mark this as a Solution".

Thanks & Regards,
S. Subashkar Sekar