Parsing Rule and Data Model Rule

[A.Velusamy]

I'm new to Cortex XSIAM..Wanted to understand how effectively parsing rule and Data model rule can be used for a particular data source and how it works?

[susekar · Accepted solution]

Hello @A.Velusamy ,

Greetings for the day.

In Cortex XSIAM, Parsing Rules and Data Model Rules are two distinct mechanisms used to manage and normalize data from third-party sources. Although both rely on Cortex Query Language (XQL), they operate at different stages of the data lifecycle and serve different purposes.

 

1. Parsing Rules (Ingest-Time)

Parsing rules are applied during the data ingestion phase, before logs are stored in a dataset. Their primary purpose is to clean, filter, and structure raw incoming data.

How They Work

Parsing rules use a subset of XQL known as XQLi (XQL for Ingestion). They process raw log input, perform transformations, and insert the processed results into a specified dataset.

Key Functions

Filtering and Cost Reduction

You can use the filter command within an [INGEST] rule to drop unnecessary logs (for example, informational traffic that provides no security value). This helps reduce storage and ingestion costs.

Field Extraction

Parsing rules extract fields from raw messages (such as Syslog strings) so they are stored as structured columns within the dataset.

Timestamp Manipulation

Normalizing the _time field during ingestion ensures logs are properly ordered and searchable.

Effectiveness and Constraints

• Parsing rules only apply to new logs ingested after the rule is activated. They do not modify historical data.

• They improve performance because data is pre-parsed and structured before being stored.

 

2. Data Model Rules (Query-Time)

Data Model Rules operate at query time and provide a normalization layer. They map fields from vendor-specific datasets (for example, fortinet_fortigate_raw) to the standardized Cortex Data Model (XDM).

How They Work

Data Model Rules act as a logical “view” on top of your datasets. When you run a query using the datamodel command, XSIAM applies these rules to translate vendor-specific field names into standardized XDM field names.

Key Functions

Unified Threat Hunting:

Enables cross-vendor searches using standardized XDM fields (for example, searching for an IP address using xdm.source.ipv4 across multiple firewall vendors).

Analytics and Correlation:

Built-in analytics and correlation rules rely on XDM-normalized fields to detect cross-platform threats consistently.

Effectiveness and Constraints:

• Data Model Rules are retroactive, meaning they apply to both historical and newly ingested data.

• If you create a user-defined Data Model Rule for a dataset that already has a default rule, the custom rule completely overrides the default rule for that dataset.

If you feel this has answered your query, please let us know by clicking like and on "mark this as a Solution".

Thanks & Regards,
S. Subashkar Sekar