XSIAM Dashboard

[A.Velusamy]

Hi, I'm working on creating a dashboard for the concept below. Has anyone already tried this or have any insights they can share?

sudden spike for data ingestions

Data ingestion exceeded threshold

Data source with correlation rules per source

[susekar · Accepted solution]

Hello @A.Velusamy ,

Greetings for the day.

Yes, several users and internal teams have implemented similar monitoring using the metrics_source dataset and the metrics_view preset in Cortex XSIAM. This approach allows you to monitor ingestion spikes, track quota thresholds, and create per-source correlation rules using the following insights and XQL patterns.

1. Built-in Dashboards for Ingestion

Cortex XSIAM provides built-in tools for monitoring ingestion that may already address part of your requirement:

• Data Ingestion Dashboard:
  Located under Dashboards & Reports > Dashboards, this dashboard provides visibility into high-volume data sources and helps identify which sources are contributing to quota overages.

• Command Center:
  Provides a near real-time overview (refreshed approximately every 30 seconds) of data sources, ingestion rates, and prevented events.

2. Identifying Sudden Spikes

To accurately detect ingestion spikes, it is important to distinguish between:

• _time: When the event originally occurred

• _insert_time: When the event was processed by XSIAM

Spikes shown in ingestion dashboards are typically tied to _insert_time, which can indicate backlog processing rather than real-time event surges.

The following XQL pattern can be used to visualize ingestion volume by product:

preset = metrics_view
| bin _time span = 1d
| comp sum(total_size_bytes) as total_in_bytes by _product, _time
| alter total_in_gb = divide(total_in_bytes, 1000000000)
| sort asc _time
| view graph type = line xaxis = _time yaxis = total_in_gb series = _product

3. Monitoring Ingestion Thresholds

To trigger alerts when ingestion reaches a defined percentage of your contracted volume (for example, 90%), you can create a Correlation Rule using the metrics_source dataset.

Example query pattern:

dataset = metrics_source
| comp sum(to_float(total_size_bytes)) as Bytes_ingested by _vendor
| alter total_gbytes = divide(Bytes_ingested, 1073741824)
| filter total_gbytes > [YOUR_THRESHOLD_IN_GB]

Note:
In the correlation rule configuration, the Query Timeframe and Schedule settings must replace any manual timeframe logic in the query itself.

4. Correlation Rules per Data Source

You can also create health monitoring correlation rules to detect when specific data sources stop reporting or exceed expected limits.

• Source Health Monitoring:
  A common approach is to alert when total_event_count_sum = 0 for a specific _collector_name or _product.

• Efficiency Tip:
  Avoid excessive grouping in the comp clause (for example, grouping by collector IP, ID, and hostname simultaneously). Overly granular grouping can cause resource exhaustion and result in false-positive “no data” alerts.

Example query to track ingestion for a specific data source (such as NGFW) by device ID:

preset = metrics_view
| filter _product = "NGFW"
| bin _time span = 1d
| comp sum(total_size_bytes) as total_in_bytes by _device_id, _time
| alter total_in_gb = divide(total_in_bytes, 1000000000)
| sort asc _time

5. Implementation Insights

• Licensing Calculation:
  Ingestion monitoring is typically based on a 7-day rolling average rather than a single-day spike.

• Data Latency Handling:
  If a correlation rule misses events due to ingestion delays, consider increasing the query timeframe (for example, from 1 hour to 2 hours) while keeping the same execution schedule to capture late-arriving data.

• Metric Availability:
  Ensure Data Ingestion Monitoring is enabled under
  Settings > Configurations > General > Server Settings to allow XSIAM to generate built-in ingestion and health alerts.

If you feel this has answered your query, please let us know by clicking like and on "mark this as a Solution".

Thanks & Regards,
S. Subashkar Sekar