Custom Parsing Rule - Cohesity

[Jesse_Siegrist]

This was a fun project. Looks like it is mostly working correctly. Cohesity syslogs come in as a big blob in one field so I messed with some parsing rules to give them their own datasets

The only known issue I'm seeing so far is the logs get duplicated into the cohesity "raw" dataset at the end.. not sure how to fix that quite yet.

[aspatil]

Hello @Jesse_Siegrist ,

The logs are correctly parsed into cohesity_backups_data_protection, cohesity_backups_api, and cohesity_backups_parsed, but they are still appearing in cohesity_backups_raw, leading to duplication.

This happened due to below:
[INGEST:vendor="Cohesity", product="Backups", target_dataset="cohesity_backups_raw", no_hit=keep]

The issue is likely that all logs are still passing through this rule after the other extractions, meaning any unfiltered logs (including already parsed logs) get dumped into cohesity_backups_raw.The no_hit=drop parameter should prevent logs from duplicating if they are correctly filtered in previous parsing steps, but it’s possible that some logs are still unaccounted for.

Check whether attached file helps:

[Jesse_Siegrist]

Ashutosh,

  Do you know what the actual behavior of "drop" is? I was hesitant to use that because I was concerned if it did not match the first filter, then the log would be discarded completely rather than moving on to the next condition. 

I asked support about it and they tried to tell me I could use 'next' rather than 'keep' or 'drop'.. which was a lie 😆

[aspatil]

• drop takes a condition similar to the XQL filter stage (same syntax), but drops every log entry that passes that condition. One can think of it as a negative filter, so drop <condition> is not equivalent to filter not <condition>.

• If no_hit = drop, then in a scenario where none of the rules in the group generates output for a given log record, that record is discarded.

https://docs-cortex.paloaltonetworks.com/r/Cortex-XDR/Cortex-XDR-Documentation/INGEST

Let me know if the provided parsing rule works.

If you feel this has answered your query, please let us know by clicking like and  on "mark this as a Solution". Thank you.

[Jesse_Siegrist]

Ashutosh,

   I think we're close here. "in a scenario where none of the rules in the group generates output for a given log record, that record is discarded"

That is the crux of the issue as I see it. So lets consider the journey of a single log. 
Is it like.. scenario 1: "check condition one, no match... check condition two.. no match -> check final condition no match -> discard log"
OR
Is it scenario 2:  "Check condition one, no match -> discard log.. no further conditions checked"  <- this is the situation I'm afeared of. 

And in the example of a match being found.. is the behavior like "Check condition 1, no match.. check condition two, match.. log gets put into condition two dataset.. no further rules processed". or will it continue to process rules after the first match?