topic Re: Syslog miner indicator in General Topics

Syslog miner indicator

AlbertoZanon — Sat, 31 Dec 2016 10:03:23 GMT

Hi, i'm trying creating a indicator for SLW injections events , extract src_ip and insert into

Down below the code attached to the syslogminer class stdlib.syslogMiner , unfortunaltely validate process reports errors with the only suggestions "Condition is invalid", could be either a formatting erro or a logical erro, hard to say.

conditions:
    type == 'THREAT'
    log_subtype == 'vulnerability'
    severity == 'medium'
    threat_name == 'HTTP SQL Injection Attempt'
    
indicators:
    src_ip
    
fields:

Is there any document how clearly to build indicators? I only forund some examples but not a full hsow-to

Regards

Re: Syslog miner indicator

lmori — Sun, 01 Jan 2017 08:14:04 GMT

Hi @AlbertoZanon,

please try this:

conditions:
  - type == 'THREAT'
  - log_subtype == 'vulnerability'
  - severity == 'medium'
  - threat_name == 'HTTP SQL Injection Attempt'
fields:
indicators:
  - src_ip

We should definitely work on the documentation of syslog miner 🙂

But conditions, fields and indicators should be lists or null. In your case they were map. You can check the YAML format here: http://yaml-online-parser.appspot.com/

Re: Syslog miner indicator

AlbertoZanon — Sun, 01 Jan 2017 09:59:37 GMT

Hi Luigi,

happy ne year! and thanks for the tip; so definitely was a formatting error. Good to know I have to follow full YML directives.

If I may just a couple of questions:

1- what's the way to include substring for a specific condition? I would like build a "contains/like" string comparison (regex related) and the code down below seems not valid for minemeld while still passes the YAML online tool check.

conditions:
  - threat_name =~ '[Ii]njection'

2- I also would like to track specific events by frequency and apply rule only if a specific threshold is reached. Do you suggest using aa new processor prototype or an external pythons script ?

Regards

Re: Syslog miner indicator

AlbertoZanon — Mon, 02 Jan 2017 15:54:06 GMT

Hi,

threat_name == 'HTTP SQL Injection Attempt'

Is no a right solution, log format contains "HTTP SQL Injection Attempt(XXX)" where XXX is the threat id, and multiple theert ids contains the same Threat name

Deinitely i need to find a regex match directive not a full equal comparison.

Regards

Re: Syslog miner indicator

KlausGroeger — Mon, 02 Jan 2017 22:26:54 GMT

I am not sure if the matching rule-set supports regex, but I definitly want regex here and some more features to configure on syslog.Miner - eg. age_out: and other.

If you want to test use a ".*" at the end of the string:

"HTTP SQL Injection Attempt.*"

I believe you already tried this, did you?

Best, Klaus

Re: Syslog miner indicator

KlausGroeger — Tue, 03 Jan 2017 10:59:05 GMT

I tried to use wildcards in threat_name but did not succeed. Even if threat_name contains a string the use of ':' (colon) is not allowed. So this one would not work:

- threat_name == 'SCAN: TCP Port Scan(8001)'

We definitely need the ability to use wildcards but I fear YAML does not support it.

Re: Syslog miner indicator

AlbertoZanon — Sun, 08 Jan 2017 18:00:32 GMT

Hi, * and other regular expressions were already tried out with no luck.

I agree with you that regex and like comparison should be added and indeed xternal YAML components like YAML::Logic already got in

Re: Syslog miner indicator

chadwalker — Mon, 09 Jan 2017 00:34:55 GMT

Hi Luigi,

I have setup some miners to obtain threat intel feeds containing IOCs from our partners in addition to the minemeld base ones like Dshield, Spamhaus etc. and I have also got my PAN FW syslog being successfully processed by minemeld. I want the IOCs from our threat feeds to be matched against my syslog for hits. I then want to be able to investigate these hits and look at the detail of the traffic. I tried writing an indicator rule and applying it to my panos syslog miner to match dest_ip as the indicator but must have got something wrong as the number of matches was growing too quickly. Can you provide some doco/guidance on using indicator rules or something to help me identify IOC hits in my syslog and investigate the detail.

Thanks

Chad.

Re: Syslog miner indicator

lmori — Mon, 09 Jan 2017 19:39:18 GMT

Problem is not about YAML itself, YAML is just a way to express conditions. Conditions are boolean expressions based on simplified JMESPath expressions. You can check for a string inside a field with something like this:

conditions:
  - type == 'THREAT'
  - log_subtype == 'spyware'
  - contains(threat_name, 'ZeroAccess') == true
fields: null
indicators:
  - src_ip

Re: Syslog miner indicator

lmori — Mon, 09 Jan 2017 19:43:09 GMT

Hi @AlbertoZanon,

please could you post a screenshot of the syslog miner rulebase ? or you could share the file named /opt/minemeld/local/config/<miner name>_rules.yml

Regards,

luigi

Re: Syslog miner indicator

AlbertoZanon — Mon, 09 Jan 2017 23:44:30 GMT

Hi @lmori, I changed the code uing contains directive as suggested and the hit mach are starting ramping up. For sure a wiki/doc is definetely useful for this section 🙂

/opt/minemeld/local/config/panos_syslog_miner_rules.ym


- conditions: [type == 'THREAT', log_subtype == 'vulnerability', 'contains(threat_name,
 ''HTTP SQL Injection Attempt'') == true']
 fields: null
 indicators: [src_ip]
 name: sql_injection
- conditions: [type == 'TRAFFIC', src_location == 'IT']
 fields: null
 indicators: [src_ip]
 name: test_traffic
- conditions: [type == 'THREAT', src_zone == 'FIBRA', dest_port != '22']
 fields: null
 indicators: [src_ip, url_idx]
 name: test_threat
- conditions: [type == 'THREAT', log_subtype == 'vulnerability', severity == 'informational',
 'contains(threat_name, ''SSH2 Login Attempt'') == true']
 fields: null
 indicators: [src_ip]
 name: ssh_login

Re: Syslog miner indicator

lmori — Tue, 10 Jan 2017 15:29:42 GMT

Hi @AlbertoZanon,

rules looks fine, but if you are looking for a way to match PAN-OS syslog messages against indicators you should look into syslogMatcher node. The syslogMatcher node can also send the details of the matched PAN-OS session to logstash for archival.

syslogMiner is useful if you want to build your own EDL with indicators extracted from PAN-OS logs.

What scenario are you interested in ?

luigi

Re: Syslog miner indicator

AlbertoZanon — Tue, 10 Jan 2017 17:47:39 GMT

Hi,

at this stage I'm not interested in logstash + Kibana/Elasticsearch as long term repository, so syslogMiner node type seems good enough.

Next step is to build up a processor logic so to evaluate indicators and metrics. Any useful docs to look around?

My wishes are:

- extract useful indicators + fields (ie generated_time) [done thanks to your suggestions]

- evaluate indicators against scripted conditions.

Some possible conditions

1) accept indicators and create output EDL only if the same same value is seeen at least a couple of time form the same ruleid/threatid

2) accept indicatos and create output EDL only if the same value is seen from different indicators rules

Regards

Re: Syslog miner indicator

lmori — Sat, 14 Jan 2017 06:09:06 GMT

Hi @AlbertoZanon

1) is not possible today. I would think that the best place to implement that logic is inside the syslog miner. Basically an indicator would be created only if some temporal logic is satisfied.

2) what do you mean by "different indicator rules"? different syslog miner rules or different feeds ?

Thanks !

luigi

Re: Syslog miner indicator

iThreatHunt — Fri, 06 Oct 2017 08:11:57 GMT

@lmori

Can syslog miner receive log from Palo Alto Networks NGFW platforms only?

I try send syslog CEF (correlated rule alert) from ArcSight SIEM to my hosted Minemeld. I captured traffic via tcpdump, Minemeld server get traffic from SIEM. But I not found any log on Minemeld

I need to send Cyber Attack alert from rule on SIEM via syslog into Minemeld.

When Minemeld receive syslog (include Source Address), It will output to stdlib.feedHCGreen.

Then PaloAlto Firewall polling via EDL. Attacker will be blocked.

Thank you.