Using the Syslog Miner

Printer Friendly Page

Using the Syslog Miner

The syslog miner can be used to extract indicators from logs coming from Palo Alto Networks next-generation firewall platforms.

 

1. Adding a syslog Miner

 

In CONFIG, click on the browse prototypes button and select the stdlib.syslogMiner prototype.

 

 

Preview of Prototype interface

Inside the prototype, click on CLONE to create a new node based on the prototype.

 

Close up of Clone button

 

2. Configuring the Miner

 

Specify a name for the new Miner, enable the OUTPUT and click OK

 

Preview of Add Node form

 

3. Connecting the Miner

 

 

Connect the new Miner to the inboundaggregator.

 

Preview of inboundaggregator inputs form

 

And press COMMIT.

 

3. Configuring syslog forwarding on PAN-OS

 

Please refer to the PAN-OS Administration Guide for instructions on how to configure log forwarding to a syslog server on PAN-OS (Software End-of-Life (EoL)). By defeault the syslog server on the VM listens for PAN-OS syslog messages on port tcp/13514.

 

Preview of Syslog Server Profile form

 

4. Checking if syslog messages are received

 

In NODES, click on the new syslog Miner and select the STATS tab. Check the SYSLOG.PROCESSED metric, this counter is incremented every time the Miner receives a syslog message.

 

Close up view of Syslog message received

 

Log format

A full list of parameters can be found in the following python script:

https://gist.github.com/jtschichold/87f59b99d98c8eac1da5 

 

Traffic logs

{
"src_zone": "Tap",
"protocol": "tcp",
"src_translated_port": "0",
"dest_translated_ip": "",
"app": "incomplete",
"src_translated_ip": "",
"log_subtype": "end",
"dest_zone": "Tap",
"bytes_in": "0",
"generated_time": "2014/05/13 16:14:29",
"serial_number": "007201000291",
"type": "TRAFFIC",
"action_flags": "0x8000000000000000",
"packets_out": "1",
"event.tags": [
"TRAFFIC",
"TRAFFIC_FIELDS_6_0"
],
"receive_time": "2014/05/13 16:14:29",
"start_time": "2014/05/13 16:14:24",
"virtual_system": "vsys1",
"src_location": "US",
"future_use1": "Apr 30 00:25:26 1",
"future_use2": "1",
"future_use3": "2014/05/13 16:14:29",
"future_use4": "0",
"future_use5": "0",
"log_forwarding_profile": "to Panorama",
"bytes": "62",
"packets_in": "0",
"sequence_number": "259937194",
"rule": "Allow-All",
"duration": "0",
"repeat_count": "1",
"dest_interface": "ethernet1/1",
"dest_port": "9191",
"category": "any",
"src_port": "55869",
"src_ip": "169.131.1.254",
"dest_user": "pancademo\\jason.bringanti",
"bytes_out": "62",
"dest_ip": "10.154.14.7",
"src_user": "",
"src_interface": "ethernet1/1",
"packets": "1",
"session_id": "74404",
"dest_translated_port": "0",
"dest_location": "10.0.0.0-10.255.255.255",
"flags": "0x64",
"action": "allow"
}

 

Threat logs

{
"src_zone": "Tap",
"protocol": "tcp",
"src_translated_port": "0",
"dest_translated_ip": "",
"app": "imap",
"misc": "Rjf4TBoDBs389u.Emf",
"src_translated_ip": "",
"log_subtype": "vulnerability",
"dest_zone": "Tap",
"generated_time": "2014/05/13 06:55:16",
"serial_number": "007201000291",
"type": "THREAT",
"action_flags": "0x8000000000000000",
"direction": "server-to-client",
"event.tags": [
"THREAT",
"THREAT_FIELDS_6_0"
],
"receive_time": "2014/05/13 06:55:16",
"content_type": "",
"virtual_system": "vsys1",
"src_location": "10.0.0.0-10.255.255.255",
"future_use1": "Mar 16 03:23:08 1",
"future_use2": "1",
"future_use3": "2014/05/13 06:55:16",
"future_use4": "0",
"src_user": "pancademo\\fausto.allen",
"dest_translated_port": "0",
"sequence_number": "1525703173",
"threat_name": "Microsoft Windows Image Color Management Remote Code Execution Vulnerability(31720)",
"rule": "Allow-All",
"repeat_count": "1",
"dest_interface": "ethernet1/1",
"dest_port": "42913",
"category": "any",
"src_port": "143",
"severity": "critical",
"src_ip": "10.154.10.156",
"dest_user": "",
"dest_ip": "66.1.1.5",
"log_forwarding_profile": "to Panorama",
"url_idx": "",
"src_interface": "ethernet1/1",
"pcap_id": "1199806114522039943",
"cloud_address": "",
"session_id": "190977",
"dest_location": "US",
"flags": "0x80000000",
"action": "alert"
}

 

Under the Hood

The following diagram depicts the message flow.

 

Rendition of syslog miner schema

Syslog messages are received by the rsyslog deamon running on the VM. Rsyslog translates the messages into JSON and sends them to the syslog miner node inside the MineMeld engine.

Labels (2)
Comments

Hi Luigi,

 

I have the miner recieving the syslog messages fine, but I'm struggling to build a rule to match/hit. I'm happy with the conditions and fields logic, but I can't figure out what to put into the indicator section. Do I need to enter RegEx to extract the IP indicator?

 

Thanks,

Tim 

Hi Tim,

could you open a discussion and post your current rule ? I will be happy to take a look.

Hi  Luigi,

 

Can Syslog Miner receive syslog log from other devices (such as SIEM, IPS)?

 

Thanks

My analysis: 

 

other products syslog(current is paloalto) ---> rsyslog ---> RabbitMQ ---> minemeld

 

so the rsyslog configuration is not write down in a physical log file. It is using "palo_alto_networks.rb" to normalize (by module "pmpanngfw" and "mmnormalize") and forward to the RabbitMQ (by module "omrabbitmq")

 

https://github.com/rsyslog/rsyslog/blob/a399cbaa21f4c11ccdc3345f4128b9f518340b58/contrib/pmpanngfw/R...

 

https://gist.github.com/jtschichold/87f59b99d98c8eac1da5

 

I think the fast way is to generate the other syslog has the same format as paloalto's or transform into a format compatible with mmnormalize

I am looking for something like this as well....Is there not a miner that we can point to a syslog/list somehwere that could be used with the DAG pusher node to dynamically populate rules based upon logs from something like Splunk?  Maybe I'm missing something here.

I have the miner u[p and running and everything looks good. PA220 is forwarding syslogs I can see them on my other syslog server. My miner still says 0. what is the best way to check if the minemeld server is getting the messages and f there are any erros.

 

 

Running PA-220 with 8.1.4 and do not get any input in the syslog-miner. tcpdump shows syslog traffic arriving at port 13514.

tcp socket is established:

root@minemeld:~# netstat -napt | grep 13514
tcp 0 0 0.0.0.0:13514 0.0.0.0:* LISTEN 13984/rsyslogd
tcp 0 0 192.168.5.20:13514 192.168.3.2:56060 ESTABLISHED 13984/rsyslogd

 

and if it isn't listening, how do we troubleshoot why?

I am also looking at getting syslog out of other devices (other type ids / listeners) which I could get into minemeld and use this as a collector of IOCs and possible feed them further as policy in PA. Anyone has any tips in that direction? I'm trying different things for few days now with not much luck.

Ask Questions Get Answers Join the Live Community
Version history
Revision #:
4 of 4
Last update:
3 weeks ago
Updated by:
 
Contributors