topic Re: MineMeld - need help importing and processing syslog data in General Topics

MineMeld - need help importing and processing syslog data

jerryshenk — Fri, 05 Aug 2016 15:42:03 GMT

I installed the MineMeld VM on my ESXi box yesterday and it came up just fine, I can login to it from the VM Console, the web console, and over SSH. I've edited the /etc/rsyslog.conf file and /etc/iptables/rules.v4 so that syslog data is coming in from the firewall to the /var/log/syslog file.

Question: How do I get MineMeld to process the syslog data? I looked at the "Using the sysloig Miner" article and have created a miner (stdlib.syslogMiner) and linked it to the inboundaggregator but, it isn't processing anything. I'm sure I'm missing something rather simple - can somebody point me in the right direction?

Re: MineMeld - need help importing and processing syslog data

lmori — Tue, 16 Aug 2016 20:32:35 GMT

Hi jerryshenk,

do you have a file named /etc/rsyslog.d/60-syslog-minemeld.conf in your instance ?

This should instruct rsyslog to parse syslog messages on port 13514/tcp into JSON using PAN-OS rulebase and push them to RabbitMQ on a queue MineMeld should listen to.

This seems complex, but it is just a short config file. Could you check ?

Re: MineMeld - need help importing and processing syslog data

AndreasTrautmann — Thu, 12 Oct 2017 17:26:25 GMT

Hi there

I've got pretty much the same "problem" as jerryshenk.

I checked for the file mentioned (60-syslog-minemeld.conf). But it does not exist in

/etc/rsyslog.d/

Can i get the file/settings from somewhere?

Thanks alot

Andreas

Re: MineMeld - need help importing and processing syslog data

AndreasTrautmann — Fri, 13 Oct 2017 15:39:13 GMT

Update:
OK, I found the file in the apt package, extracted it and put it to /etc/rsyslog.d/ together with palo_alto_networks.rb.

But still no Indicators in my syslog-miner.

The Syslog is arriving at the minemeld server, ufw is opened.

Do I need a "syslog miner rule" for it to start collecting indicators?

Any Ideas how I can further troubleshoot this?

Context Infos:
Installation on Ubuntu 16.04

Installed via ansible playbook

Thanks, best Regards

Andreas

Re: MineMeld - need help importing and processing syslog data

LucaMarchiori — Fri, 13 Oct 2017 17:11:25 GMT

@AndreasTrautmann: I'm quite new at this myself but yes, after you have syslog showing up in statistics > SYSLOG.PROCESSED, the next step is to create some rules.

I found this thread helpful:

https://live.paloaltonetworks.com/t5/MineMeld-Articles/Using-the-syslog-Miner/ta-p/77262

Re: MineMeld - need help importing and processing syslog data

AndreasTrautmann — Fri, 13 Oct 2017 17:15:14 GMT

Hi Luca

Thanks for the hint.

Unfortunately my miner does not yet receive anything (SYSLOG.PROCESSED is 0).

So my problem is further "up" somewhere in the "link" between rsyslogd and the miner.

Best Regards
Andreas

Re: MineMeld - need help importing and processing syslog data

LucaMarchiori — Fri, 13 Oct 2017 18:12:45 GMT

@AndreasTrautmann: Got you. Definitely the SYSLOG.PROCESSED counter starts moving even with zero rules present on the node itself, so that's what needs fixing first (as you already pointed out).

Re: MineMeld - need help importing and processing syslog data

HAO.BAN — Fri, 23 Mar 2018 15:00:58 GMT

@lmoriwrote:
Hi jerryshenk,
do you have a file named /etc/rsyslog.d/60-syslog-minemeld.conf in your instance ?
This should instruct rsyslog to parse syslog messages on port 13514/tcp into JSON using PAN-OS rulebase and push them to RabbitMQ on a queue MineMeld should listen to.
This seems complex, but it is just a short config file. Could you check ?

Is "/etc/rsyslog.d/palo_alto_networks.rb" the only rulebase file? Can I modify another rulebase that can make the minemeld to integrate with any other products syslogs such as any AV, FW or IPS? Do you have any instruction for creating a "rb" file? Thanks!

Re: MineMeld - need help importing and processing syslog data

DaniBCS — Fri, 21 Jun 2019 13:25:22 GMT

Did you have any progress here? I'm at the same point wondering if I need to create own .rb file and place it before or after the 60-... rb file. How does it decide which template to use?

Re: MineMeld - need help importing and processing syslog data

mboehlke — Mon, 26 Aug 2019 15:44:49 GMT

What version of PanOS are you using? I've been troubleshooting the same issue. We turned on debugging for rsyslogd and it's logging error messages while parsing the palo's syslog. It looks like the threat log format changed between 8.0.X and 8.1.X. I'm thinking that the config given to rsyslogd doesn't know how to handle the 8.1.X format?

You can see the format differences between these two links:
8.0.x Format

8.1.x Format

Re: MineMeld - need help importing and processing syslog data

DaniBCS — Tue, 27 Aug 2019 08:02:10 GMT

Well , I'm up to making it ingest non-PA syslog. The end goal is to have it ingest all sorts of logs and make aggregators which do conclusions based on multiple sources and prep inputs for others in the network.