Error when using stdlib.syslogMiner

[Leon_Twenning]

Hi together,

I am trying to import PANOS-Threat Logs into MineMeld using the syslogMiner.

I have configured the Miner and the LogForwarding via Panorama and can see the incoming logs at the Minemeld instance using tcpdump.

Still I see no indicators in my Miner-Node. The Engine Logs show following error that I think is relevant to the problem:

Spoiler

(2082)syslog._amqp_consumer ERROR: Miner_Test - Exception in consumer glet
Traceback (most recent call last):
File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/minemeld/ft/syslog.py", line 739, in _amqp_consumer
password=self.rabbitmq_password
File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/amqp/connection.py", line 165, in __init__
self.transport = self.Transport(host, connect_timeout, ssl)
File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/amqp/connection.py", line 186, in Transport
return create_transport(host, connect_timeout, ssl)
File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/amqp/transport.py", line 299, in create_transport
return TCPTransport(host, connect_timeout)
File "/opt/minemeld/engine/0.9.60/local/lib/python2.7/site-packages/amqp/transport.py", line 95, in __init__
raise socket.error(last_err)
error: [Errno 111] Connection refused

I already checked the forums for similar errors, but couldnt find anything that helped me. I also stumbled about the advice to restart rabbitmq-server, but this service doesnt exist on my instance. For installation I followed the tutorial here: 

https://live.paloaltonetworks.com/t5/MineMeld-Articles/Manually-install-MineMeld-on-Ubuntu-Server-16...

If anyone can assist me with this problem I would  be very glad!

Greetings Leon!

[andrew.copeland]

I'm having this issue as well. Same issues in the log file and I also used the build for Ubuntu 16.  This config came off of my previous installation of ubuntu 14 so I don't think it's my minemeld config. I also see established traffic from my firewalls over port 13514 so it seems that the issue is somewhere between rsyslog and the miner itself.

I think that when Luigi created the new install guide there's something missing that's required for the syslog miner to function.  @lmori are you able to confirm?

[andrew.copeland]

As an update, it looks like the error is because "rabbitmq-server" isn't installed, when it was in the Ubuntu 14 version I had running. However, installing rabbitmq doesn't fix the logs showing up in MineMeld, it only removes the errors.  It seems it's missing some other configuration, but I'm not sure what that is.

[andrew.copeland]

I believe I have fixed it, at least in the interim until it can be added to the Palo repo.  According to Luigi here rsyslog (or more appropriately the package called rsyslog-minemeld in Ubuntu 14.04) Was built by them from source with additional features enabled, and distributed through their repo.  It does not seem that rsyslog-minemeld is distrubuted in their current Xenial/16.04 repo.

http://minemeld-updates.panw.io/ubuntu xenial-minemeld main

 However, when I built a current version of rsyslog with those features; it was incompatible with the /etc/rsyslog.d/*.conf files.  I was able to find an old version of rsyslog "8.19.0", combile it, install the .deb file on my minemeld-server. I also installed I also installed via apt "librabbitmq4" and "liblognorm2" as refferenced by some of my /var/log/syslog errors.  Once I did that, all the errors went away, and IPs started showing up in my miner/output.