topic Re: No metrics showing up in a syslog analyser node in General Topics

No metrics showing up in a syslog analyser node

borising — Wed, 26 Jul 2017 18:48:31 GMT

Hi,

I followed this post the other day and have been forwarding logs from my firewall for 2 days now, but without any hits, so I am wondering if I have done something wrong? I can see in a tcpdump dump on the minemeld server, that logs are received on port 13514/TCP. Also, the logs that are sent to minemeld are dropped traffic from an EDL, so the indicators should be present.

https://live.paloaltonetworks.com/t5/MineMeld-Articles/Correlating-PAN-OS-syslog-with-indicators/ta-p/72078

I am using the stdlib.localSyslog prototype, as I just want to know whits lists I hit.

Any ideas on how to troubleshoot this?

I'm using:

PAN-OS 8.0.3-h4

Minemeld v 0.9.40

Re: No metrics showing up in a syslog analyser node

lmori — Wed, 26 Jul 2017 23:21:26 GMT

Hi @borising,

please, could you attach a screenshots of the stats of the syslog miner node ? (Nodes > <syslog miner nodes> > Stats tab on the right)

Thanks,

luigi

Re: No metrics showing up in a syslog analyser node

borising — Thu, 27 Jul 2017 14:55:30 GMT

Hi Luigi,

I have attached the screenshot, and a screenshot of the sources tab and how it looks on the Nodes page.

Regards,

Re: No metrics showing up in a syslog analyser node

lmori — Thu, 27 Jul 2017 15:01:45 GMT

Hi @borising,

could you double check the logs rsyslog in /var/log/rsyslog to see if there are errors in loading the rabbitmq modules ?

luigi

Re: No metrics showing up in a syslog analyser node

borising — Thu, 27 Jul 2017 15:12:03 GMT

Hi Luigi,

There are only 2 rsyslog.log files, which I have cat'ed below. Rsyslogd is running, as you can see.

xxx@minemeld01:/var/log$ cat rsyslog.log
xxx@minemeld01:/var/log$ cat rsyslog.log.1
Jul  2 21:44:32 minemeld01 rsyslogd: [origin software="rsyslogd" swVersion="8.17.0" x-pid="5101" x-info="http://www.rsyslog.com"] start
Jul  2 21:44:32 minemeld01 rsyslogd: rsyslogd's groupid changed to 104
Jul  2 21:44:32 minemeld01 rsyslogd: rsyslogd's userid changed to 101
Jul  2 21:45:14 minemeld01 rsyslogd: [origin software="rsyslogd" swVersion="8.17.0" x-pid="5101" x-info="http://www.rsyslog.com"] exiting on signal 15.
Jul  2 21:45:14 minemeld01 rsyslogd: [origin software="rsyslogd" swVersion="8.17.0" x-pid="5420" x-info="http://www.rsyslog.com"] start
Jul  2 21:45:14 minemeld01 rsyslogd: rsyslogd's groupid changed to 104
Jul  2 21:45:14 minemeld01 rsyslogd: rsyslogd's userid changed to 101
Jul  4 06:53:12 minemeld01 rsyslogd: [origin software="rsyslogd" swVersion="8.17.0" x-pid="5420" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
xxx@minemeld01:/var/log$ ps xau | grep rsyslogd
xxx       430  0.0  0.1  11764  1960 pts/1    S+   17:08   0:00 grep --color=auto rsyslogd
syslog    5420  0.0  0.0 378196   708 ?        Ssl  Jul02   0:03 rsyslogd

And the rsyslog version:

xxx@minemeld01:/var/log$ dpkg -l | grep rsyslog
ii  rsyslog                            8.17.0-0adiscon2trusty1                    amd64        a rocket-fast system for log processing
ii  rsyslog-minemeld                   8.16-0                                     amd64        minemeld modules for rsyslog
ii  rsyslog-mmnormalize                8.17.0-0adiscon2trusty1                    amd64        The rsyslog-mmnormalize package provides log normalization

Re: No metrics showing up in a syslog analyser node

lmori — Fri, 28 Jul 2017 09:59:36 GMT

Hi @borising,

could you check the output of this command:

sudo rabbimq_ctl list_queues | grep -i syslog

Re: No metrics showing up in a syslog analyser node

borising — Fri, 28 Jul 2017 18:14:17 GMT

Here you go:

xxx@minemeld01:~$ sudo rabbitmqctl list_queues | grep -i syslog
localSyslog:rpc 0
mbus:directslave:localSyslog:rpc        0
mbus:slave:localSyslog:rpc      0
xxx@minemeld01:~$

Re: No metrics showing up in a syslog analyser node

borising — Mon, 31 Jul 2017 13:31:46 GMT

Hi @lmori,

Any luck of finding out what causes this problem?

Thanks!

Re: No metrics showing up in a syslog analyser node

lmori — Fri, 04 Aug 2017 10:39:18 GMT

Hi @borising,

I am adding new counters in syslog matcher to help troubleshooting this, they will make into the next release. If you are in a hurry drop me an email at lmori@paloaltonetworks.com and we can have a webmeeting to debug this together.

Luigi

Re: No metrics showing up in a syslog analyser node

borising — Mon, 07 Aug 2017 06:12:32 GMT

Hi @lmori,

When do you think the next release will be available?

Regards,

Re: No metrics showing up in a syslog analyser node

lmori — Mon, 07 Aug 2017 07:33:51 GMT

Hi @borising,

should be in a week.

Luigi

Re: No metrics showing up in a syslog analyser node

borising — Mon, 07 Aug 2017 08:03:02 GMT

Hi @lmori,

Then I'll just wait, and update this message, when I have tested further 🙂

Regards,

Re: No metrics showing up in a syslog analyser node

lmori — Mon, 07 Aug 2017 11:35:32 GMT

Hi @borising,

which PAN-OS version are you using ?

Thanks,

luigi

Re: No metrics showing up in a syslog analyser node

borising — Mon, 07 Aug 2017 11:59:06 GMT

Hi @lmori,

I started out with 8.0.3-h4, but have upgraded to 8.0.4 along the way.

Regards,

Re: No metrics showing up in a syslog analyser node

borising — Sun, 20 Aug 2017 18:57:58 GMT

Hi again @lmori

I just did a debug session of when rsyslogd receives a syslog message, as I had a suspicion that it might be an issue parsing the 8.x logs.

Though I can't really get much out of it, you might 🙂

So if you think you could use the output for further debugging, then let me know.

Regards,

Bo Rising

Re: No metrics showing up in a syslog analyser node

borising — Wed, 06 Sep 2017 05:20:01 GMT

Hi @lmori

My minemeld updated a couple of hours ago to 0.9.42 (good work with the backup!), but I still can't see any indicators in the syslog node, even that my firewall have sent plenty of logs since the upgrade.

How could we troublehsoot this further?

Have a great day!

Re: No metrics showing up in a syslog analyser node

borising — Thu, 21 Sep 2017 06:51:17 GMT

Hi @lmori,

Do you think, there's a possibility that I can troubleshoot this someway myself?

Regards,