topic Re: Problems with CentOs 7 and MM 0.9.52 in General Topics

Problems with CentOs 7 and MM 0.9.52

danilo.souza — Wed, 23 Jan 2019 12:38:07 GMT

Hi guys,

I used to run standalone MM 0.9.50 with CentOS 7, perfectly. Last week I updated MM to 0.9.52 with the help of @lmori and the proccess was completed with success. See ( https://live.paloaltonetworks.com/t5/MineMeld-Discussions/Updating-MineMeld-from-0-9-50-to-the-latest-stable-version/td-p/245365 ).

However since the upload my MM doesn't work the same way. On my Dashboard is visible that my miner works fine, more than 90K indicators, but almost none of them ara available, less than 1K in the outputs (see figure below).

If we detailed the proccess, we see that the status of many nodes is "stopped". The number of indicators forwarded by the miners is high.

The number of indicators forwarded by the aggregators is almost the same.

But the number of indicators available by the outputs is extrmely low.

Has anybody experienced something similar? How you dealed with the problem?

Best Regards.

Re: Problems with CentOs 7 and MM 0.9.52

lmori — Wed, 23 Jan 2019 12:55:31 GMT

Hi @danilo.souza,

could you try this:

in /opt/minemeld/engine/core/minemeld/run open the file launcher.py
at line 284 change

mbusmaster.wait_for_chassis(timeout=10)

into this

mbusmaster.wait_for_chassis(timeout=60)

and restart

Re: Problems with CentOs 7 and MM 0.9.52

danilo.souza — Wed, 23 Jan 2019 14:08:01 GMT

Hi @lmori

first I restarted just the engine through the web interface, but not changed. Then I restarted my server and I got a bit more indicators.

I still have many nodes with the staus "stopped". And the number of the indicators in the output isn't matching yet.

What is this parameter?

mbusmaster.wait_for_chassis

Can it be changed even higher? What is the consequence?

Re: Problems with CentOs 7 and MM 0.9.52

danilo.souza — Tue, 29 Jan 2019 17:52:58 GMT

Hi @lmori I still have nodes with the staus "stopped". And the number of the indicators in the output isn't matching yet. Did you have the opportunity to see my last reply? Best regards.

Re: Problems with CentOs 7 and MM 0.9.52

danilo.souza — Mon, 04 Feb 2019 18:11:39 GMT

Hi @lmori

I'm sorry to insist. I still have nodes with the staus "stopped". I tryed to find any other evidences of the problem, but I got nothing. Unhappy, this is happening since I upgraded to 0.9.52.

Did you have the opportunity to see my last reply (https://live.paloaltonetworks.com/t5/MineMeld-Discussions/Problems-with-CentOs-7-and-MM-0-9-52/m-p/247157/highlight/true#M2930)?

Best regards.

Re: Problems with CentOs 7 and MM 0.9.52

StephenBradley — Mon, 04 Feb 2019 19:24:03 GMT

Our server team just upgraded my Minemeld to CentOS7 0.9.52 and it only lists about 29k indicators but only 43 total in the output. I have an Ubuntu 14.04 image running 0.9.52 and it has 231k indicators and 77k output.

We have never had CentOS running yet but that is our PROD operating system and they want us to use it. Right now I just have my test box feeding the PAN the EDL.

I tried bumping up to 60 on the timeout.

Re: Problems with CentOs 7 and MM 0.9.52

danilo.souza — Mon, 11 Feb 2019 11:52:56 GMT

Hi @xhoms and @lmori

Apparently there is a problem with the version 0.9.52 and the CentOS. Do you have any other known issue? Is it being addressed? It is impacting our environment. Do you advise rolling back to 0.9.50 version? If yes, How to do it?

Best regards.

Re: Problems with CentOs 7 and MM 0.9.52

lmori — Mon, 11 Feb 2019 12:24:57 GMT

Hi @danilo.souza,

yes, I am looking into this. The problem seems to be related to RabbitMQ.

If you want to install the previous version, you can use the update procedure for Ansible but in the file change the line 7 to look like roles/minemeld/tasks/core.yml:

version: "0.9.50.post4"

Re: Problems with CentOs 7 and MM 0.9.52

StephenBradley — Mon, 11 Feb 2019 16:54:13 GMT

That doesn't work either.

fatal: [127.0.0.1]: FAILED! => {"before": "bdd6879cc2c72094702d301e3e1bae4a198eed5f", "changed": false, "msg": "Local modifications exist in repository (force=no)."}

to retry, use: --limit @/minemeld-ansible/local.retry

I have tried setting things to force yes but it still fails.

Re: Problems with CentOs 7 and MM 0.9.52

danilo.souza — Tue, 12 Feb 2019 10:49:26 GMT

Hi @StephenBradley and @lmori

I can confirm it is extremely unstable. I tried rolling back to the previous version using the tips of @lmori in the previous post. At first it failed and present a fatal error. I enabled the extensions the error desapeared, but I'm still getting the "stopped" status for some nodes. The curious point is that, after the rolling back, I'm still getting the 0.9.52 vrsion in MM WEBGUI.

MM Version

Did you experienced something similar @StephenBradley ?

Best regards.

Re: Problems with CentOs 7 and MM 0.9.52

danilo.souza — Thu, 21 Feb 2019 19:15:08 GMT

Hi @lmori

I'm having some issues with the upload of MM to 0.9.52, as reported previously. After the preocedure you told me to follow to rool back to 0.9.50, my machine has not worked properly. First of all, it still shows I'm using the 0.9.52 version, as I showed in my last post.

I tried to repeat the procedure and now I get an error

MM Ansible

Could you tell me what is the best pratice to get my MM back to work? Is is really affecting (downgrading) my experience with MM.

Best regards.

Re: Problems with CentOs 7 and MM 0.9.52

lmori — Thu, 28 Feb 2019 07:47:39 GMT

Hi @danilo.souza,

something is wrong in the downloaded code, it seems to be pretty old.

Please do this:

remove your current minemeld-ansible directory and /opt/minemeld/engine, /opt/minemeld/www and /opt/minemeld/prototypes directories
copy /opt/minemeld/local contents somewhere for backup
update your CentOS 7 packages to the latest versions (yum update)
follow the instructions here: https://github.com/PaloAltoNetworks/minemeld-ansible#howto-on-centos-7rhel-7
DO NOT SPECIFY VERSION MASTER!

The new version of the playbook will install the MineMeld version that is still in beta but is way more reliable on CentOS7.

Luigi

Re: Problems with CentOs 7 and MM 0.9.52

StephenBradley — Thu, 28 Feb 2019 14:00:30 GMT

Tried but all I get is this.

TASK [minemeld : minemeld-node-prototypes repo] *********************************************************************************************************************************************************************************************
ok: [127.0.0.1]

TASK [minemeld : minemeld-node-prototypes current link] *************************************************************************************************************************************************************************************
ok: [127.0.0.1]

TASK [minemeld : minemeld-core repo] ********************************************************************************************************************************************************************************************************
fatal: [127.0.0.1]: FAILED! => {"before": "bdd6879cc2c72094702d301e3e1bae4a198eed5f", "changed": false, "msg": "Local modifications exist in repository (force=no)."}
to retry, use: --limit @/opt/minemeld-ansible/local.retry

PLAY RECAP **********************************************************************************************************************************************************************************************************************************
127.0.0.1 : ok=31 changed=0 unreachable=0 failed=1

Re: Problems with CentOs 7 and MM 0.9.52

lmori — Thu, 28 Feb 2019 16:58:57 GMT

Hi @StephenBradley,

did you delete the directory /opt/minemeld/prototypes before rerunning the playbook?

Thanks,

luigi

Re: Problems with CentOs 7 and MM 0.9.52

StephenBradley — Fri, 01 Mar 2019 13:49:55 GMT

Well it seemed to have gotten farther but now I get a firewall not running error.

Re: Problems with CentOs 7 and MM 0.9.52

lmori — Fri, 01 Mar 2019 13:51:43 GMT

Hi @StephenBradley,

could you elaborate? What error do you see?

Re: Problems with CentOs 7 and MM 0.9.52

StephenBradley — Fri, 01 Mar 2019 15:21:26 GMT

TASK [minemeld : collectd types] ************************************************************************************************************************************************************************************************************
ok: [127.0.0.1]

TASK [minemeld : open https port] ***********************************************************************************************************************************************************************************************************
fatal: [127.0.0.1]: FAILED! => {"changed": true, "cmd": ["firewall-cmd", "--zone=public", "--add-service=https"], "delta": "0:00:00.396078", "end": "2019-03-01 08:33:12.260520", "msg": "non-zero return code", "rc": 252, "start": "2019-03-01 08:33:11.864442", "stderr": "FirewallD is not running", "stderr_lines": ["FirewallD is not running"], "stdout": "", "stdout_lines": []}

RUNNING HANDLER [minemeld : restart redis] **************************************************************************************************************************************************************************************************
to retry, use: --limit @/opt/minemeld-ansible/local.retry

Re: Problems with CentOs 7 and MM 0.9.52

danilo.souza — Wed, 10 Apr 2019 16:21:26 GMT

Hi @lmori

I followed the steps you indicated but I'm still getting an error:

Have you ever experienced something similar?

Regards

Re: Problems with CentOs 7 and MM 0.9.52

StephenBradley — Wed, 10 Apr 2019 17:02:48 GMT

Did you delete the directory /opt/minemeld/prototypes before rerunning the playbook like Luigi (lmori) suggested? That fixed my problem I was having with that issue.

steve

Re: Problems with CentOs 7 and MM 0.9.52

danilo.souza — Wed, 10 Apr 2019 19:52:13 GMT

Hi @StephenBradley

for sure. When updating I delete

/opt/minemeld/prototypes

/opt/minemeld/engine

/opt/minemeld/www

Unhapply it did not work for me.

Regards