Nodes keep stopping - how to start and keep them started?

[hshawn]

Just spun up a new Minemeld server and its working however the nodes like to just stop and I am not sure how to get them to start up and stay started. Rebooting will bring everything back up and they wiull be started for about a minute then they all stop (see screenshot). At home I dont have this issue, all the nodes stay running/started and do their polling, etc without issues. I should be seeing waaaaaaay more indicators specially on the alienvault_reputation node. Thoughts?

[Sistemas_SanLucar]

Hi,

A few days ago, I installed Minemeld con Ansible and Ubuntu 18.

I think , I have a similar problem but with  urlhaus.URL.


Regards

Regards

[hshawn]

Might be related but what I am seeing is all nodes will stop instead of just some of them. 

OT but how did you get the ansible install to work on Ubuntu18? Mine will error out right away.

[lmori]

Hi @Sistemas_SanLucar,

that is a conflict with RabbitMQ version installed by default on Ubuntu 18, that is also we don't support Ubuntu 18 yet in the Ansible playbook. You should install an older RabbitMQ release (3.2.X)

[Sistemas_SanLucar]

1)a) Add the repositories fo file /etc/apt/sources.list

deb http://us.archive.ubuntu.com/ubuntu/ bionic universe /etc/apt/sources.list
deb http://minemeld-updates.panw.io/ubuntu trusty-minemeld main

1)    $ sudo apt-get update
2)    $ sudo apt-get upgrade # optional
3)    $ sudo apt-get install -y gcc git python-minimal python2.7-dev libffi-dev libssl-dev make
4)    $ wget https://bootstrap.pypa.io/get-pip.py
5)    $ sudo -H python get-pip.py
6)    $ sudo -H pip install ansible
7)    $ git clone https://github.com/PaloAltoNetworks/minemeld-ansible.git
😎    $ cd minemeld-ansible

8)a) After setp 8), we modified configuration

I copeid the files  Ubuntu-16.04.yml to Ubuntu-18.04.yml  in directory structure , these files didn't exist in setup.
Current structure:


./roles/infrastructure/vars/Ubuntu-14.04.yml
./roles/infrastructure/vars/Ubuntu-16.04.yml
./roles/infrastructure/vars/Ubuntu-18.04.yml

./roles/minemeld/vars/Ubuntu-14.04.yml
./roles/minemeld/vars/Ubuntu-16.04.yml
./roles/minemeld/vars/Ubuntu-18.04.yml

./roles/minemeld/tasks/Ubuntu-18.04-post.yml
./roles/minemeld/tasks/Ubuntu-16.04-post.yml
./roles/minemeld/tasks/Ubuntu-14.04-post.yml

8)b) As indicated at the beginning of the README.MD installation manual, we have modified the local.yml file to be able to install the stable version instead of the "dev" development one.
So that the local.yml file remains this way.

-----------------------------------------
- name: minemeld playbook
  hosts: 127.0.0.1
  connection: local
  become: true

  vars:
          #  minemeld_version: develop
    file_permissions: 'u=rwX,g=rwX,o=rX'
  # uncomment the following to install stable
    minemeld_version: master
    group_permissions: 'u=rwX,g=rX,o=rX'
  # remove comment to set custom repositories
  # core_repo: "https://github.com/jtschichold/minemeld-core.git"
  # prototype_repo: "https://github.com/jtschichold/minemeld-node-prototypes.git"
  # webui_repo: "https://github.com/jtschichold/minemeld-webui.git"

  roles:
  - infrastructure
  - minemeld
-------------------------------------------

9)    $ ansible-playbook -K -i 127.0.0.1, local.yml
10)    $ usermod -a -G minemeld # add your user to minemeld group, useful for development

[Sistemas_SanLucar]

How do i do it?

thank you

[erik6861]

I still run into this issue with Centos7, even after updating to the fixes in 9.50.

It occurs randomly whenever I begin to add to a config, after a restart from a commit. After the service starts back up, memory tends to nearly double what it was before the commit and subsequent restarts (commits) take much much longer to stop and start than normal. Throwing more memory at it does not seem to make a differnce.

The only way to fix is to keep removing nodes and commiting until you remove the one causing the problem. Then rebuild the config back to normal slowly. But it's annoying to have to keep rebuilding a config every time this issue occurs (without knowing which node is even the cause), and having to wonder if your next config commit will cause the problem again.

[lmori]

Hi @erik6861,

we have just implemented some major changes in MineMeld to improve stability on CentOS7 (and support Ubuntu 18.04). If you want to test it you can:

• install a new instance of CentOS7 with updated packages (yum update)
• follow the instructions here: https://github.com/PaloAltoNetworks/minemeld-ansible#howto-on-centos-7rhel-7
• do a full export from your current instance (SYSTEM > EXPORT BACKUP)
• restore the backup in the new instance (SYSTEM > RESTORE)
• if you are using extensions please make to reinstall and activate them

Thanks,

luigi

[erik6861]

Thanks for the info. I will definately give this a try when time permits, other projects have just taken presendence so will have to revisit in a couple of months (hopefully sooner).

I've found if I make minimal config changes one at a time and let them marinate after commit for a bit before making any other changes, I have a better chance at avoiding the issue. It's just slow going to get configured up like this or make any changes.