MineMeld hanging after reboot

Reply
L4 Transporter

MineMeld hanging after reboot

Dear Luigi,

 

We are having a lot of issues with MineMeld, I hope you can help.

 

We are deploying MineMeld with our customers on their own vmware environment in a DMZ zone with NO DHCP.

I know you prefer to use AutoFocus, MineMeld in AWS, MineMeld on Ubuntu or MineMeld with DHCP, but this is not always feasible for our customers.

 

In any case here is the setup we follow:

 

Installation

  1. Download the fast-install OVA file. This has the needed iso-file bundled in
    https://s3-eu-west-1.amazonaws.com/minemeld-dist/0_9/minemeld-vm-0.9.10-1build1.ova
  2. Deploy ova in VMWare
    All settings can be kept there default values
  3. Boot the MineMeld instance and let it fail the initialization. It will take some time as it searches in vain for a DHCP server
  4. Login to the console of the Virtual Machine with credentials:
    username: ubuntu
    password: rsplizardspock
  5. Configure a static IP for your MineMeld

    $ sudo vi /etc/network/interfaces.d/eth0.cfg

    auto eth0
    iface eth0 inet static
    address x.x.x.x
    netmask x.x.x.x
    gateway x.x.x.x
    dns-nameservers x.x.x.x

    Press "esc" and ":wq" to save the file in vi.

  6. Then enter following command. This will reset a value so MineMeld is correctly installed on next reboot.

    $ sudo rm -rf /var/lib/cloud/*

  7. Reboot the device

    $ sudo shutdown -r now

  8. The instance will reboot and (re)start the MineMeld installation.
  9. Installation is now done. You can access the Gui
    https://x.x.x.x

 

This works perfectly.

After install we can access the Gui

We can reboot the appliance via CLI, or just in VMWare give it a hard power-off and start up -> the MineMeld will run again

 

 

NEXT

Since PanOS 7.1 and higher we need to install a custom certificate on the MineMeld to get it to connect

For this we follow the following steps

 

  1. Create a new self-signed CA certificate on the Palo Alto firewall for MineMeld
  2. Export both the public and private certificate off the firewall 
    A) Export the public part (Base64 PEM) >>> certificate.crt
    B) Export the private part (Base64 PEM + select "Export private key") >>> certificate.pem
    You will need to provide a password when selecting the private key
  3. Login to MineMeld via CLI (SSH or VMWare console) and Stop nginx

    $ sudo service nginx stop

  4. Replace the files with a valid certificate and private key
    1. A) First rename the original certificate files:

      $ sudo mv /etc/nginx/minemeld.cer /etc/nginx/minemeld.cer.bck
      $ sudo mv /etc/nginx/minemeld.pem /etc/nginx/minemeld.pem.bck

    2. B) Using WinSCP, copy the newly created certificates to the "/tmp" folder
      Then copy the new certificates to the NGINX directory:

      $ sudo cp /tmp/xxx.crt /etc/nginx/minemeld.cer
      $ sudo cp /tmp/xxx.pem /etc/nginx/minemeld.pem

    3. C) Now restart the MGINX and enter the password for private key (created in step 2.B)

      $ sudo service nginx start

 

This works perfectly.

We can now see that the MineMeld is using the new certificate and connects to the PaloAlto.

 

 

BUT

If now we shutdown the MineMeld, either via de CLI (sudo shutdown) or just a hard power off in VMWare,

If we then boot the appliance again it will not boot anymore but get stuck at:

 

2017-11-09 13_35_21-New notification.png

 

You can still login using CLI:

But the cli keeps displaying “initializing minemeld”…..

 2017-11-09 14_18_15-Photos.png

 

If we press CTRL+C we cancel this and can access the CLI

ubuntu@minemeld:~$ sudo service nginx status

* nginx is not running

 

If I then do sudo service nginx start and enter the PEM pass phrase again, the service run again

 

ubuntu@minemeld:~$ sudo service nginx start

Enter PEM pass phrase:

ubuntu@minemeld:~$ sudo service nginx status

* nginx is running

 

Now we can access the GUI again, BUT

When I try to login via GUI I get following errors even before I try to enter credentials, just opening the page

 

2017-11-09 14_24_14-Settings.png

 

Then when I try my credentials

Even the default credentials….

 

2017-11-09 14_24_37-Settings.png 

 

The only way to resolve this is to delete the appliance and start all over again…

 

Is this something known bug?
Are we doing something  wrong?

Can you help us?

 

This issue can easily be reproduced using the steps above.

I have a “broken” minemeld running in our VM if you need more info / files.

 

Also, a backup of the config taken via de GUI does not include the new certificate installed. Is there a way to export all the config, including the IP config, cert, …  so we can restore the machines faster / easier?

L4 Transporter

Re: MineMeld hanging after reboot

Luigi was so kind as to have a webex and fix the issue:

 

The problem:

The certificate.pem file has a password on it -> this breaks the MineMeld on reboot

(if you used a certificate.pem file without password, you are safe)

 

Solution
1) remove the password from the certificate.pem file before importing it!!!

2) if already imported (and device not rebooted yet, so still working), you need to re-import the certificate.pem file WITHOUT the password, restart the service and reboot the MineMeld. Using the same procedure as in the install guide I provided.

3) If device was rebooten and is now in a hanging state as described in my PA community article you can still access the device using SSH, re-import new certificate without password, restart service, reboot. All other config will be retained!

 

PS: You can use openssl to remove the password (PaloAlto will not let you export the certifcate.pem file without a password on it)

openssl rsa -in minemeld.pem -out minemeld2.pem

Like what you see?

Show your appreciation!

Click Like if a post is helpful to you or if you just want to show your support.

Click Accept as Solution to acknowledge that the answer to your question has been provided.

The button appears next to the replies on topics you’ve started. The member who gave the solution and all future visitors to this topic will appreciate it!

These simple actions take just seconds of your time, but go a long way in showing appreciation for community members and the Live Community as a whole!

The Live Community thanks you for your participation!