MineMeld hanging after reboot

[mr.linus]

Dear Luigi,

We are having a lot of issues with MineMeld, I hope you can help.

We are deploying MineMeld with our customers on their own vmware environment in a DMZ zone with NO DHCP.

I know you prefer to use AutoFocus, MineMeld in AWS, MineMeld on Ubuntu or MineMeld with DHCP, but this is not always feasible for our customers.

In any case here is the setup we follow:

Installation

1. Download the fast-install OVA file. This has the needed iso-file bundled in
   https://s3-eu-west-1.amazonaws.com/minemeld-dist/0_9/minemeld-vm-0.9.10-1build1.ova
2. Deploy ova in VMWare
   All settings can be kept there default values
3. Boot the MineMeld instance and let it fail the initialization. It will take some time as it searches in vain for a DHCP server
4. Login to the console of the Virtual Machine with credentials:
   username: ubuntu
   password: rsplizardspock
5. Configure a static IP for your MineMeld

   $ sudo vi /etc/network/interfaces.d/eth0.cfg

   auto eth0
   iface eth0 inet static
   address x.x.x.x
   netmask x.x.x.x
   gateway x.x.x.x
   dns-nameservers x.x.x.x

   Press "esc" and ":wq" to save the file in vi.

6. Then enter following command. This will reset a value so MineMeld is correctly installed on next reboot.

   $ sudo rm -rf /var/lib/cloud/*

7. Reboot the device

   $ sudo shutdown -r now

8. The instance will reboot and (re)start the MineMeld installation.
9. Installation is now done. You can access the Gui
   https://x.x.x.x

This works perfectly.

After install we can access the Gui

We can reboot the appliance via CLI, or just in VMWare give it a hard power-off and start up -> the MineMeld will run again

NEXT

Since PanOS 7.1 and higher we need to install a custom certificate on the MineMeld to get it to connect

For this we follow the following steps

1. Create a new self-signed CA certificate on the Palo Alto firewall for MineMeld
2. Export both the public and private certificate off the firewall 
   A) Export the public part (Base64 PEM) >>> certificate.crt
   B) Export the private part (Base64 PEM + select "Export private key") >>> certificate.pem
   You will need to provide a password when selecting the private key
3. Login to MineMeld via CLI (SSH or VMWare console) and Stop nginx

   $ sudo service nginx stop

4. Replace the files with a valid certificate and private key
1. A) First rename the original certificate files:

   $ sudo mv /etc/nginx/minemeld.cer /etc/nginx/minemeld.cer.bck
   $ sudo mv /etc/nginx/minemeld.pem /etc/nginx/minemeld.pem.bck

2. B) Using WinSCP, copy the newly created certificates to the "/tmp" folder
   Then copy the new certificates to the NGINX directory:

   $ sudo cp /tmp/xxx.crt /etc/nginx/minemeld.cer
   $ sudo cp /tmp/xxx.pem /etc/nginx/minemeld.pem

3. C) Now restart the MGINX and enter the password for private key (created in step 2.B)

   $ sudo service nginx start

This works perfectly.

We can now see that the MineMeld is using the new certificate and connects to the PaloAlto.

BUT

If now we shutdown the MineMeld, either via de CLI (sudo shutdown) or just a hard power off in VMWare,

If we then boot the appliance again it will not boot anymore but get stuck at:

You can still login using CLI:

But the cli keeps displaying “initializing minemeld”…..

 

If we press CTRL+C we cancel this and can access the CLI

ubuntu@minemeld:~$ sudo service nginx status

* nginx is not running

If I then do sudo service nginx start and enter the PEM pass phrase again, the service run again

ubuntu@minemeld:~$ sudo service nginx start

Enter PEM pass phrase:

ubuntu@minemeld:~$ sudo service nginx status

* nginx is running

Now we can access the GUI again, BUT

When I try to login via GUI I get following errors even before I try to enter credentials, just opening the page

Then when I try my credentials

Even the default credentials….

 

The only way to resolve this is to delete the appliance and start all over again…

Is this something known bug?
Are we doing something  wrong?

Can you help us?

This issue can easily be reproduced using the steps above.

I have a “broken” minemeld running in our VM if you need more info / files.

Also, a backup of the config taken via de GUI does not include the new certificate installed. Is there a way to export all the config, including the IP config, cert, …  so we can restore the machines faster / easier?

[mr.linus · Accepted solution]

Luigi was so kind as to have a webex and fix the issue:

The problem:

The certificate.pem file has a password on it -> this breaks the MineMeld on reboot

(if you used a certificate.pem file without password, you are safe)

Solution
1) remove the password from the certificate.pem file before importing it!!!

2) if already imported (and device not rebooted yet, so still working), you need to re-import the certificate.pem file WITHOUT the password, restart the service and reboot the MineMeld. Using the same procedure as in the install guide I provided.

3) If device was rebooten and is now in a hanging state as described in my PA community article you can still access the device using SSH, re-import new certificate without password, restart service, reboot. All other config will be retained!

PS: You can use openssl to remove the password (PaloAlto will not let you export the certifcate.pem file without a password on it)

openssl rsa -in minemeld.pem -out minemeld2.pem