Update 0.9.46 Broken (at least for me)

[AaronFaby]

Hey all,

So my Minemeld appliance updated to 0.9.46 at some point which broke the system. I was able to temporarily fix the issue by just changing the symlink in /opt/minemeld/engine from 0.9.46 to 0.9.44. Obviously, this is not ideal.

When I look in the 0.9.46 folder I see that there are a lot of files that are missing that are present in the 0.9.44 folder. 0.9.46 only contains bin, lib, and local. Each of those olders are missing items as well.

Running the update script breaks everything again, so I am curious if there is a way to "force" an update that removes the previous update files. Right now the system is running in some kind of Frankenstein mode where it thinks it's on 0.9.46, however it's using the 0.9.44 engine.

Thanks!

Thanks!

[lmori]

Hi @AaronFaby,

update to 0.9.46 seems to be ok, please could you share the output of the update script?

[AaronFaby]

Here's the output of the logs:

2018-06-01 06:25:26,606 INFO:0.9.11 Current status:
2018-06-01 06:25:26,618 INFO:0.9.11 minemeld-engine: current: 0.9.44 latest: 0.9.46
2018-06-01 06:25:26,618 INFO:0.9.11 minemeld-webui: current: 0.9.46 latest: 0.9.46
2018-06-01 06:25:26,618 INFO:0.9.11 minemeld-prototypes: current: 0.9.46 latest: 0.9.46
2018-06-01 06:25:26,619 INFO:0.9.11 Removing stale version 0.9.46 of package minemeld-engine
2018-06-01 06:25:27,368 DEBUG:0.9.11 curl output: 
2018-06-01 06:25:27,428 DEBUG:0.9.11 curl output: 
2018-06-01 06:25:27,428 DEBUG:0.9.11 gpgv: /usr/bin/gpgv --ignore-time-conflict --keyring /etc/apt/trusted.gpg --keyring /etc/apt/trusted.gpg.d/minemeld.gpg /tmp/mmaupackagesgpgzC4atn /tmp/mmaupackagesPx7ten
2018-06-01 06:25:27,433 INFO:0.9.11 gpgv output: gpgv: Signature made Thu 10 May 2018 09:29:19 AM UTC using RSA key ID 7B630999
gpgv: Good signature from "Palo Alto Networks, MineMeld Team <minemeld@paloaltonetworks.com>"
gpgv:                 aka "[invalid image]"

2018-06-01 06:25:27,437 INFO:0.9.11 Marking minemeld-engine 0.9.46 for deployment
2018-06-01 06:25:27,437 INFO:0.9.11 Space available on /opt/minemeld device: 26330276KB
2018-06-01 06:25:27,437 INFO:0.9.11 Downloading package minemeld-engine version 0.9.46 from http://minemeld-updates.panw.io/stage2/0_9/minemeld-core-0.9.46-linux-x86_64.tar.gz
2018-06-01 06:25:31,526 DEBUG:0.9.11 curl output: 
2018-06-01 06:25:31,526 INFO:0.9.11 Package minemeld-engine version 0.9.46 downloaded in /tmp/minemeld-engineQDa6uy
2018-06-01 06:25:32,244 INFO:0.9.11 New extensions frigidaire created
Setting up virtualenv
New python executable in /opt/minemeld/engine/0.9.46/bin/python
Installing setuptools, pip, wheel...done.
Installing minemeld-core-0.9.46
Could not install packages due to an EnvironmentError: [Errno 28] No space left on device: '/tmp/pip-install-Ujs8Xp/networkx/networkx/linalg'

Looking in links: /tmp/minemeld-engineNMOmBJ/data
Requirement already satisfied: wheel in /minemeld-data/minemeld/engine/0.9.46/lib/python2.7/site-packages (0.31.1)
Collecting minemeld-core
Collecting Flask==0.11 (from -r /tmp/minemeld-engineNMOmBJ/data/requirements.txt (line 1))
Collecting gunicorn==19.3.0 (from -r /tmp/minemeld-engineNMOmBJ/data/requirements.txt (line 2))
Collecting psutil==3.1.1 (from -r /tmp/minemeld-engineNMOmBJ/data/requirements.txt (line 3))
Collecting Flask-Login==0.2.11 (from -r /tmp/minemeld-engineNMOmBJ/data/requirements.txt (line 4))
Collecting passlib==1.6.5 (from -r /tmp/minemeld-engineNMOmBJ/data/requirements.txt (line 5))
Collecting rrdtool==0.1.2 (from -r /tmp/minemeld-engineNMOmBJ/data/requirements.txt (line 6))
Collecting supervisor==3.1.3 (from -r /tmp/minemeld-engineNMOmBJ/data/requirements.txt (line 7))
Collecting redis==2.10.5 (from minemeld-core)
Collecting libtaxii==1.1.107 (from minemeld-core)
Collecting pytz==2015.4 (from minemeld-core)
Collecting jmespath==0.7.1 (from minemeld-core)
Collecting filelock==2.0.4 (from minemeld-core)
Collecting stix==1.1.1.6 (from minemeld-core)
Collecting pan-python==0.10.0 (from minemeld-core)
Collecting certifi (from minemeld-core)
Collecting ujson==1.34 (from minemeld-core)
Collecting unicodecsv==0.14.1 (from minemeld-core)
Collecting requests[security]==2.7.0 (from minemeld-core)
Collecting antlr4-python2-runtime==4.5.2 (from minemeld-core)
Collecting sleekxmpp==1.3.1 (from minemeld-core)
Collecting PyYAML==3.11 (from minemeld-core)
Collecting gevent==1.0.2 (from minemeld-core)
Collecting hiredis==0.2.0 (from minemeld-core)
Collecting beautifulsoup4==4.4.1 (from minemeld-core)
Collecting netaddr==0.7.18 (from minemeld-core)
Collecting lz4==0.8.2 (from minemeld-core)
Collecting click==4.1 (from minemeld-core)
Requirement already satisfied: pip>=9.0.1 in /minemeld-data/minemeld/engine/0.9.46/lib/python2.7/site-packages (from minemeld-core) (10.0.1)
Collecting amqp==1.4.6 (from minemeld-core)
Collecting greenlet==0.4.7 (from minemeld-core)
Collecting cifsdk==2.0.0b7 (from minemeld-core)
Collecting networkx==1.11 (from minemeld-core)
Done.
2018-06-01 06:25:37,162 INFO:0.9.11 New constraints file created
2018-06-01 06:25:37,485 INFO:0.9.11 Extensions reinstalled
2018-06-01 06:25:37,485 INFO:0.9.11 New constraints file moved in place
2018-06-01 06:25:37,491 INFO:0.9.11 Successfully deployed package minemeld-engine version 0.9.46
 * Stopping: minemeld
minemeld-supervisord-listener: stopped
minemeld-traced: stopped
minemeld-web: stopped
minemeld-engine: stopped
   ...done.
2018-06-01 06:25:43,023 INFO:0.9.11 Package minemeld-engine current version set to 0.9.46

The interesting line is here:

Could not install packages due to an EnvironmentError: [Errno 28] No space left on device: '/tmp/pip-install-Ujs8Xp/networkx/networkx/linalg'

However, there seems to be plenty of disk space on the server so I'm not sure what's wrong here:

Filesystem      Size  Used Avail Use% Mounted on
udev            996M   12K  996M   1% /dev
tmpfs           201M  536K  200M   1% /run
/dev/sda1       9.9G  5.1G  4.4G  55% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
none            5.0M     0  5.0M   0% /run/lock
none           1001M  4.0K 1001M   1% /run/shm
none            100M     0  100M   0% /run/user
/dev/sdb         30G  2.7G   26G  10% /minemeld-data

I can write files to /tmp with no issues. Any ideas?

Thanks!

[lmori · Accepted solution]

Hi @AaronFaby,

during the installation packages are downloaded on the tmp directory and extracted.

Could you try pointing the TMPDIR environment variable to a directory with more space available?

Thanks,

luigi

[AaronFaby]

That worked. I'm not sure why as df showed a ton of space available, but I used TMPDIR to change it to another partition and it worked. Appreciate the help.