What's new in MineMeld 0.9.32

[lmori]

Release Date: 2017-02-06

Changes to the default behavior

• To avoid data corruption, MineMeld engine now periodically checks availble disk space. If the available disk space falls below the limit of 10MB per configured node, the engine shuts down and refuses to start. If after the update the engine does not start you can try to free disk space by purging the logs with the button under SYSTEM > DASHBOARD and then click RESTART under the ENGINE section of SYSTEM > DASHBOARD. The amount of required available disk space per node is configurable via the MM_DISK_SPACE_PER_NODE environment variable.
• If the available disk space falls below 70% of the disk capacity, the tracing subsystem stops writing the logs and a warning appears under SYSTEM > DASHBOARD. As mentioned before, you can purge logs to free some disk space. When disk space becomes available again the tracing subsystem will start logging again, no restart is needed. The threshold can be configured in the traced.yml config file.
• Default logs retention for MineMeld engine is now set to 7 days

How to update: Updating MineMeld

NOTE: During the upgrade existing databases will be migrated to a new schema, this may take some minutes.

Cool new stuff

We have added support for external extensions ! with external extensions you can dynamically add new nodes and prototypes to a full MineMeld instance. Nodes and prototypes can be stored in Python wheels or git repos:

UI

• under SYSTEM > DASHBOARD you can purge logs, restart the engine, restart the API subsystem and download the engine and API logs
• under SYSTEM > DASHBOARD you can create a full config backup of the MineMeld instance. As the config may contain confidential data, the backup is generated as an encrypted zip file.
• added notifications for engine status
• now Dashboard, NODES and GRAPH present nodes based on the type of their prototype. No more cool but weird logic based on the numbers of inputs and outputs !

Core

• engine got smarter, when you change the config only the chain of the graph affected by the change is recalculated. Thanks Niels for the idea !
• now you can specify admins with a read-only role using the READ_WRITE config knob in the API config
• API subsystem now produces audit logs for operations affecting MineMeld status and config
• added new indicator types for: User-Agent, process command line, mutex, file names, Windows Registry values
• many bug fixes and stability enhancements

Prototypes

• new aggregators for the additional indicator types

Extensions

• ArcSight CEF Output node (https://github.com/PaloAltoNetworks/minemeld-cef) Thanks @Forseti !
• Youtube Miner (https://github.com/PaloAltoNetworks/youtube-miner)

What's Next

• Upload of client certificate in TAXII Miner (https://github.com/PaloAltoNetworks/minemeld-webui/issues/95)
• Support for CIDRs in output feeds (https://github.com/PaloAltoNetworks/minemeld-core/issues/162)

[andrew.stanton]

I can't find the documentation for creating the Python Wheel?  does the YAML prototype get put into the wheel as well or does the admin still have to manually place the prototype file somewhere?

[andrew.stanton]

@lmori if you could post your .whl somewhere, I could hopefully figure it out.  Learning about creating wheels now.

Could the YAML be included  via package_data or data_files in the setup.py script?

[andrew.stanton]

Nevermind, I found it:

https://github.com/PaloAltoNetworks/youtube-miner

...

[Matt.Collar]

I'm looking to change the threshold in traced.yml ("The threshold can be configured in the traced.yml config file").  What are the arguments to achieve this change?