MineMeld Articles

Featured Article
The set of config parameters supported by a node depends on the node class. Node configs are stored inside prototypes.   Base class All nodes support these parameters.   Parameters infilters: inbound filter set. Filters to apply to received indicators. outfilters: outbound filter set. Filters to apply to transmitted indicators.   Filter set Each filter set is a list of filters. Filters are checked from top to bottom, the first matching filter is applied and following filters are not checked. Default action is accept. Each filter is a dictionary with 3 keys:   name: name of the filter. conditions: list of boolean expressions to match on the indicator and indicator value. actions: list of actions to apply to the indicator. Currently the only supported actions are accept and drop   In addition to the atttributes in the indicator value, filters can match on 3 special attributes: __indicator: the indicator itself. __method: the method of the message, update or withdraw. __origin: the name of the node who sent the indicator.   Condition A condition in the filter is a boolean expression composed by: a JMESPath expression, an operator (<, <=, ==, >=, >, !=) and a value.   Example Example config in YAML: infilters: - name: accept withdraws conditions: - __method == 'withdraw' actions: - accept - name: accept URL conditions: - type == 'URL' actions: - accept - name: drop all actions: - drop outfilters: - name: accept all (default) actions: - accept   Base poller class In addition to Base class config parameters, the base poller class supports the following parameters.   Config parameters source_name: name of the source. This is added to the sources attribute of the generated indicators. Default: name of the node. attributes: dictionary of attributes for the generated indicators. This dictionary is used as template for the value of the generated indicators. Default: empty interval: polling interval in seconds. Default: 3600. num_retries: how many times the miner should try to reach the source in case of failure. If this number is exceeded, the miner waits until the next polling time to try again. Default: 2 age_out: age out policies to apply to the indicators. Default: age out check interval 3600 seconds, sudden death enabled, default age out interval 30 days.   Age out policy Age out policy is described by a dictionary with at least 3 keys:   interval: number of seconds between successive age out checks. sudden_death: boolean, if true indicators are immediately aged out when they disappear from the feed. default: age out interval. After this interval an indicator is aged out even if it is still present in the feed. If null, no age out interval is applied.   Additional keys can be used to specify age out interval per indicator type.   Age out interval Age out intervals have the following format: <base attribute>+<interval> base attribute can be last_seen, if the age out interval should be calculated based on the last time the indicator was found in the feed, or first_seen, if instead the age out interval should be based on the time the indicator was first seen in the feed. If not specified first_seen is used. interval is the length of the interval expressed in seconds. Suffixes d, h and m can be used to specify days, hours or minutes.   Example Example config in YAML for a feed where indicators should be aged out only when they are removed from the feed:   source_name: example.persistent_feed interval: 600 age_out: default: null sudden_death: true interval: 300 attributes: type: IPv4 confidence: 100 share_level: green direction: inbound   Example config in YAML for a feed where indicators are aged out when they disappear from the feed and 30 days after they have seen for the first time in the feed:   source_name: example.long_running_feed interval: 3600 age_out: default: first_seen+30d sudden_death: true interval: 1800 attributes: type: URL confidence: 50 share_level: green   Example config in YAML for a feed where indicators are aged 30 days after they have seen for the last time in the feed:   source_name: example.delta_feed interval: 3600 age_out: default: last_seen+30d sudden_death: false interval: 1800 attributes: type: URL confidence: 50 share_level: green   minemeld.ft.http.HttpFT class In addition to Base poller class config parameters, the HttpFT class supports the following parameters:   Parameters url: URL of the feed. polling_timeout: timeout of the polling request in seconds. Default: 20 verify_cert: boolean, if true feed HTTPS server certificate is verified. Default: true ignore_regex: Python regular expression for lines that should be ignored. Default: null indicator: an extraction dictionary to extract the indicator from the line. If null, the text until the first whitespace or newline character is used as indicator. Default: null fields: a dicionary of extraction dictionaries to extract additional attributes from each line. Default: {}   Extraction dictionary Extraction dictionaries contain the following keys:   regex: Python regular expression for searching the text. transform: template to generate the final value from the result of the regular expression. Default: the entire match of the regex is used as extracted value.   See Python re module for details about Python regular expressions and templates.   Example Example config in YAML where extraction dictionaries are used to extract the indicator and additional fields: url: https://www.dshield.org/block.txt ignore_regex: "[#S].*" indicator: regex: '^([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})\t([0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3}\.[0-9]{1,3})' transform: '\1-\2' fields: dshield_nattacks: regex: '^.*\t.*\t[0-9]+\t([0-9]+)' transform: '\1' dshield_name: regex: '^.*\t.*\t[0-9]+\t[0-9]+\t([^\t]+)' transform: '\1' dshield_country: regex: '^.*\t.*\t[0-9]+\t[0-9]+\t[^\t]+\t([A-Z]+)' transform: '\1' dshield_email: regex: '^.*\t.*\t[0-9]+\t[0-9]+\t[^\t]+\t[A-Z]+\t(\S+)' transform: '\1'   Example config in YAML where the text in each line until the first whitespace is used as indicator: url: https://ransomwaretracker.abuse.ch/downloads/CW_C2_URLBL.txt ignore_regex: '^#'   For a complete config example check dshield.block prototype.   minemeld.ft.csv.CSVFT class In addition to Base poller class config parameters, the CSVFT class supports the following parameters:   Parameters url: URL of the feed. polling_timeout: timeout of the polling request in seconds. Default: 20 verify_cert: boolean, if true feed HTTPS server certificate is verified. Default: true ignore_regex: Python regular expression for lines that should be ignored. Default: null fieldnames: list of field names in the file. If null the values in the first row of the file are used as names. Default:null delimiter: see csv Python module. Default: , doublequote: see csv Python module. Default: true escapechar: see csv Python module. Default: null quotechar: see csv Python module. Default: " skipinitialspace: see csv Python module. Default: false   Example Example config in YAML: url: https://sslbl.abuse.ch/blacklist/sslipblacklist.csv ignore_regex: '^#' fieldnames: - indicator - port - sslblabusech_type   For a complete config example check sslabusech.ipblacklist prototype.   minemeld.ft.json.SimpleJSON class In addition to Base poller class config parameters, the SimpleJSON class support the following parameters:   Parameters url: URL of the feed. polling_timeout: timeout of the polling request in seconds. Default: 20 verify_cert: boolean, if true feed HTTPS server certificate is verified. Default: true extractor: JMESPath expression for extracting the indicators from the JSON document. Default: @ indicator: the JSON attribute to use as indicator. Default: indicator fields: list of JSON attributes to include in the indicator value. If null no additional attributes are extracted. Default: null prefix: prefix to add to field names. Default: json   Example Example config in YAML: url: https://ip-ranges.amazonaws.com/ip-ranges.json extractor: "prefixes[?service=='AMAZON']" prefix: aws indicator: ip_prefix fields: - region - service   For a complete config example check aws.AMAZON prototype.
View full article
lmori ‎04-28-2016 03:41 AM
16,701 Views
2 Replies
4 Likes
By default the MineMeld loader installs in the VM a scheduled job inside the VM to automatically check and install the latest version of the MineMeld packages. The scheduled auto update job runs once per day.   To force a an update just run the minemeld-auto-update utility: $ /usr/sbin/minemeld-auto-update   To disable the auto update job check the article Disabling MineMeld auto update.  
View full article
lmori ‎02-04-2016 05:39 AM
20,894 Views
11 Replies
MineMeld can analyze PAN-OS syslog messages and match them against live indicators. In case of match, the result of the match (full session info and full details of the indicator) can be sent to a logstash (https://www.elastic.co/products/logstash) instance for forwarding to an external database.   1. Message flow The following diagram illustrates the message flow of the syslog analysis process. NOTE: while rsyslog is already installed and configured by the MineMeld loader on the VM, logstash is not installed by default. To save matching results in an external collector logstash must be installed and the syslog should be configured to send results to the logstash instance.   2. Adding a syslog analyzer node Under CONFIG, click + to add a new node. Select prototype stdlib.localSyslog if you don't need to send results to an external logstash instance, select prototype stdlib.localSyslogToLogstash if you plan to send results to a local logstash instance. Select an aggregator as input.   NOTE: syslog analyzer node supports only one aggregator per type as upstream node. If you have multiple aggregators, you can use multiple syslog analysis nodes or create an additional generic aggregator just for syslog analyzer.     3. Configuring syslog forwarding on PAN-OS Please refer to the PAN-OS Administration guide for instructions on how to configure log forwarding to a syslog server on PAN-OS (https://www.paloaltonetworks.com/documentation/70/pan-os/pan-os/monitoring/configure-log-forwarding.html#24849). By defeault the syslog server on the VM listens for PAN-OS syslog messages on port tcp/13514. 4. Checking counters on the syslog node Under NODES, click on the syslog analyzer node. Select STATS in the left menu, and check the counters.   SYSLOG.PROCESSED is the number of syslog messages received from PAN-OS devices and processed. TOTAL_MATCHES is the number of matching indicators found. LOGSTASH.SENT is the number of results sent to the logstash instance (if configured).   5. Evaluating indicators sources Under NODES, click on the syslog analyzer node. Select SOURCES in the left menu to see the which feeds have produced matching indicators.   6. Configuring logstash (optional) If at step 2 you have chosen prototype stdlib.localSyslogToLogstash, the syslog analyzer node will send matching results to logstash on port 5514/tcp. Use the following logstash snippet to let logstash listen on port 5514 for messages from MineMeld syslog nodes. input { tcp { port => 5514 host => '127.0.0.1' codec => 'json_lines' } } NOTE: logstash is not installed by default on the VM. Please refer to the following link for instructions on how to install logstash and an ELK stack on the VM or on an external VM: - official site https://www.elastic.co/ - step by step tutorial from DigitalOcean community https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elk-stack-on-ubuntu-14-04   Kibana view of the matching results:
View full article
lmori ‎02-04-2016 02:53 AM
15,274 Views
14 Replies
The default config provides a graph for handling inbound IPv4 Threat Indicators, IPv4 addresses related to suspicious inbound activities like brute forcing on scanning. In this article we will add a new Miner to this graph.   1. Adding a Miner   Click on CONFIG in the top navigation bar.     And press + to add the node.   2. Configuring the Miner   Select the PROTOTYPE, and leave the INPUTS field empty. Enable the OUTPUT. Press OK when done.   Note. if you leave the pointer on a prototype a tooltip appears with the description of the prototype.     3. Linking the Miner to the aggregator   Now you have created a new Miner in the candidate config, but the Miner is not linked to any downstream node.       Click on the INPUTS field of the inboundaggregator node and add the new Miner to the list. Press OK when done.     4. Commit the config   Now you should have a new Miner connected to the inboundaggregator.     Press COMMIT to apply the config.   5.  Check the engine status   Click SYSTEM in the top navigation bar to check the engine status. It should stop and then start.       The processing graph after the change should look like this:  
View full article
lmori ‎02-03-2016 02:02 AM
5,914 Views
2 Replies
The simple, default config included in MineMeld VM creates a graph to process IPv4 indicators for inbound connections, typically used to filter out scanning hosts or well known brute force attackers. For IPv4 indicators for outbound connections we can define a new sub-graph with its own set of output feeds. These new set of feeds can then be used in the destination part of the PAN-OS security policies.   1. Adding an outbound IPv4 aggregator Under CONFIG press +. Configure a new node with prototype stdlib.aggregatorIPv4Outbound and Output enabled.   2. Adding a set of feeds Under CONFIG add 3 new nodes (HC, MC and LC) for the output feeds and select the node created at point 1 as Input.   First node with stdlib.feedHCGreenWithValue as prototype   Second node with stdlib.feedMCGreenWithValue as prototype   Third node with stdlib.feedMCGreenWithValue as prototype   3. Adding a Miner Under CONFIG add a new Miner generating IPv4 outbound indicators, like zeustracker.badips. Output should be enabled.   4. Connecting the aggregator to the Miner Under CONFIG, click on the INPUTS field of the node created at step 1 and add the Miner.   5. Commit Check the resulting config and press COMMIT.   6. Check the sub-graph The resulting sub-graph should look like this:
View full article
lmori ‎02-03-2016 02:00 AM
3,717 Views
0 Replies
1 Like
In MineMeld you can configure how indicators are retrieved, processed and presented by configuring the MineMeld engine graph. The graph is composed by one or more interconnected nodes.   The definition of a node in the graph is composed by: - Name. A unique node name. - Inputs. A list of nodes the node should receive messages from. - Output. A boolean value that enables/disables messages to downstream nodes. - Class. Defines which kind of processing is applied to indicators. - Config. Configuration of the node class.   Node Inputs A node can have zero or more input nodes. The node will listen to and process messages from the input nodes. Messages between nodes can be used to publish/update indicators, or to withdraw indicators.   In the following graph, the node inboundaggregator has spamhaus_DROP, spamhaus_EDROP and dshield_blocklist as input nodes. If a node has no input nodes it is considered a Miner. In the following graph spamhaus_DROP, spamhaus_EDROP and dshield_blocklist have no input nodes because they are Miners.     Node Output A node can be configured to send messages to downstream connected nodes. In the previous picture spamhaus_DROP, spamhaus_EDROP, dshield_blocklist and inboundaggregator have their Ouput enabled.   Output nodes have their Output disabled - ok, this is a bit misleading - because they don't have downstream nodes. inboundfeedhc, inboundfeedlc, inboundfeedmc are Output nodes and their output is disabled.   Node Class The node class defines what is actually done by the node. In the previous graph spamhaus_DROP, spamhaus_EDROP and dshield_blocklist are all nodes of class minemeld.ft.http.HttpFT, this class of nodes manage feeds of indicators retrievable via URLs and send updates to downstream nodes. The feed URL and how the indicators should be extracted are defined in each node config. inboundaggregator instead is a node of class minemeld.ft.ipop.AggregateIPv4FT. This class of nodes aggregate IPv4 indicators from multiple input nodes and generate a set of non-overlapping IPv4 indicators.   Node Config The node config defines the parameters that should be used by the node class to perform its job. As an example, for the node dshield_blocklist the config defines the URL that should be used to retrieve the list of indicators, the patterns to extract the indicators and the age out policies to apply.   Node Prototype MineMeld comes with a library of node prototypes. A prototype is a semi-pre-canned node definition, and it is composed by a node class and a node config. Prototypes are available for the most commonly used feeds, processors and output nodes.   As an example, this is the zeustracker.badips prototype:   badips: development_status: STABLE node_type: miner config: source_name: zeustracker.badips attributes: type: IPv4 direction: outbound confidence: 100 share_level: green ignore_regex: '#' url: https://zeustracker.abuse.ch/blocklist.php?download=badips interval: 1800 class: minemeld.ft.http.HttpFT  
View full article
lmori ‎02-03-2016 01:58 AM
7,189 Views
0 Replies
2 Likes
Ask Questions Get Answers Join the Live Community