MineMeld Articles

Featured Article
MineMeld is a low latency processor of indicators feeds. You can think of it as the Swiss army knife of feeds.    
View full article
lmori a week ago
69,974 Views
13 Replies
MineMeld aggregator nodes support whitelists. If an indicator is on a whitelist, the aggregator nodes will not send matching indicators to downstream nodes. Whitelists can also be shared by multiple aggregators.   Aggregator nodes created using stdlib prototypes (stdlib.aggregatorDomain, stdlib.aggregatorURL, stdlib.aggregatorIPv4Generic, stdlib.aggregatorIPv4Outbound, stdlib.aggregatorIPv4Inbound) will whitelist indicators generated by Miner nodes whose name starts with the prefix wl (lowercase).   In the following example, a whitelist Miner will be created for an IPv4 aggregator node.   1. Creating a static whitelist node In CONFIG, click + to add a new node. Specify a name starting with "wl" and select stdlib.listIPv4Generic as prototype. Enable Output and then press OK.     2. Connecting the whitelist to the aggregator In CONFIG, click on the INPUTS field of the selected aggregator. In the dialog add the new whitelist node to the list of INPUTS.   3. Commit the config Just press COMMIT in the CONFIG page.   4. Adding indicators to the whitelist In NODES, click on the new whitelist node and select INDICATORS in the menu on the left.   Click + to add new indicators. Pressing OK will automatically save the indicator and the list. It could take up to 1 minute for the new indicator to be pushed downstream to the aggregator node.  
View full article
lmori 2 weeks ago
11,535 Views
9 Replies
1 Like
An easy way to use MineMeld is installing the binary packages on an Ubuntu 16.04 instance. Note: Binary packages are only available for 64 bits architectures.   Supported distributions Ubuntu Server LTS 16.04 (64 bits)   1. Hardening the instance First thing you should harden your new instance. MineMeld won't take of this for you. A good tutorial is this one.   2. Update the instance Update all the packages of the instance before installing. sudo apt update && sudo apt dist-upgrade -y 3. Setting up iptables You can use the following commands to configure iptables to allow sessions on ports used by MineMeld. Also these rules drop all IPv6 traffic, if you are running MineMeld in an IPv6 network make sure you change the suggested rules.   sudo apt install -y iptables-persistent sudo iptables -A INPUT -i lo -j ACCEPT sudo iptables -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT sudo iptables -A INPUT -p tcp -m tcp --dport 22 -j ACCEPT sudo iptables -A INPUT -p tcp -m tcp --dport 80 -j ACCEPT sudo iptables -A INPUT -p tcp -m tcp --dport 443 -j ACCEPT sudo iptables -A INPUT -p tcp -m tcp --dport 13514 -j ACCEPT sudo iptables -A INPUT -p icmp -m icmp --icmp-type 0 -j ACCEPT sudo iptables -A INPUT -p icmp -m icmp --icmp-type 3 -j ACCEPT sudo iptables -A INPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT sudo iptables -A INPUT -p icmp -m icmp --icmp-type 11 -j ACCEPT sudo iptables -P INPUT DROP sudo iptables -P FORWARD DROP sudo bash -c "iptables-save > /etc/iptables/rules.v4" sudo ip6tables -A INPUT -i lo -j ACCEPT sudo ip6tables -P INPUT DROP sudo ip6tables -P FORWARD DROP sudo bash -c "ip6tables-save > /etc/iptables/rules.v6"   4. Adding the repo GPG key Add the MineMeld repo GPG key to the APT trusted keyring:    wget -qO - https://minemeld-updates.panw.io/gpg.key | sudo apt-key add -   Double check the GPG key fingerprint, to make sure it is matching the official MineMeld GPG key (fingerprint should match characters in bold):   apt-key adv --fingerprint DD0DA1F9 Executing: gpg --ignore-time-conflict --no-options --no-default-keyring --homedir /tmp/tmp.W74MaAG3pI --no-auto-check-trustdb --trust-model always --keyring /etc/apt/trusted.gpg --primary-keyring /etc/apt/trusted.gpg --fingerprint DD0DA1F9 pub 4096R/DD0DA1F9 2016-07-15 Key fingerprint = E558 CE6E 3968 0F31 8F6C BFAC B401 E02E DD0D A1F9 uid Palo Alto Networks, MineMeld Team <minemeld@paloaltonetworks.com> [...]   5. Adding the MineMeld APT repo Add the MineMeld APT repo to the system list and update the apt cache:   sudo add-apt-repository "deb http://minemeld-updates.panw.io/ubuntu xenial-minemeld main" sudo apt update   6. Installing nginx and redis MineMeld requires nginx and redis. We have to install them before installing MineMeld package to avoid configuration conflicts: sudo apt install -y nginx redis-server 7. Installing MineMeld Install the MineMeld infrastructure package via apt. This will also automatically trigger the download of the latest MineMeld packages.    sudo apt install -o Dpkg::Options::="--force-overwrite" -y minemeld 8. Restart We should restart the instance to make sure all the configurations are applied and all the services are started in the right order: sudo shutdown -r now 9. Checking if MineMeld is running Check if the 3 MineMeld services are up and running:   $ sudo -u minemeld /opt/minemeld/engine/current/bin/supervisorctl -c /opt/minemeld/local/supervisor/config/supervisord.conf status minemeld-engine RUNNING pid 3727, uptime 0:08:50 minemeld-traced RUNNING pid 3728, uptime 0:08:50 minemeld-web RUNNING pid 3729, uptime 0:08:50 minemeld-supervisord-listener RUNNING pid 3730, uptime 0:08:50   10. BAM ! Done ! Check the Quick Tour article to get started.
View full article
lmori 2 weeks ago
16,789 Views
21 Replies
2 Likes
This article has been updated to reflect changes to the Azure AD Application registration process and to point users to a new MineMeld output node. The old node will be deprecated.   If you are not familiar with MineMeld, we recommend you start with a Quick Tour.   MineMeld can be used to aggregate multiple threat intelligence feeds and extend to your Windows Defender ATP tenant. Windows Defender ATP can ingest: IPv4 addresses File hashes URLs Domains and FQDNs There are three steps to connecting MineMeld to Windows Defender ATP: Create an application in Azure Active Directory. You will assign scopes from your Windows Defender ATP to this application, and all of the alerts tied to the threat intelligence provided will be tied to this application name. The MineMeld Miner will be associated with this application. Install the Windows Defender extension in MineMeld. Configure the extension to connect to the Windows Defender ATP tenant.   Azure Active Directory Configuration Log in to the Azure Portal (portal.azure.com). Go to Azure Active Directory. Navigate to Enterprise Applications > App Registrations > click New Application Registration.   Create a name for this application. All of the alerts tied to the threat intelligence coming from MineMeld will be attributed to this application name. We recommend calling this "Palo Alto Networks MineMeld" to avoid any confusion. NOTE: You do not need to set a redirect URI.   Click Register. From the Application page, click API Permissions.   Click Add a Permission.   Click APIs my organization uses, type “Windows” in the search bar, and select WindowsDefendertATP.   Click Application Permissions, select “Ti.ReadWrite” and then click Add Permissions.   Grant admin consent.   From the Application page, click Certificates and Secrets.   Click New Client Secret.   Copy the client secret you created.   You will also need to copy the Application ID and Directory ID.     MineMeld Configuration In MineMeld, go under SYSTEM and click the Extensions icon.   Click the GitHub icon in the lower, right-hand corner, then copy this link “https://github.com/PaloAltoNetworks/minemeld-wd-atp.git” and paste into the Repository URL field. Click the dropdown menu for Version and select “master” then click Install.   Click the checkmark to activate the extension.   The extension will activate shortly, and the empty square will signify the extension is active.   You will need to go back to the SYSTEM page and restart the API. NOTE: After the restart completes, make sure you refresh the browser page.     Setting Up the Output Node to Complete the Integration In MineMeld, click CONFIG, then click the Browse Prototype icon.   Type “windows” into the search bar to shorten the list, and select the “microsoft_wd_atp.outputBatch” node. NOTE: The “microsoft_wd_atp.output” node will be deprecated as it relies on an older API interface. Please do not use that node.   Click Clone on the top, right of the page.   Name the cloned node and add the appropriate threat feeds that you want to send to your Windows Defender ATP tenant in the INPUTS nodes section and then click OK. NOTE: To understand the concepts of input nodes and what to connect to this, refer to the MineMeld documentation on LIVEcommunity.   Click the COMMIT button in the top left of the CONFIG page.   Click NODES on the top menu and search for the node you just created. Click the node to pull up the configuration.   In Azure AD, enter the Client ID (Application), Client Secret, and Tenant (Directory) ID you copied earlier when you created the MineMeld application. NOTE: After this is done, your configuration will then be complete.     Testing To validate this is hooked up correctly, you will need to verify that an event fires if you try to access a blocked website. We recommend you create an indicator that is tied to a known good website for this, so you are not actively going to a malicious website.   Click NODES at the top and then click ADD INDICATOR   Enter in a known IP address as an INDICATOR and add it to the Input node (TYPE) you used to configure your microsoft_wd_atp.outputBatch node. Then click OK.   Wait for the indicator to be pushed to your Windows Defender ATP tenant. Then try to load that URL on a client that is running Windows Defender ATP. You should see an event fire in the Windows Defender ATP console.     Additional Information You can find out more information about this capability by reading Pushing custom Indicator of Compromise (IoCs) to Microsoft Defender ATP on the Microsoft website.
View full article
fvigo ‎05-10-2019 12:40 PM
6,569 Views
6 Replies
MineMeld can be used to aggregate multiple threat intelligence feeds and extend to your Microsoft Security products via the Microsoft Graph Security API. Azure Sentinel is one of the first Microsoft Security products to ingest IOCs from the Graph Security API for use in alerting and hunting. The Microsoft Graph Security API supports the following types of Indicators of Compromise (IOCs): Email File IP address URL Domain There are three steps to connecting MineMeld to the Microsoft Graph Security API: First, you will need to create an Application in Azure Active Directory. You will assign permissions to this application to access Microsoft Graph APIs. The MineMeld Output Node will be use the credentials tied to the application you created to connect to the Microsoft Graph. You will then install the Microsoft Graph Security API extension in MineMeld Finally, you will configure the extension to connect to the Microsoft Graph via the Security API.   Azure Active Directory Configuration Login on Azure Portal – portal.azure.com Go to Azure Active Directory Navigate to Enterprise Applications – App Registrations and click on New Application Registration The Application name will not be used or surfaced as part of this integration, but we recommend you name this to Palo Alto Networks MineMeld or something similar so you can use the correct credentials when you setup your MineMeld output node. Set the Application type to “Web App / API” The Sign-on URL is not used, but you still need to put something into this field. Please note, the Application ID and Object ID will be used to configure both the threat feed in your Microsoft Graph Security API tenant and the MineMeld extension.   Under Settings go to “Required Permissions” and click on “Add” Select Microsoft Graph as an API. You can find this by typing “graph” in the search box. Under “Application Permissions “select “Manage threat indicators this app creates or owns” Click “Select” and then click “Done” Click “Grant permissions” and click “Yes” Under “App Registrations” in the app, under “Settings” go to “Keys” and create a new key with an expiration date. Click on “Save” and copy the value of the key and save it in your notes Copy the “Application ID” that you will need later Copy the Tenant ID from Azure Active Directory Properties (Directory ID)   MineMeld Configuration On MineMeld, go under “System” and “Extensions”. Glick on the Git icon Put the URL of the Github repo: https://github.com/PaloAltoNetworks/minemeld-msgraph-secapi.git Select the version (Master) and click “Install” The installation should complete shortly afterwards: Click on the enable button and confirm: The extension will activate shortly, and the API gateway will resent as part of this activation.   Integration Configuration In MineMeld, go to Config - Prototypes In the search box type “Microsoft”. Find “Microsoft_graph_secapi.output” and click on it. Click on “Clone”   Name for the node and collect it to the Input nodes you want it to use (Threat Intel feeds). NOTE: to understand the concepts of input nodes and what to connect to this, refer to Minemeld documentation Click OK Click on Commit Under Nodes, select the node you created (MicrosoftGraphSecAPI in this example) and look at the SETTINGS page. Edit Client_ID (Azure Application ID), client_secret (Azure Application secret key) and Tenant_ID (Azure Active Directory ID) Select the Microsoft services you want to share this Threat Intelligence with by clicking on “Target Product”. NOTE: As of April 2019, Azure Sentinel is the only service capable of consuming third-party threat intelligence. Testing Azure Sentinel can be used to validate this is setup correctly. Please review these instructions for turning on Threat Intelligence in Azure Sentinel. NOTE: The MineMeld extension currently specifies the Azure Sentinel service, so that is already done for you. Once you have this setup, you can review the indicators in the logs section:
View full article
‎04-30-2019 07:52 PM
1,254 Views
0 Replies
1 Like
WARNING: This article is out of date Please use follow this article to install MineMeld on a recent Ubuntu 16.04 distribution: https://live.paloaltonetworks.com/t5/MineMeld-Articles/Manually-install-MineMeld-on-Ubuntu-Server-16-04/ta-p/253336     Click on the following link to create a new deployment of the  MineMeld ARM Template in your Microsoft Azure account:   Deploy To Azure   Accessing the instance   After the template has been fully deployed, an INSTANCEWEBUI field appears in the Outputs section of the template results. The address in the field is the FQDN of the instance.     During initialization you can connect to the instance only via SSH using the credentials supplied via the template parameters. In this phase the loader is updating the Ubuntu packages, and downloading & installing the latest version of MineMeld. This phase should take a couple of minutes:   $ ssh ubuntu@mmdemo.westeurope.cloudapp.azure.com The authenticity of host 'mmdemo.westeurope.cloudapp.azure.com (40.113.113.154)' can't be established. RSA key fingerprint is 7e:8a:71:d5:c9:83:1b:a0:a6:0b:ca:c5:dc:de:75:54. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'mmdemo.westeurope.cloudapp.azure.com,40.113.113.154' (RSA) to the list of known hosts. ubuntu@mmdemo.westeurope.cloudapp.azure.com's password: Welcome to Ubuntu 14.04.2 LTS (GNU/Linux 3.16.0-43-generic x86_64) [...] Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. Initalizing minemeld. It could take some minutes, please wait .....|....   At the end of the initialization you should see a message where all the minemeld services are RUNNING:   Initalizing minemeld. It could take some minutes, please wait ..|....... DONE ! MINEMELD STATUS: minemeld-engine RUNNING pid 43691, uptime 0:00:08 minemeld-traced RUNNING pid 43692, uptime 0:00:08 minemeld-web RUNNING pid 43693, uptime 0:00:08 ubuntu@vm1:~$   Accessing the WebUI   After the initialization is completed, the WebUI can be accessed on port 443 via https with default credentials Username: admin Password: minemeld  
View full article
lmori ‎03-12-2019 07:27 AM
15,394 Views
3 Replies
WARNING: This article is out of date Please use follow this article to install MineMeld on a recent Ubuntu 16.04 distribution: https://live.paloaltonetworks.com/t5/MineMeld-Articles/Manually-install-MineMeld-on-Ubuntu-Server-16-04/ta-p/253336     If for some reasons the supplied MineMeld cloud-init loaders for VMWare, EC2 and Azure could not work in your environment, you can fall back to the good ol' manual installation of MineMeld.   Supported distributions Ubuntu Server LTS 14.04   1. Hardening the instance First thing you should harden your new instance. MineMeld won't take of this for you. A good tutorial is this one.   2. Setting up iptables You can use the following commands to configure iptables to allow sessions on ports used by MineMeld. Port 13514/tcp is for sending syslog messages to the MineMeld instance and it is optional. Also these rules drop all IPv6 traffic, if you are running MineMeld in an IPv6 network make sure you change the suggested rules.   sudo apt-get update && sudo apt-get install -y iptables-persistent sudo iptables -A INPUT -i lo -j ACCEPT sudo iptables -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT sudo iptables -A INPUT -p tcp -m tcp --dport 22 -j ACCEPT sudo iptables -A INPUT -p tcp -m tcp --dport 80 -j ACCEPT sudo iptables -A INPUT -p tcp -m tcp --dport 443 -j ACCEPT sudo iptables -A INPUT -p tcp -m tcp --dport 13514 -j ACCEPT sudo iptables -A INPUT -p icmp -m icmp --icmp-type 0 -j ACCEPT sudo iptables -A INPUT -p icmp -m icmp --icmp-type 3 -j ACCEPT sudo iptables -A INPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT sudo iptables -A INPUT -p icmp -m icmp --icmp-type 11 -j ACCEPT sudo iptables -P INPUT DROP sudo iptables -P FORWARD DROP sudo bash -c "iptables-save > /etc/iptables/rules.v4" sudo ip6tables -A INPUT -i lo -j ACCEPT sudo ip6tables -P INPUT DROP sudo ip6tables -P FORWARD DROP sudo bash -c "ip6tables-save > /etc/iptables/rules.v6"   3. Adding the repo GPG key Add the MineMeld repo GPG key to the APT trusted keyring:    wget -qO - https://minemeld-updates.panw.io/gpg.key | sudo apt-key add -   Double check the GPG key fingerprint, to make sure it is matching the official MineMeld GPG key (fingerprint should match characters in bold):   $ apt-key adv --fingerprint DD0DA1F9 Executing: gpg --ignore-time-conflict --no-options --no-default-keyring --homedir /tmp/tmp.W74MaAG3pI --no-auto-check-trustdb --trust-model always --keyring /etc/apt/trusted.gpg --primary-keyring /etc/apt/trusted.gpg --fingerprint DD0DA1F9 pub 4096R/DD0DA1F9 2016-07-15 Key fingerprint = E558 CE6E 3968 0F31 8F6C BFAC B401 E02E DD0D A1F9 uid Palo Alto Networks, MineMeld Team <minemeld@paloaltonetworks.com> [...]   4. Adding the MineMeld APT repo Add the MineMeld APT repo to the system list:   sudo add-apt-repository "deb http://minemeld-updates.panw.io/ubuntu trusty-minemeld main"   5. Installing MineMeld Install the MineMeld infrastructure package via apt-get. This will also automatically trigger the download of the latest MineMeld packages.    sudo apt-get update && sudo apt-get install -y minemeld rsyslog-minemeld rsyslog-mmnormalize   6. Checking if MineMeld is running Check if the 3 MineMeld services are up and running:   $ sudo -u minemeld /opt/minemeld/engine/current/bin/supervisorctl -c /opt/minemeld/local/supervisor/config/supervisord.conf status minemeld-engine RUNNING pid 3727, uptime 0:08:50 minemeld-traced RUNNING pid 3728, uptime 0:08:50 minemeld-web RUNNING pid 3729, uptime 0:08:50   7. BAM ! Done ! Check the Quick Tour article to get started.
View full article
lmori ‎03-12-2019 07:26 AM
86,096 Views
39 Replies
5 Likes
WARNING: This article is out of date Please use follow this article to install MineMeld on a recent Ubuntu 16.04 distribution: https://live.paloaltonetworks.com/t5/MineMeld-Articles/Manually-install-MineMeld-on-Ubuntu-Server-16-04/ta-p/253336     To run a basic set up of MineMeld on Amazon EC2 you can use CloudFormation Launch URLs that will automatically create a new instance in your region of choice with some default settings, or create a new Ubuntu 14.04 LTS instance and specify a URL to load the user data from.   Using CloudFormation Launch URLs Use the following CloudFormation Launch URLs to spin up a new Ubuntu instance with the MineMeld loader in your region of choice:   EC2 Region CloudFormation ap-northeast-1 Launch URL ap-southeast-1 Launch URL ap-southeast-2 Launch URL eu-central-1 Launch URL eu-west-1 Launch URL sa-east-1 Launch URL us-east-1 Launch URL us-west-1 Launch URL us-west-2 Launch URL us-gov-west-1 Launch URL   Using EC2 User Data The following link contains a base64 encoded cloud-init file that can be used to automatically load MineMeld on new instances of Ubuntu 14.04 LTS. Just specify the link in the Advanced Settings while creating a new instance. https://s3-eu-west-1.amazonaws.com/minemeld-dist/0_9/minemeld-cloud-init-0.9.8-4.b64   Accessing the instance The instance can be accessed using the SSH key specified at launch. After the first boot the loader will update, download & setup MineMeld on the instance. It could take some minutes:   Welcome to Ubuntu 14.04.3 LTS (GNU/Linux 3.13.0-68-generic x86_64) [...] Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by applicable law. Initalizing minemeld. It could take some minutes, please wait ..|....... After a couple of minutes, you should see message where both minemeld-engine and minemeld-web are RUNNING.   [...] Initalizing minemeld. It could take some minutes, please wait ..|....... DONE ! MINEMELD STATUS: minemeld-engine RUNNING pid 13086, uptime 0:00:04 minemeld-web RUNNING pid 13087, uptime 0:00:04   Accessing the WebUI After the initialization is completed, the WebUI can be accessed on port 443 via https with default credentials   Username: admin Password: minemeld 
View full article
lmori ‎03-12-2019 07:25 AM
14,954 Views
1 Reply
1 Like
WARNING: This article is out of date Please use follow this article to install MineMeld on a recent Ubuntu 16.04 distribution: https://live.paloaltonetworks.com/t5/MineMeld-Articles/Manually-install-MineMeld-on-Ubuntu-Server-16-04/ta-p/253336     Note for VMWare ESXi or environments with no DHCP The best way to install MineMeld on VMWare ESXi or in environments with no DHCP is using a full Ubuntu 14.04 VM: https://live.paloaltonetworks.com/t5/MineMeld-Articles/Running-MineMeld-on-Ubuntu-Server-14-04/ta-p/98454   To run a basic set up of MineMeld on VMWare desktop you can use a simple loader that will automatically install the required packages on a stock Ubuntu 14.04 LTS cloud image and set up a basic config of MineMeld for you. This loader is packaged in a cloud-init ISO image that should be added to the standard Ubuntu OVA.   NOTE: Internet connectivity is required to setup MineMeld. First time image is booted it will automatically download all the newest required components and install them on the Ubuntu cloud image.   Video A video about installing MineMeld on VMware desktop is now availble: click here   Super fast setup To set up a demo installation, this OVA is an original Ubuntu 14.04 cloud image with an embedded MineMeld cloud-init ISO image: https://minemeld.s3.amazonaws.com/minemeld-vm-0.9.11-3build2.ova At first boot the Ubuntu 14.04 image will check for the cloud-init ISO and run the MineMeld loader.  Just download the OVA, import it into VMWare and skip to step #5 of the procedure.   1. Ubuntu 14.04 LTS cloud image MineMeld cloud-init ISO has been designed to run on Ubuntu cloud images.  Download the ova of Ubuntu trusty cloud image from here: https://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64.ova   2. MineMeld cloud-init ISO Download MineMeld cloud-init ISO from the following link: https://minemeld.s3.amazonaws.com/minemeld-cloud-init-0.9.11-3build2.iso   3. Importing the OVA Select File > Import from VMWare Fusion and select the Ubuntu trusty OVA image. When asked do not press Finish but Customize Settings .   NOTE. If VMWare throws an error about hardware incompatibility, go to  VM > Manage > Change Hardware Compatibility and change Hardware Compatibility to the most recent version.   NOTE. If VMWare throws an error about the OVF format just pressy Retry                                     4. Adding the MineMeld cloud-init ISO Under VM settings, enable CD-ROM and select the MineMeld cloud-init ISO image and boot the image.   5. Accessing MineMeld VM shell Default credentials for shell access are:   Username: ubuntu Password: rsplizardspock   6. Wait for MineMeld setup At first boot the loader contained inside the ISO image will update the VM and download & setup the latest version of MineMeld. It could take some minutes.    After a successful setup you should see a message where both minemeld-engine and minemeld-web services are RUNNING.     7. Accessing MineMeld WebUI Default credentials for WebUI access are on port 443 are:   Username: admin Password: minemeld
View full article
lmori ‎03-12-2019 07:25 AM
71,268 Views
4 Replies
6 Likes
Introduction Although MineMeld was conceived as a threat sharing platform, reality has shown many users are taking advantage of its open and flexible engine to extract dynamic data (not threat indicators) from generic API's. The highly successfull case of extracting O365 dynamic data (IP's, domains and URL's) from its public facing API Endpoint Many users relying on MineMeld to track the public IP space from cloud and CDN providers like AWS, Azure, CloudFlare, Fastly as a much more robust and scalable alternative to mapping them with FQDN objects. Or even people using MineMeld to extract the list of URL's to videos published in specific YouTube playlist or channels via the corresponding Google API. All these are examples of MineMeld being used to extract dynamic data from public API's.   Depending on the source, a new class (python code) may be needed to implement the client-side logic of the API we're willing to mine. But, in many case, the already available ready-to-consume "generic classes" could be used instead. This way the user could "mine" its generic API without the need to deep dive into the GitHub project contribution.   The "generic classes" There are, basically, three "generic classes" that can be reused in many applications: The HTTPFT class: Create a prototype for this class when you need to extract dynamic data from content delivered in HTML or PlainText (text/plain, text/html) The SimpleJSON class (I love this one!): Do you need to extract dynamic data from an API that delivers the response as a JSON Document? You're all set with a protorype of this class! The CSVFT class: Some services still use variants of CSV (delimiter-based multi-field lines) to deliver its content. The following is the rule of thumb that will let you know if the API you want to extract dynamic data from can be "mined" using MineMeld by providing just a prototype for one of these classess (without providing a single line of code!) The transport must be HTTP/HTTPS None or basic authentication (user + password) Single transaction (one call retrieves the whole indicator list – no pagination) Indicators are provided in plain, html, csv or json format. The following sections in this article will teach you how to use these generic classes to mine an example API that provides real-time temperature for four MineMeld-relateed cities in the world:   Format API URL CSV https://test.minemeld.com/csv HTML https://test.minemeld.com/html JSON https://test.minemeld.com/json   Mining a CSV API We will start with CSV because it is, probably, the easiest one between the generic classes. The theory of operations is: The CSVFT class will perform a HTTP/S API Call without (or basic) authentication. The expected result will be table-like document where every line will contain an indicator plus additional attributes separated by a known delimiter. Before the CSV parser kicks in, a regex pattern will be used to discard lines that should not be processed (i.e. comments) The prototype will provide configuration elements to the CSV parser to perform the correct field extraction from each line.   First of all, lets call the demo csv api and analyze the results:   Request -> GET /csv HTTP/1.1 Host: test.minemeld.com Response Headers <- HTTP/2.0 200 OK content-type: text/csv content-disposition: attachment; filename="minemeldtest.csv" content-length: 432 Response Body <- # Real-Time temperature of MineMeld-related cities in the world. url,country,region,city,temperature https://weather.yahoo.com/country/state/city-772777/,Spain,Catalonia,Sant Cugat Del Valles,23 https://weather.yahoo.com/country/state/city-719975/,Italy,ER,Parma,26 https://weather.yahoo.com/country/state/city-2488836/,United States,CA,Santa Clara,18 https://weather.yahoo.com/country/state/city-1968212/,Israel,Tel Aviv,Tel Aviv,30   The API returns a test/csv content and suggests us to store the results as an attachment with the name minemeldtest.csv. Regarding the content, it looks like 4 data records are provided with up to 5 fields each: url, country, region, city and temperature. There are, as well, two lines that do not provide any value and that should be discarded (the one with the comment and the one withe the field headers) And, as for the CSV parsing task, it looks like the fields are clearly delimited by the comma character. We're ready to go to configure our prototype to mine this API with the CSVFT class.   Step 1: Create a new prototype using any CSVFT-based one as starting point. We will use the prototype named "sslabusech.ipblacklist" as our starting point. Just navigate to the config panel, click on the lower right icon (the one with the three lines) to expose the prototype library and click on the sslabusech one.         Cliking on the sslabuse prototype will reveal its configuration as shown in the following picture.       The most important value in the prototype is the class is applies to. In this case, the CSVFT one we want to leverage. Our mission is to create a new prototype and to change its configuration to accomplish our goal to mine the demo CSV API. The following is the set of changes we will introduce: Name, Description and Tags (to make it searchable in the prototype library) Inside the CONFIG section: We will replace url with https://test.minemeld.com/csv We will change the indicator type to URL and set the confidence level to 100 Provide our own set of fieldnames Define the ignore_regex pattern as "^(?!https)" (to discard all lines except the ones starting with "https") Describe the source_name as minemeld-test Simply click on the NEW button and modify the prototype as shown in the following picture.       Please, take a closer look to the fieldname list and realize the first name in our prototype list to be "indicator" (in the CSV body the first field was suggested to be "url" instead). The CSV engine inside the CSVFT class will extract all comma separated values from each line and use the one matching the column named "indicator" as the value containing the indicator we want to extract. Any other fieldname will be extracted and attached as additional attributes to the indicator.   Step 2: Clone the prototype as a working node (miner) in the MineMeld engine Clicking on OK will store this brand new prototype into the library and the browser will be sent to it. Just change the search field to reveal our csv prototype and then click on it.       Now it is time to clone this prototype into a working node into the MineMeld engine. So just click on the CLONE button, give the new miner node a name and commit the new configuration.         Step 3: Verify the node status. Once the engine restarts you should see a new node in your MineMeld engine with 4 indicators in it. Click on it, then click on its LOG button and, finally, click on any log entry to reveal the indicator details.           As shown in the last picture, the extracted indicators are of URL type and additional attributes like city, region, country and temperature are attached to it.   Other optional configuration parameters supported by the CSVFT class are: fieldname: in case it to be null, then the values extracted from the first parsed lines will be used as fieldnames (remember that one of the fields must be named "indicator") delimiter , doublequote , escapechar , quotechar and skipinitialspace control the CSV parser behavior as described in the Python reference guide   Mining a HTML API In this section you will be provided with steps needed to use the HTTPFT class to mine dynamic data exposed in the response to a HTTP request (typically text/plain or text/html). If you have not done so, please review the complete process described in the section "Mining a CSV API" to understand concepts like "creating a new prototype", "cloning a prototype as a working node", etc.   To build a new HTTPFT class we first need base prototype that already leverages this class. In this example we will use the prototype named dshield.block as the base.       Let's take a deeper look to the HTML API response to figure out how to generate a valid prototype to accomplish our mission.   Request -> GET /html HTTP/1.1 Host: test.minemeld.com Response Headers <- HTTP/2.0 200 OK content-type: text/html content-length: 1626 Response Body <- <!DOCTYPE html><html><head><link rel="stylesheet" href="https://stackpath.bootstrapcdn.com/bootstrap/4.1.1/css/bootstrap.min.css" integrity="sha384-WskhaSGFgHYWDcbwN70/dfYBj47jz9qbsMId/iRN3ewGhXQFZCSftd1LZCfmhktB" crossorigin="anonymous"><title>Real-Time temperature of MineMeld-related cities in the world.</title></head><body><div class="container"><table class="table-striped"> <tr><td><code>url</code></td><td><code>country</code></td><td><code>region</code></td><td><code>city</code></td><td><code>temperature</code></td></tr> <tr><td><code class="small">https://weather.yahoo.com/country/state/city-772777/</code></td><td><code class="small">Spain</code></td><td><code class="small">Catalonia</code></td><td><code class="small">Sant Cugat Del Valles</code></td><td><code class="small">26</code></td></tr> <tr><td><code class="small">https://weather.yahoo.com/country/state/city-719975/</code></td><td><code class="small">Italy</code></td><td><code class="small">ER</code></td><td><code class="small">Parma</code></td><td><code class="small">27</code></td></tr> <tr><td><code class="small">https://weather.yahoo.com/country/state/city-2488836/</code></td><td><code class="small">United States</code></td><td><code class="small">CA</code></td><td><code class="small">Santa Clara</code></td><td><code class="small">18</code></td></tr> <tr><td><code class="small">https://weather.yahoo.com/country/state/city-1968212/</code></td><td><code class="small">Israel</code></td><td><code class="small">Tel Aviv</code></td><td><code class="small">Tel Aviv</code></td><td><code class="small">30</code></td></tr> </table></div></body></html>   So, what do we have here? A HTML table whose rows are provided in individual file lines and with each value in its own table cell.   First of all we have to get rid of all lines not belonging to table rows. We can achieve this with the ignore_regex class configuration parameter.   ignore_regex: ^(?!<tr><td>)   Next, we need a regex pattern to extract and transform our values from each line. The HTTPFT class leverages Python's re module and accepts configuration parameters both for the indicator itself and any additional attribute. Any Regular Expression strategy will be valid. We will use the following one in this example:   (<td><code class="small">([^<]+)<\/code><\/td>)(<td><code class="small">([^<]+)<\/code><\/td>)(<td><code class="small">([^<]+)<\/code><\/td>)(<td><code class="small">([^<]+)<\/code><\/td>)(<td><code class="small">([^<]+)<\/code><\/td>)   It is a large expression with 10 capturing groups. The first capturing group (\1) extracts the first cell and the second capturing group (\2) the value inside that given cell. Group 3 extracts cell number 2 and group 4 the value inside that second cell. And so on.   As the indicator (the URL) is in the first cell, then the corresponding configuration to achieve our goal must be:   indicator:     regex: '(<td><code class="small">([^<]+)<\/code><\/td>)(<td><code class="small">([^<]+)<\/code><\/td>)(<td><code class="small">([^<]+)<\/code><\/td>)(<td><code class="small">([^<]+)<\/code><\/td>)(<td><code class="small">([^<]+)<\/code><\/td>)'     transform: \2   For the remaining attributes we can leverage the same regular expression but with different transformations.   fields:     country:         regex: '(<td><code class="small">([^<]+)<\/code><\/td>)(<td><code class="small">([^<]+)<\/code><\/td>)(<td><code class="small">([^<]+)<\/code><\/td>)(<td><code class="small">([^<]+)<\/code><\/td>)(<td><code class="small">([^<]+)<\/code><\/td>)'         transform: \4     region:         regex: '(<td><code class="small">([^<]+)<\/code><\/td>)(<td><code class="small">([^<]+)<\/code><\/td>)(<td><code class="small">([^<]+)<\/code><\/td>)(<td><code class="small">([^<]+)<\/code><\/td>)(<td><code class="small">([^<]+)<\/code><\/td>)'         transform: \6     city:         regex: '(<td><code class="small">([^<]+)<\/code><\/td>)(<td><code class="small">([^<]+)<\/code><\/td>)(<td><code class="small">([^<]+)<\/code><\/td>)(<td><code class="small">([^<]+)<\/code><\/td>)(<td><code class="small">([^<]+)<\/code><\/td>)'         transform: \8     temperature:         regex: '(<td><code class="small">([^<]+)<\/code><\/td>)(<td><code class="small">([^<]+)<\/code><\/td>)(<td><code class="small">([^<]+)<\/code><\/td>)(<td><code class="small">([^<]+)<\/code><\/td>)(<td><code class="small">([^<]+)<\/code><\/td>)'         transform: \10   Combine all these configuration parameters into our desired HTTPFT class prototype as shown in the following picture.       And, finally, clone this prototype as a working (miner) node into the MineMeld engine to verify it is working as expected.        Other optional configuration parameters supported by the HTTPFT class are: verify_cert : It controls if the SSL cetificate should be verified (default - true) or not (false) user_agent : To use a customer provided HTTP User Agent string encoding : To deal with non UTF-8 (default) content.   Mining a JSON API (Please review the complete process described in the section "Mining a CSV API" to understand concepts like "creating a new prototype", "cloning a prototype as a working node", etc.)   The SimpleJSON class features a JMESPath engine to process any JSON document returned by the API call. Take your time to visit http://jmespath.org/ and follow the tutorial to be able to understand all concepts we're going to share in this section.   The following is the theory of operations for JSON miners: The node will perform a non (or basic) authenticated HTTP/S requests and a JSON document is expected to be received in the response. The document will be transformed with a JMESPath expression that MUST convert the original document into an array of simple objects (just key-value pairs). Finally, the indicator key will extract the main value and any additional field key will be extracted and attached as additional attributes to the indicator. Let's take a closer look to the test JSON API we're going to mine:   Request -> GET /json HTTP/1.1 Host: test.minemeld.com Response Headers <- HTTP/2.0 200 OK content-type: application/json content-length: 861 Response Body <- {   "description": "Real-Time temperature of MineMeld-related cities in the world.",   "result": [     {       "url": "https://weather.yahoo.com/country/state/city-772777/",       "country": "Spain",       "region": "Catalonia",       "city": "Sant Cugat Del Valles",       "temperature": "22"     },     {       "url": "https://weather.yahoo.com/country/state/city-719975/",       "country": "Italy",       "region": "ER",       "city": "Parma",       "temperature": "21"     },     {       "url": "https://weather.yahoo.com/country/state/city-2488836/",       "country": "United States",       "region": "CA",       "city": "Santa Clara",       "temperature": "20"     },     {       "url": "https://weather.yahoo.com/country/state/city-1968212/",       "country": "Israel",       "region": "Tel Aviv",       "city": "Tel Aviv",       "temperature": "28"     }   ] }   The JSON document looks quite easy and with a element (result) that already provides us the needed array of objects. So, our JMESPath extractor will be:   extractor: result   You can check the expressión in the JMESPath site to verify this expression will return the following array of objects   [ { "url": "https://weather.yahoo.com/country/state/city-772777/", "country": "Spain", "region": "Catalonia", "city": "Sant Cugat Del Valles", "temperature": "22" }, { "url": "https://weather.yahoo.com/country/state/city-719975/", "country": "Italy", "region": "ER", "city": "Parma", "temperature": "21" }, { "url": "https://weather.yahoo.com/country/state/city-2488836/", "country": "United States", "region": "CA", "city": "Santa Clara", "temperature": "20" }, { "url": "https://weather.yahoo.com/country/state/city-1968212/", "country": "Israel", "region": "Tel Aviv", "city": "Tel Aviv", "temperature": "28" } ]   At this point we just need to identify the object attributes that contains 1) the indicator itself and 2) any additional attribute we want to attach to the indicator. In our case, the configuration for it will be:   indicator: url fields: - country - region - city - temperature   It is time to put all these configuration statements into a SimpleJSON class prototype.  We can use, for example, the aws.AMAZON standard library prototype as the base.         Did you noticed the "json" prefix in all extracted additional attributes? You can control that and a few other behaviors of the class with the following optional class configuration elements: prefix: that will be attached to any additional attribute attached to the indicator. verify_cert : It controls if the SSL cetificate should be verified (default - true) or not (false). headers : A list that allows you add additional headers to the HTTP request. client_cert_required , cert_file and key_file can be used to force a HTTP request with client certificate authentication. Bonus Track: using indicator attributes Wondering why would anyone extract additional attributes from the feed and not just the indicator value?   Lets's imagine we want to provide two feeds with Yahoo Weather urls of cities: One of them will be called "time-to-beach" and will list URL's of cities where population is over 30°C and, probably, preparing themselves to go to the beach or outdoor swim pools. The other one called "no-beach-time-yet" will list cities with current temperature bellow 30°C   We can achieve that with the input and output filtering capabilities of the MineMeld engine nodes. Let me share with you a couple of screenshots of the prototypes that will do this job:         Clone each one of these two prototypes as working output nodes and connect their inputs to the JSON miner you created. That should build a graph like the one shown in the picture.         At the time of writing of this article, only one of the four cities in the feed is over 30°C.   GET /feeds/no-beach-time-yet HTTP/1.1 ... https://weather.yahoo.com/country/state/city-2488836/ https://weather.yahoo.com/country/state/city-719975/ https://weather.yahoo.com/country/state/city-772777/ ---- GET /feeds/time-to-beach HTTP/1.1 ... https://weather.yahoo.com/country/state/city-1968212/
View full article
xhoms ‎07-07-2018 01:57 AM
7,001 Views
3 Replies
2 Likes
Use MineMeld DAGPusher to create a basic NAC application
View full article
xhoms ‎03-06-2018 11:26 AM
6,564 Views
1 Reply
1 Like
MineMeld was conceived as a Threat Indicator Sharing platform. Its main goal is as  a tool to help users consume all sorts of threat indicators from diverse sources, aggregate them, age them, then finally make them actionable by featuring very flexible output nodes. Over time, MineMeld has incorporated Data API's and local storage for indicators, which has extended its range of use cases. In this article in the Using MineMeld... series, we'll cover the use case of MineMeld as a Incident Response Platform.   The following bullets describe the need for Incident Response we'll solve in this article: You own a VirusTotal Intelligence account that you use to perform hunting for malware samples with YARA rules. The corresponding VirusTotal notification service provides you with fresh malware incidents (SHA256 hashes) seen on its community of users that matches the YARA rules you've provided. You have deployed PA-Series Next Generation Firewalls for segmentation (either Internet Gateway or Datacenter) with WildFire subscription that unmask unknown malware crossing your network (incident) in less than 5 minutes. A set of endpoints is secured with TRAPS (effectively blocking zero day malware). But you also take care of a large base of endpoints protected by legacy AV solutions that do not provide protections against zero day malware. But that same AV solution can block execution of binaries provided someone could keep a list of SHA256 hashes belonging to malicious files that should be blacklisted. So you want MineMeld to provide a response keeping an updated output node with malware hashes coming both from VT and WildFire incidents that will be consumed by the legagy AV solution. This way you buy time to execute the migration of the large installed base to TRAPS. Step 1. Deploy MineMeld First, visit   https://live.paloaltonetworks.com/t5/MineMeld/ct-p/MineMeld   and select the article (from the top right) about installing and running MineMeld appropriate to your environment. Note, if using the VMWare desktop instructions (https://live.paloaltonetworks.com/t5/MineMeld-Articles/Running-MineMeld-on-VMWare-desktop/ta-p/72038) you can go ahead with the "Super fast setup" but please download the cloud-init ISO and mount it on first boot. Assuming an IP comes via DHCP and you have internet access, your VM will automatically be updated  to the latest version of Minemeld.   Make note of MineMelds IP address (from an ifconfig) and login from your browser (defaults to username: admin / password: minemeld)   Step 2. Create the MineMeld Graph At the end of this article you'll find copy&paste ready snipets. But we highly encourage you to follow the steps to get used to MineMeld's WEB UI features and procedures.   Let's start with a blank configuration. The following is the summary of steps we'll take: Clone the out-of-the-box stdlib.feedHCGreenWithValue prototype into our graph as output node. Clone the out-of-the-box stdlib.aggregatorSHA256 prototype into our graph as aggregator node. Create a new prototype based on the virustotal.notifications one to be able to share VT intelligence into the green share level. Create a new prototype based on the stdlib.localDB  one to be able to store WildFire events into the green share level. Link the nodes to form a connected graph.   To start we must reach the prototype library that is available by navigating to small icon in the down left part of the Config Tab.     In this example we'll trust all indicators coming from VirusTotal and WildFire. This means that any SHA256 hash delivered by these miners will have a value of 100 in the confidence attribute. So we can just clone the  stdlib.feedHCGreenWithValue prototype that will accept all indicators with confidence > 75 shared in the green level.         We can, as well, clone the  stdlib.aggregatorSHA256 from the prototype library into our graph.         The VirusTotal miner prototype in the library is still tagged as experimental which means that all indicators used by it will be placed in the red share level. As our output node only accepts indicators in the green share level, we must create a new prototype using the one in the library as a base but chaging the share level attribute value.       Note that the prototype in the library shares in the red channel. Click on "new" to create a new prototype that shares in the green level instead.     Now it is time to clone this recently created prototype into our graph.         We, finally, need a miner node that could receive async events (WildFire logs from a PANOS device) and store them into a local database. This miner must also provide aging for the indicators so they are removed from the list at a given expiration time. This is what the class minemeld.ft.localdb.Miner exposed through the stdlib.localDB prototype is all about.   As in the case of the VirusTotal prototype, this node is tagged as experimental which means we must first create a new prototype to share indicators in the green level before we can clone it into our graph.               At this moment our configuration should look like the one in the following picture. Note that the nodes are not linked. We can bind them together by clicking on the input cell of both the aggregator and the output node.     Once all nodes are conneted we just need to commit the configuration to get our working graph.     The graph can be shown by clicking on any node in the Nodes Tab and selecting the graph view.      Step 3. Configure authentication in the output node A good practice is to use authentication in the output nodes so only approved clients can grab indicators from it. MineMeld features a very flexible feed authentication mechanism: The administrator can create "feed users" that can latter be attached as "tags" to the output nodes. A given output node can host multiple tags which means it will accepts connections from more that one customer.   First of all let's access the feed user panel available in the Admin main tab. Once there, a new feed user can be added by clicking on the plus icon.        The feed user must then be attached to the output node. That can be achieved by clicking on the output node from the Nodes main Tab. In the node's status pannel you must click on the TAG row to attach the feed user authentication to that node.               We are all set for now. We just need the miners to start gathering indicators for us.   Step 4. Connect MineMeld with VirusTotal You might have noticed the small error icon besides the VirusTotal miner node. That error indicates that the node is not fully configured. A mandatory attribute is not yet provided: The VirusTotal Intelligece API Key.   The key can be obained from the VirusTotal page under the profile settings. Note, please, that the API Key must come from a Virus Total Account with Intelligence access permissions. If you're not familiar with VirusTotal malware hunting features please visit https://www.virustotal.com/intelligence/help/malware-hunting/ before using the VT miner.       Once you have copied the key to your clipboard just paste it into the API KEY field available in the status tab of the virustotal node panel.       No commit is needed. Just force a polling by clicking on the icon at the end right of the LAST RUN field. If everything is right the indicators field will be updated with a non zero value. If you want to take a look to these indicators just navigate to the output node an click on the FEED BASE URL to get a plain text version of the list.     Step 5. Configure WildFire incident forwarding The last step in our mission today is to capture WildFire events from our PANOS device to extract SHA256 indicators out of them. To achieve this we'll leverage the HTTP Forwarding feature introduced with PANOS 8.0.   The following is our theory of operations: PANOS generates a WildFire event log with multiple fields. One of them is the "file_digest" that contains the SHA256 of the recently detected malware. A log forwarding object is attached to the security rule that features WildFire inspection The log forwarding object accepts all events (no filter needed) in the WildFire category, formats them in a way that the miner node can consume and forwards it using HTTPS to MineMeld. MineMeld's default distribution includes a self signed certificate. For the HTTPS protocol to work you must: Issue a certificate for MineMeld's web server from a PKI that is in the trust list of the PANOS Device. Alternatively you can use the PANOS Device as a PKI and generate the MineMeld certificate from it. Follow the instructions in https://live.paloaltonetworks.com/t5/MineMeld-Discussions/Minemeld-SSL-Certificates/m-p/112938 to import that certificate into your MineMeld instance. First of all we need a new HTTP Server Profile for the MineMeld. We must provide: A name An address or FQDN Protocol and Port Method Credentials (we can use any MineMeld admin user credential. Feed user credentials won't work)     The localDB miners's API accepts payloads either in JSON or PlainText format. In any case the type and indicator fields are the only mandatory ones. Optionals fields are: share_level: to override miner's share_level value. A given threat intel source might want to specify the share level of a given indicator. ttl: the amount of seconds the indicators should be kept into the store. A null ttl value means the indicator won't ever expire. In this example we'll use a 1h ttl so you can verify how the indicators are aged out by the miner. any other attribute will just be attached to the indicator to enrich it.   A couple of comments in the payload format shown in the picture above: The url format must be that exact literal just changing the text in red to match the miner's node name we want to forward the indicators to. In our example /config/data/WildFireEvent_indicators/append A Content-Type header with the value application/json is needed for the MM API to accept json data. The parameter t must be equal to localdb to forward the received data to a localdb type of miner. The optional parameter h can contain a node name. If it exists the MineMeld engine will trigger a node "sync request" to the named node after the data has been processed. In this case we want the indicator captured to be automatically flushed down the graph path so we request a sync of the WildFireEvent node. Note the $filedigest variable in the payload field. This tells the PANOS forwarding engine to replace that space in the payload with the corresponding field value in the event log. The next step is to create a new log forwarding object to feature this collection of HTTP server and payload format we've just defined.     Finally we must attach this log forwarding object to the security policy rules that features WildFire inspection.     If you do not want to wait for WildFire incidents just trigger a new one by downloading the WildFire test PE file. Once you get your incident just confirm that it is available in the MineMeld's output feed.               MineMeld's feed output node, by default, provides the content in a plain text format. But you can modify this behavior with additional attributes as explained in Parameters for the output node to easy the consumption by third parties. And, why not, contribute to the MineMeld OpenSource project with new output nodes like, McAfee DXL (TIE) or Tanium.   Annex 1: the prototypes Local Prototypes used in this use case (/opt/minemeld/local/prototypes/minemeldlocal.yml)   author: minemeld-web description: Local prototype library managed via MineMeld WebUI prototypes: localDBGreen: class: minemeld.ft.localdb.Miner config: attributes: confidence: 100 share_level: green interval: 3600 description: 'list of indicators. Use a name starting with "wl" to create a whitelist for an aggregator ' development_status: EXPERIMENTAL indicator_types: - any node_type: miner tags: - ConfidenceAny - ShareLevelAny virusTotalShareGreen: class: minemeld.ft.vt.Notifications config: age_out: default: 365d interval: 7200 sudden_death: false attributes: confidence: 100 share_level: green description: 'Miner for VirusTotal Intelligence Notifications feed. ' development_status: EXPERIMENTAL indicator_types: - md5 - sha256 - sha1 node_type: miner tags: - ConfidenceHigh - ShareLevelRed   Graph configuration (Config -> Import)   nodes: VirusTotal: inputs: [] output: true prototype: minemeldlocal.virusTotalShareGreen WildFireEvent: inputs: [] output: true prototype: minemeldlocal.localDBGreen sha256agg: indicator_types: - sha256 inputs: - VirusTotal - WildFireEvent node_type: processor output: true prototype: stdlib.aggregatorSHA256 sha256out: indicator_types: - any inputs: - sha256agg node_type: output output: false prototype: stdlib.feedHCGreenWithValue Annex 2: the API In MineMeld 0.9.42 we have introduced a new type of Miner (stdlib.localDB) exposing the HTTP-based API used in this article. In the following examples we assume that you have created a localDB Miner called FooBar.   A2.1: Push new indicators via JSON You can push new indicators using JSON. The ttl attribute is the expiration time in seconds of the indicator. If omitted the default config value of the Miner will be used. Only the indicator and type attributes are mandatory. curl -XPOST -H "Content-Type: application/json" -u admin:minemeld "https://192.168.55.150/config/data/FooBar_indicators/append?h=FooBar&t=localdb" -d ' { "indicator": "8.8.8.8", "type": "IPv4", "comment": "usual Google DNS Public IP", "share_level": "green", "confidence": 100, "ttl": 3600 }'  A2.2: Delete indicators via JSON You can use the same API call to delete an indicator, by setting the ttl attribute to 0: curl -XPOST -H "Content-Type: application/json" -u admin:minemeld "https://192.168.55.150/config/data/FooBar_indicators/append?h=FooBar&t=localdb" -d ' { "indicator": "8.8.8.8", "type": "IPv4", "ttl": 0 }' A2.3: Push new indicators via text format You can also use a simple text format to push a list of indicators: Indicators are divided by a blank line The first line of an indicator is the type of the indicator The second line is the indicator itself After you can specify the attributes, where a line is the attribute name and the line after is the attribute value Example: curl -XPOST -H "Content-Type: application/text" -u admin:minemeld "https://192.168.55.150/config/data/FooBar_indicators/append?h=FooBar&t=localdb" -d ' URL https://www.google.com comment test comment confidence 80 share_level green ttl 1800 IPv4 8.8.8.8 comment usual Google DNS Public IP share_level green confidence 100 ttl 86400 ' A2.4: Delete indicators via text format You can use the same API call to delete an indicator, by setting the  ttl  attribute to  0 : curl -XPOST -H "Content-Type: application/text" -u admin:minemeld "https://192.168.55.150/config/data/FooBar_indicators/append?h=FooBar&t=localdb" -d ' URL https://www.google.com ttl 0 IPv4 8.8.8.8 ttl 0 '  
View full article
xhoms ‎12-04-2017 10:41 AM
21,263 Views
15 Replies
1 Like
Overview In this article we introduce the classes available in the base MineMeld distribution and how to use them to create a managed services offering for your customer base. Either to differentiate yourself from these pure resellers that do not add any value or to build a new income source.   Installing your MineMeld instance If this is your first time with MineMeld then you better read the Getting Started article and deploy your first MineMeld instance following the article from the following list that better suits your needs. Running MineMeld on VMWare desktop Running MineMeld on Amazon EC2 Running MineMeld on Microsoft Azure Manually install MineMeld on Ubuntu Server 14.04 Base offering In this case example we will start with a basic offering consisting on a high-risk IPv4 feed that any customer should block at the Internet Gateway. That feed will consist on the aggregation of the following three standard miners: dshield.block spamhaus.DROP spamhaus.EDROP In other words, we will create the following graph.     To reach to this configuration point you just have to import the following configuration snippet in the configuration section of the WebUI.   nodes: dshield_blocklist: output: true prototype: dshield.block myBaseOffering: inputs: - spamhaus_EDROP - spamhaus_DROP - dshield_blocklist output: true prototype: stdlib.aggregatorIPv4Generic spamhaus_DROP: output: true prototype: spamhaus.DROP spamhaus_EDROP: output: true prototype: spamhaus.EDROP freeBaseOffering: inputs: - myBaseOffering output: false prototype: stdlib.feedHCGreen         All MineMeld feeds are exposed using a basic authentication mechanism. By default only the MineMeld admistrator user and password can be used to grab the feeds contents. Note, please, that if you check the feed URLs from a WebBrowser that is already connected to the MineMeld WebUI then you won't experience the authentication request because your browser has already been authenticated and do have a session cookie.   To implementa a granular control of feed content by customer or customer type you must create a "feed user" and give it a password in the Admin section of the WebUI.           Once the feed user has been created you must assign it an authentication tag and attach that tag to the output node.         Now it is time to commit the configuration changes.     To check that everything is fine so far just validate the feed is working by attempting to access its data with the corresponding user and password (basic authentication).     user$ curl https://yourBrand:xxxx@minemeld/feeds/freeBaseOffering 185.35.62.0-185.35.62.255 80.82.77.0-80.82.77.255 1.10.16.0-1.10.31.255 1.32.128.0-1.32.191.255 101.192.0.0-101.195.255.255 101.202.0.0-101.202.255.255 101.203.128.0-101.203.159.255 101.248.0.0-101.249.255.255 101.252.0.0-101.253.255.255 102.176.0.0-102.177.255.255 102.181.0.0-102.181.255.255 103.16.76.0-103.16.76.255 ...   Now it is time to configure the Internet Gateway to block any packet coming from or going to any IP address belonging to these ranges. In the following example I will configure an IP External Dynamic List object linked to the feed we have just created and a couple of Policy Based Forwarding rules to discard traffic when these IP's are involved.         If your customer wants to be able to monitor the activity of traffic matching the EDL then you can use Security Policies with drop action instead of Policy Based Forwarding rules.       Adding intelligence to a premium customer What about suggesting a specific SMB customer to subscribe to a premium service where other intelligence data is combined into his reputation fee?   We will continue with our example adding a new miner (the TOR exit nodes) to our instance and modifying the graph in such a way that only a given specific premium customer is allowed to consume this extended feed.       Try to clone nodes from the standard library to create the graph or, if you want to go straight to the point, just import the following configuration snippet into you configuration (use the “Append” button)   nodes: torExitNodes: inputs: [] output: true prototype: blutmagie.tor_exit_nodes premiumOffering: inputs: - myBaseOffering - torExitNodes output: true prototype: stdlib.aggregatorIPv4Generic premiumCustomerA: inputs: - premiumOffering output: false prototype: stdlib.feedHCGreen     And remember to create a new feed user for this premium customer and attach its corresponding authentication tag to the output feed.         Reputation Managed Service The next step in your offering could be to provide a managed service to customize the reputation feed with any arbitrary source data. To do that we’ll add a “local list” (prototype: stdlib.listIPv4Generic) miner for one specific premium customer so the field engineers have a bucket in which any indicator requested by the customer could be placed.       Just append the following configuration snippet to append the two remaining nodes to complete the graph.   nodes: bucketCustomerA: inputs: [] output: true prototype: stdlib.listIPv4Generic customerA: inputs: - premiumOffering - bucketCustomerA output: true prototype: stdlib.aggregatorIPv4Inbound   Note that, at this moment, the premiumCustomerA input will still be attached to the premiumOffering aggregator node. So we must change it to the recently created aggregator node called customerA.       This new bucketCustomerA node provides a way to add new indicators to the managed customer feed. The easiest way is to use the node’s WebUI feature.             MineMeld's API provides you with a way to upload indicators into a list to deal with bulk operation. A nice implementation is available in minemeld-sync tool available in GitHub.   Other Managed Services ideas I hope this article gave you interest in exploring ways to leverage MineMeld to add value to your NGFW deals and, eventualy, use it as a revenue stream.   The following is a list of ideas you could evaluate to extend your value proposition: Automatic provisioning of IOC's to customer's Internet Gateways when a major malware campaign is published (URL's, Domains and IP's). Offering yourself as a bridge between the local security /vertical market CSIRT and the interested customers. Managed service to limit YouTube video exposure by channel or playlist ( more info in Filtering YouTube videos to only approved playlists ) Automatic enforcement of SOC alerts
View full article
xhoms ‎12-04-2017 04:20 AM
5,473 Views
0 Replies
7 Likes
Overview In some cases you might face the need to create a policy rule in a Palo Alto Networks next generation firewall that targets a large list of IP addresses that shares a common schema.   For example: All printers in a set of branch office networks that happens to be the ".7" in a collection of subnets where the third byte is a variable: "192.168.x.0/24" All even networks in the collection "10.10.x.0/24" Using a set of wildcard masks could ease the description of these lists. For example, the first case fits within the wildcard mask 192.168.0.7/255.255.0.255 and the second example in the wildcard mask "10.10.0.0/255.255.1.0"   A miner extension available at https://github.com/PaloAltoNetworks/wildcardip-miner can be used to generate these lists of IP's from user provided wildcard masks.   In this article you'll find all the steps needed to deploy MineMeld and configure a Palo Alto Networks NGFW using the wildcard extension miner.   Step 1. Deploy MineMeld First, visit   https://live.paloaltonetworks.com/t5/MineMeld/ct-p/MineMeld   and select the article (from the top right) about installing and running MineMeld appropriate to your environment. Note, if using the VMWare desktop instructions (https://live.paloaltonetworks.com/t5/MineMeld-Articles/Running-MineMeld-on-VMWare-desktop/ta-p/72038) you can go ahead with the "Super fast setup" but please download the cloud-init ISO and mount it on first boot. Assuming an IP comes via DHCP and you have internet access, your VM will automatically be updated  to the latest version of Minemeld.   Make note of MineMelds IP address (from an ifconfig) and login from your browser (defaults to username: admin / password: minemeld)   Step 2. Add the wildcardip extension MineMeld introduced support for external extensions in the version 0.9.32. You can install them either localy or from a GitHub repository. To use the GitHub (recommended) option you must access to the repo URL available in its main page ("Clone or Download" button)     The URL for the repo is https://github.com/PaloAltoNetworks/wildcardip-miner.git. Use this URL in the "Install Extension From Git" option in MineMeld's WebUI (accessible through the "System" menu)     Once the extension is successfully downloaded, the next step is to activate it.     Step 3. Creating a wildcardip node The wildcard extension provides an example prototype that must be customized for our needs. The following is the sequence of tasks needed to add a node: Create a "New" prototype using the example provided one. Configure this new prototype with the needed list of wildcard masks "Clone" the just created prototype to convert it into a working node. All these tasks are performed under the "Configuration" main menu in MineMeld.       Once we reach the prototype library, use the "search" field to find the wildcard prototype provided by the extension.       Click over the prototype to display its contents and to be able to access the "new" and "clone" operations. In this case we need to create a "new" prototype that suits our needs.       Clicking on "new" will show the YAML prototype editor.       This example creates a wildcard that will generate IP address objects that will match addresses .1 to .15 in a list of networks 192.168.[0-255].0. The meaning of the rest of the attributes in the prototype is: age_out: It describes the "age out policy" of indicators for this node. In this case: default = null : indicators are not aged out interval = 86400 : check for aged out indicators once a day sudden_death = true : remove indicators if they're not longer in the input list interval: the amount of seconds between polls on this miner. We used 86400 to regenerate the list once a day. But it is safe to use larger values. max_entries: a wildcard-class specific attribute that allows the user define a safety limit on the entries that should be generated by this node. The miner will fail to generate entries if the provided wildcard masks extend over this limit. wildcard_list: a list of wildcard masks to be expanded. Verify, please, the YAML syntax (using TAB of each entry in the list) Once the new prototype is added to the library, we can generate a node out of it using the "Clone" procedure.         Next task is to attach an output node to our recently created wildcard miner. Take note of the name provided to the miner because it will be used in the output creation.     Clone the stdlib.feedHCGreen prototype from the list as a node and configure the willdcard miner as its input.     At this moment, configuration should look like the following screen capture. Commit the new configuration.     Step 4: Enable feed authentication Once the configuration is commited a new graph will be available in the "Nodes" main menu.     The number of indicators (256) corresponds is the result of the expansion of the wildcard mask provided in the prototype (192.168.0.0/255.255.0.240)   The details panel of the output contains the URL link to the feed. Clicking in it will provide the resulting list of the wildcard expansion.       At this moment in time is important to note that you have been able to see the feed contents because the browser have a session cookie with the MineMeld WebUI. You can verify it by opening a new "incognito mode" window in your browser and attempting to access the URL from there. You should receive an authentication error in this case.   Before we keep going on, you must replace the original MineMeld SSL self-signed certificates with a valid ones issued by your local CA. The article How to Generate New MineMeld HTTPS Cert explains how to use a NGFW or Panorama as a local CA.   MineMeld feed authentication framework contains the following elements: Feed User: A user/password pair with a list of "Access Tags" it belongs to. Access Tag: A label that can be attached to a feed to limit its exposure to only the users belonging to that tag. Feed Users are created from the "Admin" main menu.     And access tags created clicking on the user's "Access" cell.     Next task is to attach this recently created access tag to the feed. The way to do this is by navigating to the feed node and clicking in its access cell.     All these changes apply at runtime. No commit is needed. We're ready for the next step.   Step 5: PANOS Configuration NGFW from Palo Alto Networks feature a dynamic object called "External Dynamic List" (EDL) that can be used to consume the feed generated by MineMeld.   The following components are needed to successfuly complete the EDL configuration: The URL of the feed. A valid user/password pair that belongs to the access tag of that given feed. Certificate Profile including the CA that issued MineMeld's SSL certificate (for PANOS 8.0) NGFW Policy referencing the EDL (PANOS won't try to download the EDL until it is consumed somewhere)     User and password fields will not appear until you select one certificate profile. For PANOS 7.1 or before you must use the https://<username>:<password>@url syntax instead.     Final EDL configuration should look like the following screenshot.     Once configuration is commited, list entries should appear (provided that you are consuming this EDL anywhere in your NGFW configuration)     Annex A: Advanced MineMeld wildcard use case Imagine that you're asked to extend the previous case removing the ".5" nodes from the list. This could be described like substract the list created by "192.168.0.5/255.255.0.255" from the list created by "192.168.0.0/255.255.0.240".   This can be achieved easily with the "whitelist" feature of the stdlib.aggregatorIPv4Generic processor. We just need to add a new wildcard miner with the mask "192.168.0.5/255.255.0.255", give it a name starting with "wl_" (whitelist) and combine it with the miner from the previous exercice.     If you take a look to the feed output, you'll notice the "aggregator magic". It now describes the address spectrum as a list of ranges. For each network you have two ranges: 192.168.x.0-192.168.x.4 and 192.168.x.6-192.168.x.15    
View full article
xhoms ‎11-27-2017 04:45 AM
8,018 Views
3 Replies
1 Like
Local indicators MineMeld supports Miners where the list of indicators is stored in a local database inside the MineMeld instance. These Miners can be used to define a static list of malicious indicators or a static whitelist (for more details about using a Miner as a whitelist check the article https://live.paloaltonetworks.com/t5/MineMeld-Articles/Creating-whitelists/ta-p/72250).   You can add, delete and change the indicators stored in these Miners directly from the WebUI (the following screenshot shows an example of a whitelist, same applies to malicious indicator lists):   Prototypes for Miners of local indicators stdlib.listDomainGeneric List of domain indicators stdlib.listIPv4Generic List of IPv4 indicators stdlib.listIPv6Generic List of IPv6 indicators stdlib.listURLGeneric List of URL indicators   Uploading indicators to MineMeld Using the MineMeld API you can upload indicators to Miners of local indicators. This can be automated using the minemeld-sync.py script: https://gist.github.com/jtschichold/95f3906566b18b50cf2e3e1a44f1e785   Requirements To use the script Python 2.7.9+ is required. If you are planning to use the script from Linux or Mac OS X, python should be already available in your environment. Otherwise you can download and install python from here https://www.python.org   List of indicators The list of indicators to be uploaded should be stored in a plain text file, one line per indicator: 8.8.8.8 8.8.4.4 10.0.0.0/8 You can also add comments to each indicator, to be stored in the indicator comment attribute in MineMeld: # Google public DNS (this will be placed in the comment attribute) 8.8.8.8 # Google public DNS (this will be placed in the comment attribute) 8.8.4.4 # Private network (in the comment attribute, again) 10.0.0.0/8 You can also specify custom attributes, with the format @<attribute name>: <attribute value>: # Google public DNS (this will be placed in the comment attribute) # @direction: inbound 8.8.8.8 # Google public DNS (this will be placed in the comment attribute) # @direction: inbound 8.8.4.4 # Private network (in the comment attribute, again) # @direction: inbound # @confidence: 60 10.0.0.0/8   How to Download the script from this webpage and save it in a file called minemeld-sync.py. If you have wget installed you can use: wget -O minemeld-sync.py https://gist.githubusercontent.com/jtschichold/95f3906566b18b50cf2e3e1a44f1e785/raw Use the following command line to upload the IPv4 indicators stored in the file indicators.lst to the MineMeld instance with URL https://192.168.1.1, using the username admin and password minemeld: python minemeld-sync.py -m https://192.168.1.1 -u admin -p minemeld -t IPv4 --share-level red wlWhiteListIPv4 indicators.lst (OPTIONAL) By default old indicators are not removed from the Miner. If you prefer a full sync of the local list with the remote miner, add the options --update and --delete to the command line:   python minemeld-sync.py -m https://192.168.1.1 -u admin -p minemeld -t IPv4 --delete --update --share-level red wlWhiteListIPv4 indicators.lst   Remote certificate verification By default remote MineMeld certificate is verified using certifi package (if installed), or using the CA bundle file or CA certs directory specified via the --ca-path option: python minemeld-sync.py -m https://192.168.1.1 --ca-path /etc/ssl/certs -u admin -p minemeld -t IPv6 IPv6ListMiner my-ipv6-addresses.lst   To disable remote certificate verification use the option -k: python minemeld-sync.py -m https://192.168.1.1 -k -u admin -p minemeld -t IPv6 IPv6ListMiner my-ipv6-addresses.lst  
View full article
lmori ‎07-14-2017 01:59 AM
13,743 Views
9 Replies
2 Likes
User credentials for MineMeld UI are stored in /opt/minemeld/local/config/api/wsgi.htpasswd in the standard Apache2 htpasswd format.   1. Installing htpasswd utility It's not installed by default, it is contained in the apache2-utils package. $ sudo apt-get install apache2-utils   2. Changing UI admin password $ sudo htpasswd /opt/minemeld/local/config/api/wsgi.htpasswd admin New password: Re-type new password: Updating password for user admin   3. Restarting minemeld-web To enforce the new credentials the MineMeld Web API should be restarted. This doesn't restart the MineMeld engine, just the frontend component. $ sudo -u minemeld /opt/minemeld/engine/current/bin/supervisorctl -c /opt/minemeld/supervisor/config/supervisord.conf restart minemeld-web minemeld-web: stopped minemeld-web: started    NOTE:  To add a new user to the UI just use the htpasswd command at step 2 with a new username instead of admin.
View full article
lmori ‎07-02-2017 11:59 PM
11,862 Views
7 Replies
Using MineMeld TAXII output nodes and IBM QRadar Threat Intelligence app, it is possible to populate IBM QRadar reference sets with Threat Intelligence indicators processed by MineMeld. Reference Sets can then be used in IBM QRadar rules to detect suspicious activities   After installing IBM QRadar Threat Intelligence app (available on IBM QRadar App Exchange), follow this procedure to connect IBM QRadar to MineMeld: Create one or more TAXII DataFeed Output nodes in MineMeld Define authentication for TAXII DataFeed nodes (optional) Upload CA certificate to IBM QRadar Threat Intelligence app Configure one or more reference sets on IBM QRadar (optional) Configure MineMeld TAXII Feeds on IBM QRadar Threat Intelligence app 1. Create TAXII DataFeed Output nodes in MineMeld Use stdlib.taxiiDataFeed prototype to instantiate one or more output nodes. Each output node becomes a new TAXII data feed IBM QRadar can grab indicators from. In this picture all the taxiiKnownCampaigns* nodes are TAXII DataFeed nodes.   2.  Define authentication for  TAXII DataFeed nodes This step is required only if you have enabled authentication for feed access.   Note for MineMeld on AutoFocus Authentication for feeds is automatically enabled in MineMeld on Autofocus.   2.1. Define a feed user Under Admin > Feeds Users create a new user and associate an access tag to it. The new feed user does not have access to the Admin WebUI, but only to feeds tagged with at least one of the tags listed in the ACCESS field. In the following picture SOC_QRadar user has access only to feeds tagged with siem.   2.2. Configure tags on TAXII DataFeed nodes Under Nodes select the TAXII DataFeed nodes and add the access tag.                                   3. Upload CA certificate to IBM QRadar TI app IBM QRadar Threat Intelligence app requires a valid certificate on the TAXII server. If the certificate on your MineMeld instance is signed by a private CA or a CA not known to IBM QRadar Threat Intelligence app, you have to upload the certificate on the app. The certificate should be in PEM format and the extension of the file should be pem.   Note for MineMeld on AutoFocus Download the GoDaddy Class2 Root certificate here https://certs.godaddy.com/repository/gd-class2-root.crt, change the extension to pem and upload it to the app.                                                       4. Configure one or more reference sets on IBM QRadar This step is optional. To keep MineMeld indicators separated from other sources you can define a new Reference Set for each MineMeld DataFeed.                                                                   5. Configure TAXII Feeds In the IBM QRadar Threat Intelligence app, select Add TAXII Feed.   5.1. Configure TAXII server parameters In TAXII Endpoint set https://<minemeld address>/taxii-discovery-service If feeds authentication is enabled on MineMeld, select HTTP Basic in Authentication Method and set Username and Password of a MineMeld feed user with access to the TAXII DataFeed. Then click on Discover.                                       5.2. Select the collection In the next dialog select the Collection, and set the appropriate Observable Type - that is the type of the indicators in the MineMeld TAXII DataFeed. Each MineMeld TAXII DataFeed node is seen as a separate Collection.   5.3. Select the target reference set In the next dialog, select the target reference set.   5.4. Save & Poll Click on Save.     In the TAXII Feed list, click on Poll Now to retrieve the indicators from the datafeed.  
View full article
lmori ‎02-16-2017 03:50 AM
11,142 Views
7 Replies
1 Like
Think of a prototype as a node template that can be instantiated inside the MineMeld engine config to define a new node.   Learn about customizing prototypes in this article.
View full article
lmori ‎07-21-2016 05:51 PM
22,630 Views
18 Replies
1 Like
In a default config the minemeld-auto-update utility connects once per day to MineMeld auto update API to check if a new version of a MineMeld package has been released. If a new version is detected, the package is automatically downloaded and installed.   It is possible to prevent this by deleting the auto update cron job: $ sudo rm -rf /etc/cron.daily/minemeld-auto-update To reenable the auto update cron job, create the file  /etc/cron.daily/minemeld-auto-update with the following contents: #!/bin/bash /usr/sbin/minemeld-auto-update >> /var/log/minemeld-auto-update.log 2>&1  and mark it as executable.
View full article
lmori ‎02-24-2016 03:17 AM
3,889 Views
1 Reply
By default the MineMeld loader installs in the VM a scheduled job inside the VM to automatically check and install the latest version of the MineMeld packages. The scheduled auto update job runs once per day.   To force a an update just run the minemeld-auto-update utility: $ /usr/sbin/minemeld-auto-update   To disable the auto update job check the article Disabling MineMeld auto update.  
View full article
lmori ‎02-04-2016 05:39 AM
20,133 Views
11 Replies
MineMeld can analyze PAN-OS syslog messages and match them against live indicators. In case of match, the result of the match (full session info and full details of the indicator) can be sent to a logstash (https://www.elastic.co/products/logstash) instance for forwarding to an external database.   1. Message flow The following diagram illustrates the message flow of the syslog analysis process. NOTE: while rsyslog is already installed and configured by the MineMeld loader on the VM, logstash is not installed by default. To save matching results in an external collector logstash must be installed and the syslog should be configured to send results to the logstash instance.   2. Adding a syslog analyzer node Under CONFIG, click + to add a new node. Select prototype stdlib.localSyslog if you don't need to send results to an external logstash instance, select prototype stdlib.localSyslogToLogstash if you plan to send results to a local logstash instance. Select an aggregator as input.   NOTE: syslog analyzer node supports only one aggregator per type as upstream node. If you have multiple aggregators, you can use multiple syslog analysis nodes or create an additional generic aggregator just for syslog analyzer.     3. Configuring syslog forwarding on PAN-OS Please refer to the PAN-OS Administration guide for instructions on how to configure log forwarding to a syslog server on PAN-OS (https://www.paloaltonetworks.com/documentation/70/pan-os/pan-os/monitoring/configure-log-forwarding.html#24849). By defeault the syslog server on the VM listens for PAN-OS syslog messages on port tcp/13514. 4. Checking counters on the syslog node Under NODES, click on the syslog analyzer node. Select STATS in the left menu, and check the counters.   SYSLOG.PROCESSED is the number of syslog messages received from PAN-OS devices and processed. TOTAL_MATCHES is the number of matching indicators found. LOGSTASH.SENT is the number of results sent to the logstash instance (if configured).   5. Evaluating indicators sources Under NODES, click on the syslog analyzer node. Select SOURCES in the left menu to see the which feeds have produced matching indicators.   6. Configuring logstash (optional) If at step 2 you have chosen prototype stdlib.localSyslogToLogstash, the syslog analyzer node will send matching results to logstash on port 5514/tcp. Use the following logstash snippet to let logstash listen on port 5514 for messages from MineMeld syslog nodes. input { tcp { port => 5514 host => '127.0.0.1' codec => 'json_lines' } } NOTE: logstash is not installed by default on the VM. Please refer to the following link for instructions on how to install logstash and an ELK stack on the VM or on an external VM: - official site https://www.elastic.co/ - step by step tutorial from DigitalOcean community https://www.digitalocean.com/community/tutorials/how-to-install-elasticsearch-logstash-and-kibana-elk-stack-on-ubuntu-14-04   Kibana view of the matching results:
View full article
lmori ‎02-04-2016 02:53 AM
14,732 Views
14 Replies
The default config installed by the MineMeld loader is really simple, and can be seen in the following graph.   The 3 green nodes on the left are called Miners and are responsible for periodically retrieving indicators from 3 different feeds and push them downstream to the connected nodes using update messages. Miners are also responsible for aging out indicators: when indicators disappear from the original feed or when an indicator is considered dead, the corresponding Miner instructs the downstream nodes of removing the indicator via a withdraw message.   The central red node is a Processor node. In this specific configuration, the processor node is an IPv4 aggregator node and aggregates IPv4 indicators received from the 3 Miners and sends downstream the aggregated indicators.   The 3 yellow nodes on the right are Output nodes. These nodes receive indicators from the processor nodes and transform them into a format that could be directly consumed by external entities. In the default config the 3 output nodes translates the indicators received from the aggregator node into a format that can be consumed using the PAN-OS Dynamic Block List feature. All 3 output nodes in this graph receive the same set of indicators from the aggregator node, but each of them stores a different subset of these indicators based on the configured input filters. inboundfeedhc accepts  only indicators with confidence level > 75, inboundfeedmc only indicators with confidence level < 75 and > 50, inboundfeedlc indicators with confidence level < 50. These subset of indicators are stored into 3 different DBLs that can be used in different ways inside the PAN-OS configuration.   1. Login Default credentials are admin / minemeld     2. Dashboard The dashboard page can be used to check the overall status of the MineMeld processing engine.   In the first row you can find the number of active nodes per type, the current total number of indicators stored inside the graph and an historical chart of the total number of indicators.  The second row is dedicated to Miners, the first panel reports the total number of indicators stored in the miner nodes, and the number of indicators that have been added and aged out since engine start. The historical chart on the right represents the number of indicators added and aged out by the miner nodes in the last period of time. The third row reports the same statistics of the second row but related to Outputs.   3. Nodes The nodes view reports the status of every single active node of the graph.   4. Node status Clicking on a node in the Nodes view shows the details of the configuration of the node   the historical charts of the metrics of the node   and the connected graph where the node lives   5. Nodes and prototypes Nodes are created using libraries of prototypes. Each prototype defines the type of node and its configuration. To check the details of a prototype, just click on a node in the Nodes view and move the pointer over the prototype field   Clicking on the prototype name shows all the details about that prototype   6. PAN-OS External Dynamic Lists To use MineMeld feeds with PAN-OS External Dynamic Lists (named Dynamic Block Lists before PAN-OS 7.1), select the output node responsible for the feed in the Nodes view and copy the FEED BASE URL     In PAN-OS, under Object > External Dyanmic Lists (or Object > Dynamic Block Lists in PAN-OS before 7.1) define a new list using the output node URL   7. Whitelist indicators To whitelist a network, range or a single IPv4 address, click on wlWhiteListIPv4 under Nodes and select the INDICATORS option on the left. This list of indicators is treated as a whitelist by the aggregator.   To add a new indicator click on + and specify the IPv4 indicator. Changes will be automatically saved by pressing OK.      
View full article
lmori ‎02-03-2016 02:26 AM
63,956 Views
2 Replies
2 Likes
The default config provides a graph for handling inbound IPv4 Threat Indicators, IPv4 addresses related to suspicious inbound activities like brute forcing on scanning. In this article we will add a new Miner to this graph.   1. Adding a Miner   Click on CONFIG in the top navigation bar.     And press + to add the node.   2. Configuring the Miner   Select the PROTOTYPE, and leave the INPUTS field empty. Enable the OUTPUT. Press OK when done.   Note. if you leave the pointer on a prototype a tooltip appears with the description of the prototype.     3. Linking the Miner to the aggregator   Now you have created a new Miner in the candidate config, but the Miner is not linked to any downstream node.       Click on the INPUTS field of the inboundaggregator node and add the new Miner to the list. Press OK when done.     4. Commit the config   Now you should have a new Miner connected to the inboundaggregator.     Press COMMIT to apply the config.   5.  Check the engine status   Click SYSTEM in the top navigation bar to check the engine status. It should stop and then start.       The processing graph after the change should look like this:  
View full article
lmori ‎02-03-2016 02:02 AM
5,646 Views
2 Replies
The simple, default config included in MineMeld VM creates a graph to process IPv4 indicators for inbound connections, typically used to filter out scanning hosts or well known brute force attackers. For IPv4 indicators for outbound connections we can define a new sub-graph with its own set of output feeds. These new set of feeds can then be used in the destination part of the PAN-OS security policies.   1. Adding an outbound IPv4 aggregator Under CONFIG press +. Configure a new node with prototype stdlib.aggregatorIPv4Outbound and Output enabled.   2. Adding a set of feeds Under CONFIG add 3 new nodes (HC, MC and LC) for the output feeds and select the node created at point 1 as Input.   First node with stdlib.feedHCGreenWithValue as prototype   Second node with stdlib.feedMCGreenWithValue as prototype   Third node with stdlib.feedMCGreenWithValue as prototype   3. Adding a Miner Under CONFIG add a new Miner generating IPv4 outbound indicators, like zeustracker.badips. Output should be enabled.   4. Connecting the aggregator to the Miner Under CONFIG, click on the INPUTS field of the node created at step 1 and add the Miner.   5. Commit Check the resulting config and press COMMIT.   6. Check the sub-graph The resulting sub-graph should look like this:
View full article
lmori ‎02-03-2016 02:00 AM
3,613 Views
0 Replies
1 Like
In MineMeld you can configure how indicators are retrieved, processed and presented by configuring the MineMeld engine graph. The graph is composed by one or more interconnected nodes.   The definition of a node in the graph is composed by: - Name. A unique node name. - Inputs. A list of nodes the node should receive messages from. - Output. A boolean value that enables/disables messages to downstream nodes. - Class. Defines which kind of processing is applied to indicators. - Config. Configuration of the node class.   Node Inputs A node can have zero or more input nodes. The node will listen to and process messages from the input nodes. Messages between nodes can be used to publish/update indicators, or to withdraw indicators.   In the following graph, the node inboundaggregator has spamhaus_DROP, spamhaus_EDROP and dshield_blocklist as input nodes. If a node has no input nodes it is considered a Miner. In the following graph spamhaus_DROP, spamhaus_EDROP and dshield_blocklist have no input nodes because they are Miners.     Node Output A node can be configured to send messages to downstream connected nodes. In the previous picture spamhaus_DROP, spamhaus_EDROP, dshield_blocklist and inboundaggregator have their Ouput enabled.   Output nodes have their Output disabled - ok, this is a bit misleading - because they don't have downstream nodes. inboundfeedhc, inboundfeedlc, inboundfeedmc are Output nodes and their output is disabled.   Node Class The node class defines what is actually done by the node. In the previous graph spamhaus_DROP, spamhaus_EDROP and dshield_blocklist are all nodes of class minemeld.ft.http.HttpFT, this class of nodes manage feeds of indicators retrievable via URLs and send updates to downstream nodes. The feed URL and how the indicators should be extracted are defined in each node config. inboundaggregator instead is a node of class minemeld.ft.ipop.AggregateIPv4FT. This class of nodes aggregate IPv4 indicators from multiple input nodes and generate a set of non-overlapping IPv4 indicators.   Node Config The node config defines the parameters that should be used by the node class to perform its job. As an example, for the node dshield_blocklist the config defines the URL that should be used to retrieve the list of indicators, the patterns to extract the indicators and the age out policies to apply.   Node Prototype MineMeld comes with a library of node prototypes. A prototype is a semi-pre-canned node definition, and it is composed by a node class and a node config. Prototypes are available for the most commonly used feeds, processors and output nodes.   As an example, this is the zeustracker.badips prototype:   badips: development_status: STABLE node_type: miner config: source_name: zeustracker.badips attributes: type: IPv4 direction: outbound confidence: 100 share_level: green ignore_regex: '#' url: https://zeustracker.abuse.ch/blocklist.php?download=badips interval: 1800 class: minemeld.ft.http.HttpFT  
View full article
lmori ‎02-03-2016 01:58 AM
6,987 Views
0 Replies
2 Likes
Ask Questions Get Answers Join the Live Community
Labels